Problem

Writing DataFrame contents in Delta Lake format to an S3 location can cause an error:

com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3;
Status Code: 403; Error Code: 403 Forbidden; Request ID: C827672D85516BA9; S3 Extended Request ID:

Cause

A write operation involving the Delta Lake format requires permissions that other file formats do not need. For example, Delta Lake requires creation of a _delta_log directory. The write operation also needs to check the latest version of the commit logs. You need to add extra permissions to IAM and bucket roles to enable the write operation to complete successfully.

Solution

Add the following permissions to enable writing of Delta tables:

1. Add these permissions to the IAM policy JSON:

   ["s3:PutObject","s3:DeleteObject", "s3:ListBucket", "s3:GetObject", "s3: PutObjectAcl"]

2. Add these permissions to the bucket policy JSON:

   ["s3:GetObject","s3:GetObjectVersion","s3:PutObject","s3:DeleteObject","s3:ListBucket","s3:GetBucketLocation"]

Alternatively, you can add permissions using an IAM policy in JSON format, as shown here:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-bucket"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
          "s3:PutObjectAcl"
        ],
    "Resource": "arn:aws:s3:::my-bucket/subfolder/*"
    }
  ]
}