Access denied error when trying to read or write to an S3 bucket in a separate AWS account

Disable ACLs in the BucketOwnerEnforced setting in S3 Object Ownership, or ensure that access to the objects in the S3 bucket is permitted.

Written by alberto.umana

Last published at: October 24th, 2024

Problem

When attempting to access S3 buckets that are located in a different AWS account from where the bucket was originally created, you encounter an AccessDeniedException error.

Caused by: java.nio.file.AccessDeniedException: https://<bucket-name>.s3.us-east-1.amazonaws.com prod/_delta_log/00000000000000002581.json: getFileStatus on https://<bucket-name>.s3.us-east-1.amazonaws.com prod/_delta_log/00000000000000002581.json: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://<bucket-name>.s3.us-east-1.amazonaws.com prod/_delta_log/00000000000000002581.json {} Hadoop 3.3.4, aws-sdk-java/1.12.390 Linux/5.15.0-1070-aws OpenJDK_64-Bit_Server_VM/25.372-b07 java/1.8.0_372 scala/2.12.15 kotlin/1.6.0 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: <request-id-here>, Extended Request ID: <extended-request-id-here>, Cloud Provider: AWS, Instance ID: <instance-id-here> credentials-provider: com.amazonaws.auth.BasicSessionCredentials credential-header: <credential-header-here>
Credential=REDACTED_ACCESS_KEY(xxxxxxxx)/20241017/us-east-1/s3/aws4_request signature-present: true (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: <request-id-here>; S3 Extended Request ID: <extended-request-id-here>; Proxy: null), S3 Extended Request ID: <extended-request-id-here>:403 Forbidden

Cause

There are two possible causes. 

  • The IAM you’re using for read-write functions to the S3 bucket is from a different AWS account than the bucket owner account. When a Databricks cluster writes files to an S3 bucket, the BucketOwnerEnforced setting defaults to enabled.  
  • The S3 bucket’s ACL settings limit access to the objects in the bucket. 

Solution

Disable ACLs in the BucketOwnerEnforced setting in S3 Object Ownership. This turns off ACLs and the bucket owner account automatically owns and has full control over all objects in the bucket.

For more information, please review the AWS Disabling ACLs for all new buckets and enforcing Object Ownership documentation. 

If you require ACLs to remain enabled, make sure you also enable access to the objects in the S3 bucket for read-write access from other workspaces. For more detail, please review the AWS Controlling ownership of objects and disabling ACLs for your bucket documentation.