Problem
When attempting to access S3 buckets that are located in a different AWS account from where the bucket was originally created, you encounter an AccessDeniedException
error.
Caused by: java.nio.file.AccessDeniedException: https://<bucket-name>.s3.us-east-1.amazonaws.com prod/_delta_log/00000000000000002581.json: getFileStatus on https://<bucket-name>.s3.us-east-1.amazonaws.com prod/_delta_log/00000000000000002581.json: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://<bucket-name>.s3.us-east-1.amazonaws.com prod/_delta_log/00000000000000002581.json {} Hadoop 3.3.4, aws-sdk-java/1.12.390 Linux/5.15.0-1070-aws OpenJDK_64-Bit_Server_VM/25.372-b07 java/1.8.0_372 scala/2.12.15 kotlin/1.6.0 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: <request-id-here>, Extended Request ID: <extended-request-id-here>, Cloud Provider: AWS, Instance ID: <instance-id-here> credentials-provider: com.amazonaws.auth.BasicSessionCredentials credential-header: <credential-header-here>
Credential=REDACTED_ACCESS_KEY(xxxxxxxx)/20241017/us-east-1/s3/aws4_request signature-present: true (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: <request-id-here>; S3 Extended Request ID: <extended-request-id-here>; Proxy: null), S3 Extended Request ID: <extended-request-id-here>:403 Forbidden
Cause
There are two possible causes.
- The IAM you’re using for read-write functions to the S3 bucket is from a different AWS account than the bucket owner account. When a Databricks cluster writes files to an S3 bucket, the
BucketOwnerEnforced
setting defaults to enabled. - The S3 bucket’s ACL settings limit access to the objects in the bucket.
Solution
Disable ACLs in the BucketOwnerEnforced
setting in S3 Object Ownership. This turns off ACLs and the bucket owner account automatically owns and has full control over all objects in the bucket.
For more information, please review the AWS Disabling ACLs for all new buckets and enforcing Object Ownership documentation.
If you require ACLs to remain enabled, make sure you also enable access to the objects in the S3 bucket for read-write access from other workspaces. For more detail, please review the AWS Controlling ownership of objects and disabling ACLs for your bucket documentation.