Problem

When writing cluster delivery logs in the configured log delivery S3 bucket, you get a 403 AccessDenied error.

AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request
ID: XXXXXXXX; S3 Extended Request ID: XXXXXXXXXXX; Proxy: null)

Cause

Your S3 bucket has a policy that enforces encryption requirements. 

The default server-side encryption for sending the logs to S3 bucket is SSE-S3 unless the encryption type on the bucket has been changed. For example, a policy only allows data to be written if it’s encrypted using server-side encryption with AWS Key Management Service (SSE-KMS) with a specific KMS key. The policy denies writes if any other type of encryption is used.

Solution

Apply KMS encryption to cluster logs in S3. Use the Databricks API to create a new cluster or update a cluster configuration. 

For more information on cluster_log_conf, review the Create new cluster API documentation.

Example JSON

"cluster_log_conf": {
        "s3": {
            "destination": "s3://test-bucket-for-dbricks/logs/",
            "region": "us-east-1",
            "enable_encryption": true,
            "encryption_type": "sse-kms",
            "kms_key": "arn:aws:kms:us-east-1:12345678901:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
            "canned_acl": "bucket-owner-full-control"
        }