Cannot use IAM roles with table ACL

You cannot obtain IAM credentials when table ACLs are enabled because access to the EC2 instance metadata service is blocked.

Written by Adam Pavlacka

Last published at: May 16th, 2022

Problem

You want to use IAM roles when table ACLs are enabled, but you get an error saying credentials cannot be located.

NoCredentialsError: Unable to locate credentials

Cause

When a table ACL is enabled, access to the EC2 instance metadata service is blocked.

This is a security measure that prevents users from obtaining IAM access credentials.

Solution

You can explicitly provide AWS credentials in your notebook by using boto3.

You need to supply the values for aws_access_key_id and aws_secret_access_key.

%python

import logging
import boto3
from botocore.exceptions import ClientError

# Get a list objects in bucket
try:
  s3 = boto3.client(
    's3',
    aws_access_key_id='<access-key-id>',
    aws_secret_access_key='<secret-access-key>'
  )

  response = s3.list_objects_v2(Bucket='<aws-bucket-name>')

  # Output the object in the bucket
  keys = []
  print('Object existing in bucket:')
  for obj in response['Contents']:
    keys.append(obj['Key'])
  print(keys)
except ClientError as e:
  logging.error(e)
Delete

Warning

You should not disable process isolation in your Spark config as this can be a security risk.