You want to use IAM roles when table ACLs are enabled, but you get an error saying credentials cannot be located.
NoCredentialsError: Unable to locate credentials
When a table ACL is enabled, access to the EC2 instance metadata service is blocked.
This is a security measure that prevents users from obtaining IAM access credentials.
You can explicitly provide AWS credentials in your notebook by using boto3.
You need to supply the values for
import logging import boto3 from botocore.exceptions import ClientError # Get a list objects in bucket try: s3 = boto3.client( 's3', aws_access_key_id='<access-key-id>', aws_secret_access_key='<secret-access-key>' ) response = s3.list_objects_v2(Bucket='<aws-bucket-name>') # Output the object in the bucket keys =  print('Object existing in bucket:') for obj in response['Contents']: keys.append(obj['Key']) print(keys) except ClientError as e: logging.error(e)
You should not disable process isolation in your Spark Config as this can be a security risk.