Problem
You are using Databricks Utilities (dbutils) to access a S3 bucket, but it fails with a No role specified and no roles available error.
You have confirmed that the instance profile associated with the cluster has the permissions needed to access the S3 bucket.
Unable to load AWS credentials from any provider in the chain: [com.databricks.backend.daemon.driver.aws.AwsLocalCredentialContextTokenProvider@ff3090c: No role specified and no roles available., com.databricks.backend.daemon.driver.aws.ProxiedIAMCredentialProvider@31f1245d: User does not have any IAM roles]
Cause
This can occur when both of the following items are true:
- The cluster has IAM Role Passthrough enabled.
- The cluster has an Instance Profile enabled.
When IAM Role Passthrough is enabled, every other authentication mechanism set at the cluster or notebook level is overwritten by IAM passthrough authentication.
Solution
- When using a regular instance profile, make sure that IAM Role Passthrough is disabled.
- Review the documentation on secure access to S3 buckets using instance profiles to ensure that the correct IAM role is configured.
- When using the IAM Role Passthrough feature, make sure you have correctly configured the meta instance profile. For more information, review the documentation on accessing S3 buckets using IAM credential passthrough with Databricks SCIM.