S3 connection fails with No role specified and no roles available

Problem

You are using Databricks Utilities (dbutils) to access a S3 bucket, but it fails with a No role specified and no roles available error.

You have confirmed that the instance profile associated with the cluster has the permissions needed to access the S3 bucket.

Unable to load AWS credentials from any provider in the chain: [com.databricks.backend.daemon.driver.aws.AwsLocalCredentialContextTokenProvider@ff3090c: No role specified and no roles available., com.databricks.backend.daemon.driver.aws.ProxiedIAMCredentialProvider@31f1245d: User does not have any IAM roles]

Cause

This can occur when both of the following items are true:

  • The cluster has IAM Role Passthrough enabled.
  • The cluster has an Instance Profile enabled.

When IAM Role Passthrough is enabled, every other authentication mechanism set at the cluster or notebook level is overwritten by IAM passthrough authentication.

Solution