Problem
Your code snippets that use AWS services fail with a java.lang.IllegalArgumentException: No region provided error in Databricks Runtime 7.0 and above. The same code worked in Databricks Runtime 6.6 and below.
You can verify the issue by running the example code snippet in a notebook. In Databricks Runtime 7.0 and above, it will return the exception shown in the example error message.
Example code snippet (Scala):
import com.amazonaws.regions.Regions import com.amazonaws.regions.Regions import com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClientBuilder import com.amazonaws.services.securitytoken.AWSSecurityTokenService val sts: AWSSecurityTokenService = AWSSecurityTokenServiceClientBuilder.standard() .withRegion(Regions.fromName("us-west-2")) .build()
Example error message:
java.lang.IllegalArgumentException: No region provided at com.amazonaws.AmazonWebServiceClient.setRegion(AmazonWebServiceClient.java:509) at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.setRegion(AWSSecurityTokenServiceClient.java:1436) at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:456) at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424) at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
Cause
There was a change in the AWS Java SDK version 1.11.655 which causes this issue in Databricks Runtime 7.0 and above.
Solution
You must set the sts_regional_endpoints value to regional on the cluster. This ensures that the AWS STS endpoint for the currently configured region is used.
- Open the Clusters page.
- Select a cluster.
- Click Edit.
- Click Advanced Options.
- Click Spark.
- In the Environment Variables field, enter AWS_STS_REGIONAL_ENDPOINTS="regional".
- Save the change and start, or restart, the cluster.