AWS services fail with No region provided
error.
Problem
Your code snippets that use AWS services fail with a java.lang.IllegalArgumentException: No region provided
error in Databricks Runtime 7.0 and above. The same code worked in Databricks Runtime 6.6 and below.
You can verify the issue by running the example code snippet in a notebook. In Databricks Runtime 7.0 and above, it will return the exception shown in the example error message.
Example code snippet:
import com.amazonaws.regions.Regions
import com.amazonaws.regions.Regions
import com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClientBuilder
import com.amazonaws.services.securitytoken.AWSSecurityTokenService
val sts: AWSSecurityTokenService = AWSSecurityTokenServiceClientBuilder.standard()
.withRegion(Regions.fromName("us-west-2"))
.build()
Example error message:
java.lang.IllegalArgumentException: No region provided
at com.amazonaws.AmazonWebServiceClient.setRegion(AmazonWebServiceClient.java:509)
at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.setRegion(AWSSecurityTokenServiceClient.java:1436)
at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:456)
at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
Cause
There was a change in the AWS Java SDK version 1.11.655 which causes this issue in Databricks Runtime 7.0 and above.
Solution
You must set the sts_regional_endpoints value to regional
on the cluster. This ensures that the AWS STS endpoint for the currently configured region is used.
- Open the Clusters page.
- Select a cluster.
- Click Edit.
- Click Advanced Options.
- Click Spark.
- In the Environment Variables field, enter
AWS_STS_REGIONAL_ENDPOINTS="regional"
. - Save the change and start, or restart, the cluster.