AWS services fail with No region provided error.

Problem

Your code snippets that use AWS services fail with a java.lang.IllegalArgumentException: No region provided error in Databricks Runtime 7.0 and above. The same code worked in Databricks Runtime 6.6 and below.

You can verify the issue by running the example code snippet in a notebook. In Databricks Runtime 7.0 and above, it will return the exception shown in the example error message.

Example code snippet:

import com.amazonaws.regions.Regions
import com.amazonaws.regions.Regions
import com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClientBuilder
import com.amazonaws.services.securitytoken.AWSSecurityTokenService


 val sts: AWSSecurityTokenService = AWSSecurityTokenServiceClientBuilder.standard()
.withRegion(Regions.fromName("us-west-2"))
.build()

Example error message:

java.lang.IllegalArgumentException: No region provided
        at com.amazonaws.AmazonWebServiceClient.setRegion(AmazonWebServiceClient.java:509)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.setRegion(AWSSecurityTokenServiceClient.java:1436)
        at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:456)
        at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
        at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)

Cause

There was a change in the AWS Java SDK version 1.11.655 which causes this issue in Databricks Runtime 7.0 and above.

Solution

You must set the sts_regional_endpoints value to regional on the cluster. This ensures that the AWS STS endpoint for the currently configured region is used.

  1. Open the Clusters page.
  2. Select a cluster.
  3. Click Edit.
  4. Click Advanced Options.
  5. Click Spark.
  6. In the Environment Variables field, enter AWS_STS_REGIONAL_ENDPOINTS="regional".
  7. Save the change and start, or restart, the cluster.