Compute clusters get an authorization failure when cluster tags are used

If you use custom tags and instance pools, set the tags at the pool level.

Written by daniel.ruiz

Last published at: January 14th, 2025

Problem

You are launching a compute cluster that uses instance pools and one or more tags, when you encounter a tag policy violation error message. 

run failed with error message
Cluster '0604-122935-xxxxxx' was terminated. Reason: AWS_AUTHORIZATION_FAILURE (CLIENT_ERROR). Parameters: databricks_error_message:Failure happened when talking to AWS, AWS API error code: TagPolicyViolation AWS error message: The resource is missing the tag key(s) '<custom-tag>' specified by the tag policy., aws_api_error_code:TagPolicyViolation.

Cause

According to the Pool best practices documentation, “Tags from pools propagate to the underlying cloud provider instances, but the cluster’s tags do not.”

If you set custom tags on the cluster, and not the instance pool, the tags do not propagate to the cloud provider. As a result, you get a tag policy violation error on AWS.

Solution

If you want to use custom tags and instance pools, you must set the tags at the pool level.

If you do not use instance pools, you can set your tags at the cluster level.