Problem

You are unable to deploy interactive/job clusters or classic/pro SQL warehouses in AWS due to a SECURITY_DAEMON_REGISTRATION_EXCEPTION error.

Cluster '1011-202716-ix9g3s7k' was terminated. Reason: SECURITY_DAEMON_REGISTRATION_EXCEPTION (SERVICE_FAULT). Parameters: instance_id:i-0e9b63a71d38c2645, databricks_error_message:Failed to set up the Spark container due to an error when registering the container to security daemon.

Cause

The VPC being used is missing an STS interface endpoint for the Databricks subnets. For typical use cases this endpoint is required. Clusters and other compute resources in the classic compute plane use it to connect to AWS native services.

Solution

You must add an interface endpoint to your customer managed VPC.

Follow the instructions in the Access an AWS service using an interface VPC endpoint documentation to create the endpoint.

After you have created the endpoint, you can use it for AWS STS. For more information, review the Create a VPC endpoint for AWS STS documentation.

After the endpoint is up-and-running, you can deploy your clusters.