Problem:

Your cluster fails to start with an error:

InvalidGroup.NotFound. The security group 'sg-XYZ' does not exist in VPC 'vpc-XYZ'

Cause:

The network security group policy is not correctly configured.

Solution:

1. Contact your network engineering team to verify the security group policy is correctly associated with the Databricks workspace VPC.
2. Ensure the security group sg-XYZ exists in the VPC vpc-XYZ.
3. If the security group does not exist, create it using the appropriate console or API commands.
4. If the security group exists, ensure it is correctly associated with the workspace VPC.
5. If the security group is associated correctly, verify the inbound and outbound rules are configured to allow network traffic for the Databricks cluster.
6. After updating the security group configuration, restart the cluster and verify that it launches correctly.