Cluster fails to start with InvalidGroup.NotFound error

If the network security group policy is not correctly configured your clusters will fail to start.

Written by Adam Pavlacka

Last published at: December 21st, 2023

 

Problem:

Your cluster fails to start with an error:

InvalidGroup.NotFound. The security group 'sg-XYZ' does not exist in VPC 'vpc-XYZ'

Cause:

The network security group policy is not correctly configured.

Solution:

  1. Contact your network engineering team to verify the security group policy is correctly associated with the Databricks workspace VPC.
  2. Ensure the security group sg-XYZ exists in the VPC vpc-XYZ.
  3. If the security group does not exist, create it using the appropriate console or API commands.
  4. If the security group exists, ensure it is correctly associated with the workspace VPC.
  5. If the security group is associated correctly, verify the inbound and outbound rules are configured to allow network traffic for the Databricks cluster.
  6. After updating the security group configuration, restart the cluster and verify that it launches correctly.