Problem

When you try to use an existing network security group (NSG) to deploy a Databricks workspace in Azure using Terraform, Terraform fails with the following error. 

Failed to prepare subnet '<subnet-name>'. Please try again later. Error details: 'Gateway authentication failed for 'Microsoft.Network'. Diagnostic information: timestamp <timestamp>', tracking id '<tracking-id>', request correlation id '<request-correlation-id>.''

Cause

Your NSG is missing a configuration which permits communication over port range 8443-8451 and port 3306, or you have incorrect Azure Resource tags causing conflicts with your Azure tagging policies. For more information, refer to the Assign policy definitions for tag compliance documentation.

Solution

Verify your existing NSG has port range 8443-8451 and port 3306 open for communication. It is possible that the NSG was created using a template that did not include these ports by default. For more information, refer to the “Troubleshooting” section of the Deploy Azure Databricks in your Azure virtual network (VNet injection) documentation.

If the network security group is correctly configured to allow communication on these ports, review the Azure Resource tags. Validate that none of the tags cause conflicts with your Azure policies, such as an additional space at the end of the tag name.