Problem

You are unable to launch clusters in an Azure VNET injected Databricks workspace when NAT gateway is configured.

Invalid argument: Cannot launch the cluster because the user specified an invalid argument. Internal error message: NAT Gateway /subscriptions/.../resourceGroups/. .../providers/Microsoft.Network/natGateways/... cannot be deployed on subnet containing Basic SKU Public IP addresses or Basic SKU Load Balancer.

NIC /subscriptions/.../resourceGroups/.../providers/Microsoft.Network/networkInterfaces/...-publicNIC/ipConfigurations/ipConfig in subnet /subscriptions/.../resourceGroups/npdac-rg/providers/Microsoft.Network/virtualNetworks/npdac-vnet/subnets/... has reference to Basic SKU Public IP address or Load Balancer /subscriptions/.../resourceGroups/.../providers/Microsoft.Network/publicIPAddresses/....

Cause

Secure cluster connectivity (also known as no public IP or NPIP) is not enabled on the workspace. This results in a conflict as the NAT gateway can not be placed on subnets containing Basic SKU public IP addresses or Basic SKU load balancer configurations. This is a documented NAT gateway limitation.

Solution

You should Enable secure cluster connectivity on your workspace. After secure cluster connectivity is enabled, the workspace functions normally.

For more information, review the Egress with VNet injection documentation.

If you cannot use secure cluster connectivity for some reason, you should configure the workspace to use a single IP and align the firewall settings.

For more information, review the Assign a single public IP for VNet-injected workspaces using Azure Firewall KB article.