Problem
When creating a secret scope in Databricks, you encounter the following error on the specified Azure Key Vault.
“com.databricks.common.client.DatabricksServiceHttpClientException: PERMISSION_DENIED:Invalid permissions on the specified KeyVault https:/xxxxxxxxxx.vault.azure.net/.Wrapped Message: Status code 403, {"error":{"code":"Forbidden","message":"Client address is not authorized and caller was ignored because bypass is set to None\r\nClient address: <client-address>\r\nCaller: name=AzureDatabricks;appid=<app-id>;oid=<oid>\r\nVault: <nvault>;location=japaneast","innererror":{"code":"ForbiddenByFirewall"}}}”
Cause
The Databricks control plane uses the control plane network address translation (NAT) IP for communicating with external resources like Azure Key Vault. Azure Key Vault's firewall settings, however, do not allow the control plane NAT IP, resulting in a 403 Forbidden
error.
Additionally, accessing Azure Key Vault uses the control plane NAT IP even when secure cluster connectivity (SCC) is enabled. For more information, refer to the Enable secure cluster connectivity documentation.
Solution
Adjust the Azure Key Vault's firewall settings to either allow the Databricks control plane NAT IP or configure the Azure Key Vault to allow trusted Microsoft services to bypass the firewall.
Allow the control plane NAT IP
- Identify the control plane NAT IP address from the error message.
- Go to your Azure Key Vault resource in the Azure portal.
- Navigate to Firewalls and virtual networks.
- Under Allow access from:, select "Allow public access from specific virtual networks and IP addresses".
- Add the control plane NAT IP address to the list of allowed IP addresses.
Allow trusted Microsoft services to bypass the firewall
- Navigate to the Firewalls and virtual networks settings of your Azure Key Vault.
- Under Exception, check the box for "Allow trusted Microsoft services to bypass this firewall". This enables services like Azure Databricks to access the Key Vault even if their IP is not explicitly allowlisted.
For more information, refer to the “Configure your Azure Key Vault instance for Azure Databricks” section of the Secret Management documentation and the “Azure Databricks control plane addresses” section of the IP addresses and domains for Azure Databricks services and assets documentation.