Troubleshoot key vault access issues

Troubleshoot Azure key vault access issues. Verify firewall. Enable secrets.

Written by arvind.ravish

Last published at: May 17th, 2022

You are trying to access secrets, when you get an error message.

com.databricks.common.client.DatabricksServiceException: INVALID_STATE: Databricks could not access keyvault: https://xxxxxxx.vault.azure.net/.

There is not a single root cause for this error message, so you will have to do some troubleshooting.

Confirm permissions are correctly set on the key vault

  1. Load the Azure Portal.
  2. Open Key vaults.
  3. Click the key vault.
  4. Click Access policies.
  5. Verify the Get and List permissions are applied.

Inspect the firewall configuration on the key vault

  1. Load the Azure Portal.
  2. Open Key vaults.
  3. Click the key vault.
  4. Click Networking.
  5. Click Firewalls and virtual networks.
  6. Select Private endpoint and selected networks.
  7. Verify that Allow trusted Microsoft services to bypass this firewall? is set to Yes.
  8. Attempt to access the secrets.

Azure key vault firewall configuration screen.

If you can view the secrets, the issue is resolved.

If you are still getting the INVALID_STATE: Databricks could not access keyvault error, continue troubleshooting.

List all secrets in the secret scope

  1. Open a notebook.
  2. List all secrets in scope.
%python

dbutils.secrets.list("<scopename>")

Try to access individual secrets

Try to access a few different, random secrets.

%python

dbutils.secrets.get("<KeyvaultSecretScope>", "<SecretName>")

If some secrets can be fetched, while others fail, the failed secrets are either disabled or inactive.

Enable individual secrets

  1. Load the Azure Portal.
  2. Open Key vaults.
  3. Click the key vault.
  4. Click Secrets.
  5. Click the secret and verify that the status is set to Enabled.

Azure key vault secret enabled.

If the secret is disabled, enable it, or create a new version.

Verify that individual secrets are working

Try to access the previously failed secrets.

%python

dbutils.secrets.get("<KeyvaultSecretScope>", "<SecretName>")

You can fetch all of them.