Problem

When you save your subnet information, you store it locally. When you later try to retrieve the subnet information again, you encounter an authorization error.

This Azure storage request is not authorized. The storage account's 'Firewalls and virtual networks' settings may be blocking access to storage services. Please verify your Azure storage credentials or firewall exception settings.

Cause

Subnet information is dynamic because its serverless. Storing subnet information locally creates a static version, which becomes outdated when you try to use it later.

Solution

Use a script that leverages the Network Configuration Controller (NCC) API to fetch subnet information dynamically, and then allowlist the subnets. This approach integrates with serverless and automates Azure Data Lake Storage (ADLS) firewall allowlisting.

1. Create an NCC object.
   • Endpoint: POST https://accounts.azuredatabricks.net/api/2.0/accounts/{{accountId}}/network-connectivity-configs
   • Permission: Databricks Account Admin
   • Payload: name (<string>), region (<string>)
2. Attach the NCC object to one or more workspaces.
   • Endpoint: PATCH https://accounts.azuredatabricks.net/api/2.0/accounts/{{accountId}}/workspaces/{{<workspace-id>}}
   • Permission: Databricks Account Admin
   • Payload: network_connectivity_config_id (<string>)
3. Obtain the NCC object details and record the subnet IDs.
   • Endpoint: GET https://accounts.azuredatabricks.net/api/2.0/accounts/{{accountId}}/workspaces/{{workspaceId}}/network-connectivity-configs
   • Permission: Databricks Workspace Admin
4. Add the subnet IDs to the firewall of the Azure storage accounts.
   • Endpoint: (Azure CLI command) az storage account network-rule add 
   • Parameters: --resource-group, --account-name, --subscription, --subnet
5. Verify connectivity.
   • No API endpoint required. Verify by creating a serverless cluster and accessing the desired storage accounts.