You can use an Azure Firewall to create a VNet-injected workspace in which all clusters have a single IP outbound address. The single IP address can be used as an additional security layer with other Azure services and applications that allow access based on specific IP addresses.

1. Set up an Azure Databricks Workspace in your own virtual network.

2. Set up a firewall within the virtual network. See Create an NVA. When you create the firewall, you should:

• Note both the private and public IP addresses for the firewall for later use.
• Create a network rule for the public subnet to forward all traffic to the internet:
  • Name: any arbitrary name
  • Priority: 100
  • Protocol: Any
  • Source Addresses: IP range for the public subnet in the virtual network that you created
  • Destination Addresses: 0.0.0.0/1
  • Destination Ports: *

3. Create a Custom Route Table and associate it with the public subnet.

a. Add custom routes, also known as user-defined routes (UDR) for the following services. Specify the Azure Databricks region addresses for your region. For Next hop type, enter Internet, as shown in creating a route table.

• Control Plane NAT VIP
• Webapp
• Metastore
• Artifact Blob Storage
• Logs Blob Storage

b. Add a custom route for the firewall with the following values:

• Address prefix: 0.0.0.0./0
• Next hop type: Virtual appliance
• Next hop address: The private IP address for the firewall.

c. Associate the route table with the public subnet.

4. Validate the setup

• Create a cluster in the Azure Databricks workspace.
• Next, query blob storage to your own paths or run %fs ls in a cell.
• If it fails, confirm that the route table has all required UDRs (including Service Endpoint instead of the UDR for Blob Storage)

For more information, see Route Azure Databricks traffic using a virtual appliance or firewall.