dnsmasq is a tool for installing and configuring DNS routing rules for cluster nodes. You can use it to set up routing between your Databricks environment and your on-premise network.
Use the following cluster-scoped init script to configure dnsmasq for a cluster node.
- Use netcat (nc) to test connectivity from the notebook environment to your on-premise network.
nc -vz <on-premise-ip> 53
- Create the base directory you want to store the init script in if it does not already exist.
dbutils.fs.mkdirs("dbfs:/databricks/<init-script-folder>/")
- Create the script.
AWS Scala example
dbutils.fs.put("/databricks/<init-script-folder>/dns-masq.sh",""" #!/bin/bash ######################################## Configure on-prem dns access. ######################################## sudo apt-get update -y sudo apt-get install dnsmasq -y --force-yes ## Add dns entries for internal your-company.net name servers echo server=/databricks.net/<dns-server-ip> | sudo tee --append /etc/dnsmasq.conf ## Find the default DNS settings for the EC2 instance and use them as the default DNS route ec2_dns=cat /etc/resolv.conf | grep "nameserver"; | cut -d' ' -f 2 echo "Old dns in resolv.conf $ec2_dns" echo "server=$ec2_dns" | sudo tee --append /etc/dnsmasq.conf ## configure resolv.conf to point to dnsmasq service instead of static resolv.conf file mv /etc/resolv.conf /etc/resolv.conf.orig echo nameserver 127.0.0.1 | sudo tee --append /etc/resolv.conf sudo systemctl disable --now systemd-resolved sudo systemctl enable --now dnsmasq """, true)
DeleteAzure Scala example
dbutils.fs.put("/databricks/<init-script-folder>/dns-masq.sh",""" #!/bin/bash sudo apt-get update -y sudo apt-get install dnsmasq -y --force-yes ## Add dns entries for internal nameservers echo server=/databricks.net/<dns-server-ip> | sudo tee --append /etc/dnsmasq.conf ## Find the default DNS settings for the instance and use them as the default DNS route azvm_dns=cat /etc/resolv.conf | grep "nameserver"; | cut -d' ' -f 2 echo "Old dns in resolv.conf $azvm_dns" echo "server=$azvm_dns" | sudo tee --append /etc/dnsmasq.conf ## configure resolv.conf to point to dnsmasq service instead of static resolv.conf file mv /etc/resolv.conf /etc/resolv.conf.orig echo nameserver 127.0.0.1 | sudo tee --append /etc/resolv.conf sudo systemctl disable --now systemd-resolved sudo systemctl enable --now dnsmasq """, true)
Delete - Check that the script exists.
display(dbutils.fs.ls("dbfs:/databricks/<init-script-folder>/dns-masq.sh"))
- Configure the init script that you just created as a cluster-scoped init script. You will need the full path to the location of the script (dbfs:/databricks/<init-script-folder>/dns-masq.sh).
- Launch a zero-node cluster to confirm that you can create clusters.