How to Configure Custom DNS Settings Using dnsmasq

dnsmasq is a tool for installing and configuring DNS routing rules for cluster nodes. You can use it to set up routing between your Databricks environment and your on-premise network.

Warning

If you use your own DNS server and it goes down, you will experience an outage and will not be able to create clusters.

Use the following cluster-scoped init script to configure dnsmasq for a cluster node.

  1. Use netcat (nc) to test connectivity from the notebook environment to your on-premise network.

    %sh
    # test DNS connectivity from a notebook
    nc -vz 10.X.Y.Z 53
    
  2. Create the base directory you want to store the init script in if it does not already exist.

    dbutils.fs.mkdirs("dbfs:/databricks/<directory>/")
    
  3. Create the script.

    %scala
    dbutils.fs.put("/databricks/<directory>/dns-masq.sh";,"""
    #!/bin/bash
    ########################################
    Configure on-prem dns access.
    ########################################
    
    sudo apt-get update -y
    sudo apt-get install dnsmasq -y --force-yes
    
    ## Add dns entries for internal your-company.net name servers
    echo server=/databricks.net/10.X.Y.Z | sudo tee --append /etc/dnsmasq.conf
    
    ## Find the default DNS settings for the EC2 instance and use them as the default DNS route
    
    ec2_dns=cat /etc/resolv.conf | grep "nameserver"; | cut -d' ' -f 2
    echo "Old dns in resolv.conf $ec2_dns"
    
    echo "server=$ec2_dns" | sudo tee --append /etc/dnsmasq.conf
    
    
    ## configure resolv.conf to point to dnsmasq service instead of static resolv.conf file
    mv /etc/resolv.conf /etc/resolv.conf.orig
    echo nameserver 127.0.0.1 | sudo tee --append /etc/resolv.conf
    sudo service dnsmasq restart
    """, true)
    
  4. Check that the script exists.

    display(dbutils.fs.ls("dbfs:/databricks/<directory>/dns-masq.sh"))
    
  5. Configure the cluster to run the script.

  6. Launch a zero-node cluster to confirm that you can create clusters.