Configure custom DNS settings using dnsmasq

Learn how to configure custom DNS settings using dnsmasq.

Written by brian.sears

Last published at: May 10th, 2023

dnsmasq is a tool for installing and configuring DNS routing rules for cluster nodes. You can use it to set up routing between your Databricks environment and your on-premise network.

Delete

Warning

If you use your own DNS server and it goes down, you will experience an outage and will not be able to create clusters.

Use the following cluster-scoped init script to configure dnsmasq for a cluster node.

  1. Use netcat (nc) to test connectivity from the notebook environment to your on-premise network.
    nc -vz <on-premise-ip> 53
  2. Create the base directory you want to store the init script in if it does not already exist.
    dbutils.fs.mkdirs("dbfs:/databricks/<init-script-folder>/")
  3. Create the script.

    AWS Scala example

    dbutils.fs.put("/databricks/<init-script-folder>/dns-masq.sh","""
    #!/bin/bash
    ########################################
    Configure on-prem dns access.
    ########################################
    
    sudo apt-get update -y
    sudo apt-get install dnsmasq -y --force-yes
    
    ## Add dns entries for internal your-company.net name servers
    echo server=/databricks.net/<dns-server-ip> | sudo tee --append /etc/dnsmasq.conf
    
    ## Find the default DNS settings for the EC2 instance and use them as the default DNS route
    
    ec2_dns=cat /etc/resolv.conf | grep "nameserver"; | cut -d' ' -f 2
    echo "Old dns in resolv.conf $ec2_dns"
    
    echo "server=$ec2_dns" | sudo tee --append /etc/dnsmasq.conf
    
    ## configure resolv.conf to point to dnsmasq service instead of static resolv.conf file
    mv /etc/resolv.conf /etc/resolv.conf.orig
    echo nameserver 127.0.0.1 | sudo tee --append /etc/resolv.conf
    sudo systemctl disable --now systemd-resolved
    sudo systemctl enable --now dnsmasq
    """, true)
    Delete

    Azure Scala example

    dbutils.fs.put("/databricks/<init-script-folder>/dns-masq.sh","""
    #!/bin/bash
    sudo apt-get update -y
    sudo apt-get install dnsmasq -y --force-yes
    
    ## Add dns entries for internal nameservers
    echo server=/databricks.net/<dns-server-ip> | sudo tee --append /etc/dnsmasq.conf
       
    ## Find the default DNS settings for the instance and use them as the default DNS route
    azvm_dns=cat /etc/resolv.conf | grep "nameserver"; | cut -d' ' -f 2
    echo "Old dns in resolv.conf $azvm_dns"
    echo "server=$azvm_dns" | sudo tee --append /etc/dnsmasq.conf
        
    ## configure resolv.conf to point to dnsmasq service instead of static resolv.conf file
    mv /etc/resolv.conf /etc/resolv.conf.orig
    echo nameserver 127.0.0.1 | sudo tee --append /etc/resolv.conf
    sudo systemctl disable --now systemd-resolved
    sudo systemctl enable --now dnsmasq
    """, true)
    Delete
  4. Check that the script exists.
    display(dbutils.fs.ls("dbfs:/databricks/<init-script-folder>/dns-masq.sh"))
  5. Configure the init script that you just created as a cluster-scoped init script. You will need the full path to the location of the script (dbfs:/databricks/<init-script-folder>/dns-masq.sh).
  6. Launch a zero-node cluster to confirm that you can create clusters.