Configure a cluster to use a custom NTP server

By default Databricks clusters use public NTP servers. This is sufficient for most use cases, however you can configure a cluster to use a custom NTP server. This does not have to be a public NTP server. It can be a private NTP server under your control. A common use case is to minimize the amount of Internet traffic from your cluster.

Update the NTP configuration on a cluster

1. Create a ntp.conf file with the following information:

   # NTP configuration
   server <ntp-server-hostname> iburst
   

   where <ntp-server-hostname> is a NTP server hostname or a NTP server IP address.

   If you have multiple NTP servers to list, add them all to the file. Each server should be listed on its own line.

2. Upload the ntp.conf file to /dbfs/databricks/init_scripts/ on your cluster.

3. Create the script ntp.sh on your cluster:

   dbutils.fs.put("/databricks/init_scripts/ntp.sh","""
   #!/bin/bash
   echo "<ntp-server-ip> <ntp-server-hostname>" >> /etc/hosts
   cp /dbfs/databricks/init_scripts/ntp.conf /etc/
   sudo service ntp restart""",True)
   

4. Confirm that the script exists:

   display(dbutils.fs.ls("dbfs:/databricks/init_scripts/ntp.sh"))
   

5. Click Clusters, click your cluster name, click Edit, click Advanced Options, click Init Scripts.

6. Select DBFS under Destination.

7. Enter the full path to ntp.sh and click Add.

8. Click Confirm and Restart. A confirmation dialog box appears. Click Confirm and wait for the cluster to restart.

Verify the cluster is using the updated NTP configuration

1. Run the following code in a notebook:

   %sh ntpq -p
   

2. The output displays the NTP servers that are in use.