Configure a cluster to use a custom NTP server

Configure your clusters to use a custom NTP server (public or private) instead of using the default server.

Written by xin.wang

Last published at: December 8th, 2022

By default Databricks clusters use public NTP servers. This is sufficient for most use cases, however you can configure a cluster to use a custom NTP server. This does not have to be a public NTP server. It can be a private NTP server under your control. A common use case is to minimize the amount of Internet traffic from your cluster.

Update the NTP configuration on a cluster

  1. Create a ntp.conffile with the following information:
    # NTP configuration
    server <ntp-server-hostname> iburst
    where <ntp-server-hostname> is a NTP server hostname or a NTP server IP address.
    If you have multiple NTP servers to list, add them all to the file. Each server should be listed on its own line.
  2. Upload the ntp.conf file to /dbfs/databricks/init_scripts/ on your cluster.
  3. Create the script ntp.shon your cluster:
    %python
    
    dbutils.fs.put("/databricks/init_scripts/ntp.sh","""
    #!/bin/bash
    echo "<ntp-server-ip> <ntp-server-hostname>" >> /etc/hosts
    cp /dbfs/databricks/init_scripts/ntp.conf /etc/
    sudo service ntp restart""",True)
  4. Confirm that the script exists:
    %python
    
    display(dbutils.fs.ls("dbfs:/databricks/init_scripts/ntp.sh"))
  5. Click Clusters, click your cluster name, click Edit, click Advanced Options, click Init Scripts.
  6. Select DBFS under Destination.
  7. Enter the full path to ntp.sh and click Add.
  8. Click Confirm and Restart. A confirmation dialog box appears. Click Confirm and wait for the cluster to restart.

Verify the cluster is using the updated NTP configuration

  1. Run the following code in a notebook:
    %sh ntpq -p
  2. The output displays the NTP servers that are in use.
Was this article helpful?