When you launch an Azure Databricks cluster, you get an UnknownHostException error.
You may also get one of the following error messages:
- Error: There was an error in the network configuration. databricks_error_message: Could not access worker artifacts.
- Error: Temporary failure in name resolution.
- Internal error message: Failed to launch spark container on instance XXX. Exception: Could not add container for XXX with address X.X.X.X.mysql.database.azure.com: Temporary failure in name resolution.
These errors indicate an issue with DNS settings.
- Primary DNS could be down or unresponsive.
- Artifacts are not being resolved, which results in the cluster launch failure.
- You may have a host record listing the artifact public IP as static, but it has changed.
Identify a working DNS server and update the DNS entry on the cluster.
- Start a standalone Azure VM and verify that the artifacts blob storage account is reachable from the instance.
`telnet dbartifactsprodeastus.blob.core.windows.net 443`.
- Verify that you can reach your primary DNS server from a notebook by running a ping command.
- If your DNS server is not responding, try to reach your secondary DNS server from a notebook by running a ping command.
- Launch a Web Terminal from the cluster workspace.
- Edit the /etc/resolv.conf file on the cluster.
- Update the nameserver value with your working DNS server.
- Save the changes to the file.
- Restart systemd-resolved.
$ sudo systemctl restart systemd-resolved.service
If you are still having DNS issues, you should try the following steps:
- Verify that port 43 (used for whois) and port 53 (used for DNS) are open in your firewall.
- Add the Azure recursive resolver (168,.63.129.16) to the default DNS forwarder. Review the VMs and role instances documentation for more information.
- Verify that nslookup results are identical between your laptop and the default DNS. If there is a mistmatch, your DNS server may have an incorrect host record.
- Verify that everything works with a default Azure DNS server. If it works with Azure DNS, but fails with your custom DNS, your DNS admin should review your DNS server settings.