Not found 404 errors when selecting system tables

Update DNS resolution to reach the current and correct storage endpoint.

Written by jeremy.ramirez

Last published at: March 22nd, 2025

Problem

You are using custom DNS in your Databricks data plane environment (VNet Azure and VPC for AWS) and are trying to select information from system tables when you get a 404 error message. You may also have seen unknown host exceptions or connection timeout errors.

 

Example error

FileReadException: Error while reading file delta-sharing:/…. Caused by: IOException: java.util.concurrent.ExecutionException: io.delta.sharing.spark.util.UnexpectedHttpStatus: HTTP request failed with status: HTTP/1.1 404 The specified resource does not exist. {"error":{"code":"ResourceNotFound","message":"The specified resource does not exist.\nRequestId:00000000-0000-0000-0000-000000000000\nTime:0000-00-00T00:00:00.00000000"}}, while accessing URI of shared table file Caused by: ExecutionException: io.delta.sharing.spark.util.UnexpectedHttpStatus: HTTP request failed with status: HTTP/1.1 404 The specified resource does not exist.

 

Info

Unknown host exceptions or connection timeouts can occur if the rotated IP address is not assigned to any storage endpoint when it fails. The 404 error message occurs when the previous IP address is rotated to a storage endpoint on a different region.

 

 

Cause

Your custom DNS was mapped to a static IP address for Databricks artifact storage that was also used by our system tables. Databricks public IP addresses can rotate within the range specified, depending on the cloud provider.

Review IP addresses and domains for Databricks services and assets (AWS | Azure) for your cloud provider and region.

 

Solution

The short term workaround is to update your custom DNS to the current IP address or CNAME of the system tables endpoint of your region. You can get this IP address with a serverless cluster, or any other machine connected to a public DNS server using a nslookup command.

You need to modify the domain depending on your cloud and region used.

  • AWS: nslookup system-tables-prod-<your-region>-uc-metastore-bucket.s3.<your-region>.amazonaws.com
  • Azure: nslookup <your-region>.dfs.core.windows.net

Databricks recommends using a DNS conditional forwarder to reference the updated domain instead of using static IP addresses with custom DNS.