Problem

When trying to query a federated table using SQL serverless compute, you receive an error message. 

com.databricks.backend.common.rpc.DatabricksDatabaseException: org.apache.spark.sql.AnalysisException: Table or view not found: Schema.Table;
Caused by: org.apache.spark.SparkException: com.microsoft.sqlserver.jdbc.SQLServerException: Reason: An instance-specific error occurred while establishing a connection to SQL Server. Connection was denied since Deny Public Network Access is set to Yes (https://docs.microsoft.com/azure/azure-sql/database/connectivity-settings#deny-public-network-access). To connect to this server, use the Private Endpoint from inside your virtual network (https://docs.microsoft.com/azure/sql-database/sql-database-private-endpoint-overview#how-to-set-up-private-link-for-azure-sql-database). ClientConnectionId:XXXXXX

Cause

You have an NCC private endpoint with public network access disabled. When public network access is disabled, the serverless control plane is not allowed to access the SQL server. Only connections from private endpoints are allowed. 

Solution

Explicitly set up private connectivity for serverless.

First, validate connection status using serverless to your SQL server. Run the following command in a notebook.

%sh
nc -vz DB_NAME-prod.mysql.database.azure.com:3306

If the serverless connection is not allowed, the following message is returned. 

ConnectException: Connection timed out (Connection timed out)

Then, set up NCC connectivity for the serverless environment using the Microsoft Configure private connectivity from serverless compute documentation.

For more information, refer to the Microsoft Enable Azure Private Link as a simplified deployment documentation. For more information particularly about denying public network access, refer to the Microsoft Connectivity settings for Azure SQL Database and Azure Synapse Analytics documentation.