DLT pipeline fails with error DLT ERROR CODE: EXECUTION_SERVICE_STARTUP_FAILURE

Use the dbutils.fs.refreshMounts() command in the DLT code.

Written by alberto.umana

Last published at: November 17th, 2024

Problem

When using a recently rotated, deleted or expired service principal secret with Auto Loader or Delta Live Tables pipelines, you encounter an error message. 

com.databricks.pipelines.common.CustomException: [DLT ERROR CODE: EXECUTION_SERVICE_STARTUP_FAILURE] HTTP Error 401; url='https://login.microsoftonline.com//oauth2/token' AADToken: HTTP connection to <https://login.microsoftonline.com//oauth2/token> failed for getting token from AzureAD.; requestId=''; contentType='application/json; charset=utf-8'; response '{"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app ''."

 

Cause

The token was renewed or rotated when rotating, deleting, or expiring a service principal secret. This causes authentication failure with Microsoft Entra ID (Azure Active Directory). 

 

Solution

  1. Generate a new Microsoft Entra ID service principal secret in the Azure portal or Azure CLI.
  2. Update the secret in the Databricks secret scope using the Databricks CLI or UI.
  3. Unmount and remount the Azure storage mount points in the Databricks workspace using the new secret, otherwise updated secret will not be picked up.
  4. Use the dbutils.fs.refreshMounts() command in the DLT code to force the DLT cluster to pick up the latest configuration.

 

For more information, review the Connect to Azure Data Lake Storage Gen2 and Blob Storage and Mounting cloud object storage on Azure Databricks documentation.