Long jobs fail when accessing ADLS

Long running jobs that use Azure AD credential passthrough to access ADLS fail after 1 hour.

Written by huaming.liu

Last published at: December 9th, 2022

Problem

You are using Azure Active Directory (Azure AD) credential passthrough to access Azure Data Lake Storage (ADLS) resources.

Jobs that run longer than one hour fail with a HTTP401 error message.

com.microsoft.azure.datalake.store.ADLException: Error reading from file /local/Users/<path-to-file>

Operation OPEN failed with HTTP401 : null
Last encountered exception thrown after 5 tries. [HTTP401(null),HTTP401(null),HTTP401(null),HTTP401(null),HTTP401(null)]

Cause

The lifetime of an Azure AD passthrough token is one hour. When a command is sent to the cluster that takes longer than one hour, it fails if an ADLS resource is accessed after the one hour mark.

This is a known issue.

Solution

You must rewrite your queries, so that no single command takes longer than an hour to complete.

It is not possible to increase the lifetime of an Azure AD passthrough token. The token is retrieved by the Azure Databricks replicated principal. You cannot edit its properties.

Please review the ADLS credential passthrough limitations documentation for more information.