Library installation attempted on the driver node of the cluster failed

Uninstall and reinstall the library.

Written by shashank.chaudhary

Last published at: January 17th, 2025

Problem

When attempting to install libraries from Unity Catalog (UC) volumes on a shared cluster, the process fails with the following error.

 

Library installation attempted on the driver node of the cluster failed. 
Please contact Databricks support to resolve the issue. Error Code: INVALID_ACCESS_TOKEN_ERROR. Error Message: java.util.concurrent.ExecutionException: org.apache.spark.sql.AnalysisException: 403: Invalid access token.

 

Cause

The original creator of the cluster no longer has workspace access, and in UC-shared clusters, Databricks validates library installation requests using the user’s identity token. As a result, any actions tied to the former user’s identity fail, including library installations.

 

Solution

  1. Uninstall the problematic library from the cluster where the issue occurred. 
  2. Have the new user (the one currently managing the cluster) reinstall the library. 

 

This ensures the library installation is performed using the current user’s identity token, which has valid permissions.