Non-admin users with 'CAN MANAGE' permissions on a specific DLT cluster can see the Spark UI but cannot view the driver logs

Set spark.databricks.acl.needAdminPermissionToViewLogs to false.

Written by kingshuk.das

Last published at: March 14th, 2025

Problem

You are a non-admin user with 'CAN MANAGE' permissions on a specific Delta Live Tables (DLT) pipeline’s cluster. You want to view the driver logs, but they don’t appear in the Apache Spark UI.  

 

Cause

By default, only the DLT pipeline owner and workspace admins have permission to view the cluster driver logs.

 

Solution

Set spark.databricks.acl.needAdminPermissionToViewLogs to false to enable non-admin users to view driver logs. 

 

Using the REST API

Use the Databricks REST API to edit the pipeline to include the configuration. Add the following configuration under the ‘configuration’ field in the API call. For details, review the Edit a pipeline API documentation.

{
 "configuration": {
  "spark.databricks.acl.needAdminPermissionToViewLogs": "false"
 }
}

 

 Note

The API is of PUT type. Please make sure to add ALL other existing configs while editing the pipeline.

 

 

Using the Databricks UI

  1. Navigate to your Databricks workspace. 
  2. Go to the Settings tab of the DLT pipeline.
  3. Scroll down to Advanced options and add the following configuration. spark.databricks.acl.needAdminPermissionToViewLogs false
  4. Save the changes and try again to view the driver logs.