PERMISSION_DENIED error while running AutoML experiment with group-assigned cluster

Create a /Workspace/Groups/ folder for each group you plan to use with a group cluster to ensure correct permissions.

Written by priyanshi.david

Last published at: May 30th, 2025

Problem

While trying to create an AutoML experiment with a dedicated access mode cluster assigned to a group, you encounter the following error.

AutomlServiceError: Failed to create Automl experiment. Status Code: 403: Error: b'{"error_code": "PERMISSION_DENIED", "message": "unknown does not have View permissions on <folder>. Please contact the owner or an administrator for access."}’

 

Cause

When using a group cluster in Databricks, user permissions are scoped to the group. This means all actions on the cluster are performed using the group’s permissions, not the individual user’s. 

 

As a result, even if a user personally has access to a folder (like /Users/<username>), the cluster cannot access it unless the group also has permission. This mismatch causes the AutoML experiment to fail with a PERMISSION_DENIED error.

 

Solution

Creating a dedicated folder such as /Workspace/Groups/<group-name> and giving the group CAN MANAGE permissions ensures the group has full access to read from and write to that location.

  1. Create a /Workspace/Groups/<group-name> folder for the group you plan to use with the group cluster.
  2. Assign CAN MANAGE permissions on the folder to the group. 
  3. Add experiment_dir = “/Workspace/Groups/<group-name>” to your AutoML code.
  4. Make sure that whatever group you are using has the "Workspace Access" entitlement enabled.

 

For more information on best practices for managing group clusters, review the Assign compute resources to a group (AWS | Azure | GCP) documentation.