Shell command %sh ls does not work on DBFS files or directories when using a shared cluster

Use a single access mode cluster, dbutils.fs.ls, or migrate to volumes.

Written by alberto.umana

Last published at: February 26th, 2025

Problem

You try to perform the %sh ls shell command on a Databricks File System (DBFS) file or directory in your Unity Catalog-enabled workspace using a shared cluster. You grant SELECT permission on any file. Despite the permission, you receive the following error message.

 

ls: cannot access '/dbfs/{path}/': No such file or directory

 

Cause

Shared access mode has stricter access controls and limitations that affect file system operations. Specifically, shared access mode does not support FUSE for DBFS root and mounts, which means that direct file system operations using shell commands like %sh ls do not work. 

For details, review the Compute access mode limitations for Unity Catalog (AWS | Azure | GCP) documentation.

 

Solution

There are three options available. 

  1. Use single access mode. Single access mode allows you to perform file system operations using shell commands like %sh ls.
  2. Use dbutils.fs.ls instead of %sh ls. This command is designed to work with DBFS in Databricks and allows you to list the contents of a DBFS file in shared access mode.
  3. Migrate from DBFS mount points to volumes. Volumes allow you to perform file system operations using shell commands like %sh ls in shared access mode clusters. For more information, refer to the Volumes (AWS | Azure | GCP) documentation.