Problem

You are using dbutils to access an external location (AWS | Azure | GCP) that is mounted on managed tables in a shared cluster. When you try to list the path to the location, it fails with an INVALID_PARAMETER_VALUE.LOCATION_OVERLAP error message. 

The error says the given path overlaps with managed storage.

dbutils.fs.ls("<storage-blob>://path/")

AnalysisException: [RequestId=96dd6185-e0dc-4fe0-94ad-bd8ab05fbd8e ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url '<storage-blob>://path' overlaps with managed storage

Cause

Running the list command on a managed directory is not supported in Unity Catalog. Catalog/schema storage locations are reserved for managed storage.

Solution

External tables cannot overlap with catalog/schema storage locations, but they can be created under a subdirectory of the root location. You should not create an external table at or above the root location used for the catalog/schema.

For example, assume the root location is <storage-blob>://<some-root>. The corresponding catalog/schema location is equivalent to the managed storage location which is <storage-blob>://<some-root>/__unitystorage/catalogs/<catalog-id>.

You can create an external location under some-root/ as long as it does not overlap the managed table. Given the example, <storage-blob>://<some-root>/<some-path>/<external-table-path> is a valid path for an external location.

If you try to list the contents of this example location, the result would be successful.