Problem

When you try to write a dataset with an external path, your job fails. 

Query

CREATE TABLE IF NOT EXISTS <database-name>.<schema-name>.<table-name> USING DELTA LOCATION '{location}';

Error

UnityCatalogServiceException: [RequestId=934c88aa-3f54-4ed3-ba30-830265560ae6 ErrorClass=INVALID_PARAMETER_VALUE.INVALID_PARAMETER_VALUE] Input path s3://path/preview overlaps with other external tables or volumes. Conflicting tables/volumes: 

Cause

Two tables cannot be created in the same location, even across different catalogs. Either there is already an external table defined on the path, or the input path overlaps with other external tables or volumes. 

Solution

Clarify which tables or volumes are using which location on a selected schema. Identify the location of each table and ensure that you are not using the same path for multiple tables. 

from pyspark.sql.functions import col
df = spark.sql("SHOW tables").collect()
for row in df:
table = row["tableName"]
display(table)
table_detail =spark.sql("describe table extended {}".format (table))
table_detail = table_detail.filter(col("col_name") == "Location")
display(table_detail)

Optionally, to get a volume path, use the following command. 

DESCRIBE volume <volume-name>;

Then use the SHALLOW CLONE command to create a new table with the same schema and data as an existing table, but with a different location. 

For more information on managing external tables, review the External tables (AWS | Azure | GCP) documentation. 

For more information on using SHALLOW CLONE, refer to the Shallow clone for Unity Catalog tables (AWS | Azure | GCP) documentation.