Overlapping paths error when querying both Hive and Unity Catalog tables

If a Hive table and a Unity Catalog table both refer to the same external storage path you cannot query them in the same notebook cell.

Written by John.Lourdu

Last published at: January 20th, 2023

Problem

You are running queries when you get an overlapping paths error message.

org.apache.spark.SparkException: Your query is attempting to access overlapping paths through multiple authorization mechanisms, which is not currently supported.

Cause

An overlapping paths error happens when a single cell in a notebook queries both an Apache Hive table and a Unity Catalog table that both refer to the same external storage path.

Solution

Split your Hive table queries and Unity Catalog table queries into different cells.

Delete

Info

You can easily tell the difference between the two table types by the namespace notation used.

Hive tables use two-level namespace notation <database/schema_name>.<table_name>, while Unity Catalog table use a three-level namespace notation <catalog_name>.<database/schema_name>.<table_name>.