Problem

You attempt to use Iceberg external JARs to perform Iceberg maintenance operations (such as expiring snapshots older than one day, or cleaning up files not referenced by metadata under a table location) on Databricks Runtime 14.3 LTS or above. 

You encounter the following error message. 

RuntimeException: org.apache.spark.sql.AnalysisException: Multiple sources found for iceberg (com.databricks.sql.transaction.tahoe.uniform.sources.IcebergBrowseOnlyDataSource, org.apache.iceberg.spark.source.IcebergSource), please specify the fully qualified class name.

Caused by: AnalysisException: Multiple sources found for iceberg (com.databricks.sql.transaction.tahoe.uniform.sources.IcebergBrowseOnlyDataSource, org.apache.iceberg.spark.source.IcebergSource), please specify the fully qualified class name.

Cause

In Databricks Runtime 14.3 LTS and above, Databricks includes its own implementation and support for Iceberg, which introduces a definition of the "iceberg" data source. 

If you continue using external JARs, you create a conflict between the internal system and your external JAR because Iceberg Apache Spark runtime library (JAR) provides a separate definition for the "iceberg" source. 

When the Spark job attempts to use Iceberg maintenance operations, Spark finds multiple registrations for the "iceberg" data source and fails with the "Multiple sources found for iceberg" error. 

Solution

Using Iceberg external JARs is no longer supported. Databricks recommends instead using Managed Iceberg (in public preview, available on Databricks Runtime 16.4 LTS and above), which is designed to simplify the management of Iceberg tables and can help in running predictive optimizations. For more information, review the Announcing full Apache Iceberg™ support in Databricks documentation. 

If you cannot use 16.4 LTS and above, conduct Iceberg maintenance operations outside Databricks. 

For more information on Iceberg maintenance, review the Apache Iceberg Maintenance documentation.