Trying to perform WRITE over UNION ALL causes error

Use Databricks Runtime 15.4 LTS or above.

Written by MuthuLakshmi.AN

Last published at: March 21st, 2025

Problem

In Databricks Runtime 14.3 LTS and below, when you perform WRITE over UNION ALL, you receive an error stating the query is not supported by Photon. Upon checking the SQL query plan page in the Apache Spark UI, you see additional information. 

== Photon Explanation ==
Photon does not fully support the query because:
Unsupported node: Union

 

Cause

The Photon implementation in Databricks Runtime 14.3 LTS and below only runs UNION in Photon when there is a SHUFFLE above the UNION. If there is WRITE instead, the UNION does not run in Photon.

 

Solution

To enable support for WRITE over UNION ALL, use Databricks Runtime 15.4 LTS or above. As of Databricks Runtime 16.1, the ability to perform WRITE over UNION ALL is available by default.