Converting from Parquet to Delta Lake fails

Converting a file from Parquet to Delta Lake fails with a partition error when you have a subdirectory. Expecting 0 partition column(s), but found 1 partition column(s)

Written by Jose Gonzalez

Last published at: May 10th, 2022

Problem

You are attempting to convert a Parquet file to a Delta Lake file.

The directory containing the Parquet file contains one or more subdirectories.

The conversion fails with the error message: Expecting 0 partition column(s): [], but found 1 partition column(s): [<column_name>] from parsing the file name: <path_to_the_file_location>;.

Cause

The conversion process is attempting to process the subdirectory as a partition. This causes the error message.

Solution

If you are using Databricks Runtime 7.5 or below, ensure that directories containing Parquet files do not have subdirectories.

This issue is resolved in Databricks Runtime 8.0 and above.