Problem

Your table writes are failing when attempting to read from a Delta table and returning the following error. 

Caused by: java.lang.RuntimeException: Corrupted parquet page (14)
…

Cause

This error often occurs when an external system overwrites files that are managed by Delta, even though the message says Corrupted parquet page.

 

The sequence of events looks like the following example. The timestamps are to show the order of events and that the corrupt page error refers to the initial Parquet file ingestion at the beginning of the sequence. 

12:39 AM: Parquet file ingested into Databricks (not written using Databricks). 

1:00 AM: Reads fail for an unknown reason.

6:09 AM: File is rewritten using a Databricks writer but not Delta.

6:30 AM: File read throws the corrupt page error. This file reads the file with a modification time of 12:39 AM. 

Solution

Upgrade to Databricks Runtime 13.3 LTS or above. Databricks Runtime 13.3 LTS improves file consistency checks and produces a clearer error message which you can then action. 

Caused by: com.databricks.common.filesystem.InconsistentReadException: The file might have been updated during query execution. Ensure that no pipeline updates existing files during query execution and try again.

To temporarily mitigate the issue, you can read the files with the OSS vectorized reader. To use the vectorized reader, set the following Apache Spark confs for your cluster.

spark.sql.parquet.enableVectorizedReader true
spark.databricks.io.parquet.fastreader.enabled false
spark.databricks.io.parquet.nativeReader.enabled false

For details on how to apply Spark configs, refer to the “Spark configuration” section of the Compute configuration reference (AWS | Azure | GCP) documentation.