Problem

You are writing to a Delta table when you get an AnalysisException error indicating a schema mismatch.

'AnalysisException: A schema mismatch detected when writing to the Delta table (Table ID: bc10as3e-e12va-4f325-av10e-4s38f17vr3dd3)'. input_df.write.format("delta").mode("overwrite").save(target_delta_table_path)

Cause

The scheme mismatch is due to a change in the source schema and target schema. This usually happens when you introduce new columns to the target table during the write operation. This change in schema is a case of schema evolution. If these changes are expected, you should enable the mergeSchema property.

Solution

Modify the write command and set the mergeSchema property to true.

input_df.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(target_delta_table_path)

For more information, please review the Enable schema evolution (AWS | Azure | GCP) documentation.