Problem
You are writing to a Delta table when you get an AnalysisException
error indicating a schema mismatch.
'AnalysisException: A schema mismatch detected when writing to the Delta table (Table ID: bc10as3e-e12va-4f325-av10e-4s38f17vr3dd3)'. input_df.write.format("delta").mode("overwrite").save(target_delta_table_path)
Cause
The scheme mismatch is due to a change in the source schema and target schema. This usually happens when you introduce new columns to the target table during the write operation. This change in schema is a case of schema evolution. If these changes are expected, you should enable the mergeSchema
property.
Solution
Modify the write command and set the mergeSchema
property to true.
input_df.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(target_delta_table_path)
For more information, please review the Enable schema evolution (AWS | Azure | GCP) documentation.