Delta Merge cannot resolve nested field

Delta Merge fails with a `Delta Merge cannot resolve 'field' due to data type mismatch` error message.

Written by Adam Pavlacka

Last published at: May 10th, 2022

Problem

You are attempting a Delta Merge with automatic schema evolution, but it fails with a Delta Merge: cannot resolve 'field' due to data type mismatch error message.

Cause

This can happen if you have made changes to the nested column fields.

For example, assume we have a column called Address with the fields streetName, houseNumber, and city nested inside.

Attempting to add an additional field, or remove a field, causes any upcoming insert or update transaction on the table to fail, even if mergeSchema is true for the transaction.

Solution

This behavior is by design.

The Delta automatic schema evolution feature only supports top level columns. Nested fields are not supported.

Please review the Delta Lake Automatic schema evolution (AWS | Azure | GCP) documentation for more information.