Problem

If your Autoloader streaming job specifies the schema manually and also using schema evolution, you may encounter an error when upgrading your Auto Loader job configuration from Databricks Runtime 13.3 LTS to 14.3 LTS.

com.databricks.sql.cloudfiles.errors.CloudFilesIllegalArgumentException: Schema evolution mode addNewColumns is not supported when the schema is specified. To use this mode, you can provide the schema through cloudFiles.schemaHints instead.

Cause

As of Databricks Runtime 14.3 LTS, You can no longer provide both .option("avroSchema",your-schema) and .option("cloudFiles.schemaEvolutionMode","addNewColumns") parameters.

Solution

Choose to configure manually or via schema evolution.

For schema evolution, remove avroSchema and use .option("cloudFiles.schemaHints",your-schema)

For manual configuration, set .option("cloudFiles.schemaEvolutionMode","none").

For more information, please review the Configure schema inference and evolution in Auto Loader (AWS | Azure | GCP) documentation.