Upgrading to 14.3 LTS gives the error "com.databricks.sql.cloudfiles.errors.CloudFilesIllegalArgumentException"

Problem If your Autoloader streaming job specifies the schema manually and also using schema evolution, you may encounter an error when upgrading your Auto Loader job configuration from Databricks Runtime 13.3 LTS to 14.3 LTS. com.databricks.sql.cloudfiles.errors.CloudFilesIllegalArgumentException: Schema evolution mode addNewColumns is not supporte...

0 min reading time

Idle clusters causing inefficient resource use and increased costs

Problem You have a streaming job that runs continuously, processing batches of data as they arrive. However, the time interval between batches can vary significantly, ranging from one minute to two hours, leaving the cluster idle for extended periods throughout the day. You experience inefficient resource usage and increased costs. Cause You have no...

0 min reading time