Problem Your legacy Databricks Connect jobs start failing with a java.lang.ClassCastException error message. The error is not associated with any specific commands but seems to affect multiple Databricks Connect commands or jobs. Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.spark.sql.catalyst.trees.TreePattern$ to fi...

Problem Delta writes can result in the creation of empty files if the source is empty. This can happen with a regular Delta write or a MERGE INTO (AWS | Azure | GCP) operation. If your streaming application is writing to a target Delta table and your source data is empty on certain micro batches, it can result in writing empty files to your target D...

Problem Using key-value parameters in a multi task workflow is a common use case. It is normal to have multiple tasks running in parallel and each task can have different parameter values for the same key. These key-value parameters are read within the code and used by each task. For example, assume you have four tasks: task1, task2, task3, and task...

Problem You are querying a Delta table shared in Delta Sharing and you get a RESOURCE_LIMIT_EXCEEDED error. The error may reference a timeout. io.delta.sharing.spark.util.UnexpectedHttpStatus: HTTP request failed with status: HTTP/1.1 400 Bad Request {"errorCode":"RESOURCE_LIMIT_EXCEEDED","message":"A timeout occurred when processing the table. If i...

Problem You want to list the details of a Unity Catalog table, like location, size etc. You are trying to use the describe detail command, but it returns a permission denied error message. describe detail poc.schema1.testtable Error Message: Error in SQL statement: ExecutionException: java.util.concurrent.ExecutionException: com.databricks.sql.manag...