Problem Delta writes can result in the creation of empty files if the source is empty. This can happen with a regular Delta write or a MERGE INTO (AWS | Azure | GCP) operation. If your streaming application is writing to a target Delta table and your source data is empty on certain micro batches, it can result in writing empty files to your target D...

Problem You want to list the details of a Unity Catalog table, like location, size etc. You are trying to use the describe detail command, but it returns a permission denied error message. describe detail poc.schema1.testtable Error Message: Error in SQL statement: ExecutionException: java.util.concurrent.ExecutionException: com.databricks.sql.manag...

Problem You are trying to migrate to serverless compute but you are encountering an issue with the Apache Spark  delta.retentionDurationCheck property not working correctly. For example, this sample code snippet does not work when you are using serverless compute: spark.sql("SET spark.databricks.delta.retentionDurationCheck.enabled=false") spark.sql...

Problem Using key-value parameters in a multi task workflow is a common use case. It is normal to have multiple tasks running in parallel and each task can have different parameter values for the same key. These key-value parameters are read within the code and used by each task. For example, assume you have four tasks: task1, task2, task3, and task...

Problem Your legacy Databricks Connect jobs start failing with a java.lang.ClassCastException error message. The error is not associated with any specific commands but seems to affect multiple Databricks Connect commands or jobs. Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.spark.sql.catalyst.trees.TreePattern$ to fi...

Problem You are querying a Delta table shared in Delta Sharing and you get a RESOURCE_LIMIT_EXCEEDED error. The error may reference a timeout. io.delta.sharing.spark.util.UnexpectedHttpStatus: HTTP request failed with status: HTTP/1.1 400 Bad Request {"errorCode":"RESOURCE_LIMIT_EXCEEDED","message":"A timeout occurred when processing the table. If i...

Problem Apache Spark streaming jobs in Delta Lake may fail with errors indicating that the input schema contains nested fields that are capitalized differently than the target table. [DELTA_NESTED_FIELDS_NEED_RENAME]   The input schema contains nested fields that are capitalized differently than the target table. They need to be renamed to avoid the...