RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table
Problem You are querying a Delta table shared in Delta Sharing and you get a RESOURCE_LIMIT_EXCEEDED error. The error may reference a timeout. io.delta.sharing.spark.util.UnexpectedHttpStatus: HTTP request failed with status: HTTP/1.1 400 Bad Request {"errorCode":"RESOURCE_LIMIT_EXCEEDED","message":"A timeout occurred when processing the table. If i...
1 min reading timeDatabricks Connect job fails after a Databricks Runtime update
Problem Your legacy Databricks Connect jobs start failing with a java.lang.ClassCastException error message. The error is not associated with any specific commands but seems to affect multiple Databricks Connect commands or jobs. Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.spark.sql.catalyst.trees.TreePattern$ to fi...
0 min reading timeDelta writing empty files when source is empty
Problem Delta writes can result in the creation of empty files if the source is empty. This can happen with a regular Delta write or a MERGE INTO (AWS | Azure | GCP) operation. If your streaming application is writing to a target Delta table and your source data is empty on certain micro batches, it can result in writing empty files to your target D...
1 min reading timeMulti-task workflows using incorrect parameter values
Problem Using key-value parameters in a multi task workflow is a common use case. It is normal to have multiple tasks running in parallel and each task can have different parameter values for the same key. These key-value parameters are read within the code and used by each task. For example, assume you have four tasks: task1, task2, task3, and task...
0 min reading timePERMISSION_DENIED error when accessing Unity Catalog tables
Problem You want to list the details of a Unity Catalog table, like location, size etc. You are trying to use the describe detail command, but it returns a permission denied error message. describe detail poc.schema1.testtable Error Message: Error in SQL statement: ExecutionException: java.util.concurrent.ExecutionException: com.databricks.sql.manag...
0 min reading time