Updated July 27th, 2023 by Rajeev kannan Thangaiah

Databricks Connect job fails after a Databricks Runtime update

Problem Your legacy Databricks Connect jobs start failing with a java.lang.ClassCastException error message. The error is not associated with any specific commands but seems to affect multiple Databricks Connect commands or jobs. Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.spark.sql.catalyst.trees.TreePattern$ to fi...

0 min reading time
Updated December 2nd, 2022 by Rajeev kannan Thangaiah

Delta writing empty files when source is empty

Problem Delta writes can result in the creation of empty files if the source is empty. This can happen with a regular Delta write or a MERGE INTO (AWS | Azure | GCP) operation. If your streaming application is writing to a target Delta table and your source data is empty on certain micro batches, it can result in writing empty files to your target D...

1 min reading time
Updated December 5th, 2022 by Rajeev kannan Thangaiah

Multi-task workflows using incorrect parameter values

Problem Using key-value parameters in a multi task workflow is a common use case. It is normal to have multiple tasks running in parallel and each task can have different parameter values for the same key. These key-value parameters are read within the code and used by each task. For example, assume you have four tasks: task1, task2, task3, and task...

0 min reading time
Updated April 19th, 2023 by Rajeev kannan Thangaiah

RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table

Problem You are querying a Delta table shared in Delta Sharing and you get a RESOURCE_LIMIT_EXCEEDED error. The error may reference a timeout. io.delta.sharing.spark.util.UnexpectedHttpStatus: HTTP request failed with status: HTTP/1.1 400 Bad Request {"errorCode":"RESOURCE_LIMIT_EXCEEDED","message":"A timeout occurred when processing the table. If i...

1 min reading time
Updated November 30th, 2023 by Rajeev kannan Thangaiah

PERMISSION_DENIED error when accessing Unity Catalog tables

Problem You want to list the details of a Unity Catalog table, like location, size etc. You are trying to use the describe detail command, but it returns a permission denied error message. describe detail poc.schema1.testtable Error Message: Error in SQL statement: ExecutionException: java.util.concurrent.ExecutionException: com.databricks.sql.manag...

0 min reading time
Load More