Updated March 15th, 2023 by Atanu.Sarkar

Failed to create query error when upgrading external metastore to Unity Catalog

Problem You are trying to migrate your default Hive metastore to Unity Catalog by following the steps in the Upgrade a schema or multiple tables to Unity Catalog (AWS | Azure) documentation. You have created the necessary storage credential (AWS | Azure) as well as the external location (AWS | Azure). Permissions are correct. You click Create Query ...

0 min reading time
Updated January 20th, 2023 by Atanu.Sarkar

R commands fail on custom Docker cluster

Problem You are trying to run R notebooks on a custom Docker cluster (AWS | Azure), but they immediately fail. When you try to execute an R notebook, it returns an error saying the notebook was cancelled. When you review the Cluster driver and worker logs (AWS | Azure) you see a there is no package called 'Rserve' error. Tue Aug 30 16:24:34 UTC 2022...

1 min reading time
Updated May 17th, 2022 by Atanu.Sarkar

Update job permissions for multiple users

When you are running jobs, you might want to update user permissions for multiple users. You can do this by using the Databricks job permissions API (AWS | Azure | GCP) and a bit of Python code. Instructions Copy the example code into a notebook. Enter the <job-id> (or multiple job ids) into the array arr[]. Enter your payload{}. In this examp...

0 min reading time
Updated January 20th, 2023 by Atanu.Sarkar

Job timeout when connecting to a SQL endpoint over JDBC

Problem You have a job that is reading and writing to an SQL endpoint over a JDBC connection. The SQL warehouse fails to execute the job and you get a java.net.SocketTimeoutException: Read timed out error message. 2022/02/04 17:36:15 - TI_stg_trade.0 - Caused by: com.simba.spark.jdbc42.internal.apache.thrift.transport.TTransportException: java.net.S...

0 min reading time
Load More