Updated October 26th, 2022 by rakesh.parija

Run a custom Databricks runtime on your cluster

The majority of Databricks customers use production Databricks runtime releases (AWS | Azure | GCP) for their clusters. However, there may be certain times when you are asked to run a custom Databricks runtime after raising a support ticket. Warning Custom Databricks runtime images are created for specific, short-term fixes and edge cases. If a cust...

1 min reading time
Updated May 16th, 2022 by rakesh.parija

Parquet timestamp requires Hive metastore 1.2 or above

Problem You are trying to create a Parquet table using TIMESTAMP, but you get an error message. Error in SQL statement: QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.UnsupportedOperationException: Parquet does not support timestamp. See HIVE-6384 Example code %sql CREATE EXTERN...

0 min reading time
Updated July 1st, 2022 by rakesh.parija

Service principal cannot create access token

Problem You are trying to create a token on behalf of a service principal , using /2.0/token-management/on-behalf-of/tokens in the REST API but are getting a PERMISSION_DENIED error. { "error_code": "PERMISSION_DENIED", "message": "User xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx does not have permission to use tokens." } Cause This happens when the servic...

1 min reading time
Updated May 15th, 2023 by rakesh.parija

Maximum execution context or notebook attachment limit reached

Problem Notebook or job execution stops and returns either of the following errors: Run result unavailable: job failed with error message Context ExecutionContextId(1731742567765160237) is disconnected. Can’t attach this notebook because the cluster has reached the attached notebook limit. Detach a notebook and retry. Cause When you attach a noteboo...

0 min reading time
Updated May 31st, 2022 by rakesh.parija

No USAGE permission on database

Problem You are using a cluster running Databricks Runtime 7.3 LTS and above. You have enabled table access control for your workspace (AWS | Azure | GCP) as the admin user, and granted the SELECT privilege to a standard user-group that needs to access the tables. A user tries to access an object in the database and gets a SecurityException error me...

0 min reading time
Updated May 11th, 2022 by rakesh.parija

Latest PyStan fails to install on Databricks Runtime 6.4

Problem You are trying to install the PyStan PyPi package on a Databricks Runtime 6.4 Extended Support cluster and get a ManagedLibraryInstallFailed error message. java.lang.RuntimeException: ManagedLibraryInstallFailed: org.apache.spark.SparkException: Process List(/databricks/python/bin/pip, install, pystan, --disable-pip-version-check) exited wit...

0 min reading time
Updated August 21st, 2023 by rakesh.parija

Error: Received command c on object id p0

Problem You have imported Python libraries, but when you try to execute Python code in a notebook you get a repeating message as output. Example 1: INFO:py4j.java_gateway:Received command c on object id p0INFO:py4j.java_gateway:Received command c on object id p0INFO:py4j.java_gateway:Received command c on object id p0INFO:py4j.java_gateway:Received ...

0 min reading time
Updated May 19th, 2022 by rakesh.parija

Display file and directory timestamp details

In this article we show you how to display detailed timestamps, including the date and time when a file was created or modified. Use ls command The simplest way to display file timestamps is to use the ls -lt <path> command in a bash shell. For example, this sample command displays basic timestamps for files and directories in the /dbfs/ folde...

1 min reading time
Updated May 9th, 2022 by rakesh.parija

Databricks Connect reports version error with Databricks Runtime 6.4

Problem You are using the Databricks Connect client with Databricks Runtime 6.4 and receive an error message which states that the client does not support the cluster. Caused by: java.lang.IllegalArgumentException: The cluster is running server version `dbr-6.4` but this client only supports Set(dbr-5.5). You can find a list of client releases at ht...

0 min reading time
Updated December 21st, 2022 by rakesh.parija

Job fails with Java IndexOutOfBoundsException error

Problem Your job fails with a Java IndexOutOfBoundsException error message: java.lang.IndexOutOfBoundsException: index: 0, length: <number> (expected: range(0, 0)) When you review the stack trace you see something similar to this: Py4JJavaError: An error occurred while calling o617.count. : org.apache.spark.SparkException: Job aborted due to s...

2 min reading time
Load More