Databricks Connect reports version error with Databricks Runtime 6.4

Written by rakesh.parija

Last published at: May 9th, 2022

Problem

You are using the Databricks Connect client with Databricks Runtime 6.4 and receive an error message which states that the client does not support the cluster.

Caused by: java.lang.IllegalArgumentException: The cluster is running server version `dbr-6.4` but this client only supports Set(dbr-5.5). You can find a list of client releases at https://pypi.org/project/databricks-connect/#history, and install the right client version with `pip install -U databricks-connect==<version>`. For example, to install the latest 5.1 release, use `pip install -U databricks-connect==5.1.*`. To ignore this error and continue, set DEBUG_IGNORE_VERSION_MISMATCH=1.

Cause

Improvements were made to Databricks Runtime 6.4 which are incompatible with the Databricks Connect client 6.4.1 and below.

Solution

Upgrade the Databricks Connect client to 6.4.2.

Follow the documentation to set up the client (AWS | Azure) on your local workstation, making sure to set the databricks-connect value to 6.4.2.

pip install databricks-connect==6.4.2
Delete

Warning

Setting DEBUG_IGNORE_VERSION_MISMATCH=1 is not recommended, as it does not resolve the underlying compatibility issues.