Databricks Connect reports version error with Databricks Runtime 6.4

Problem

You are using the Databricks Connect client with Databricks Runtime 6.4 and receive an error message which states that the client does not support the cluster.

Caused by: java.lang.IllegalArgumentException: The cluster is running server version `dbr-6.4` but this client only supports Set(dbr-5.5). You can find a list of client releases at https://pypi.org/project/databricks-connect/#history, and install the right client version with `pip install -U databricks-connect==<version>`. For example, to install the latest 5.1 release, use `pip install -U databricks-connect==5.1.*`. To ignore this error and continue, set DEBUG_IGNORE_VERSION_MISMATCH=1.

Cause

Improvements were made to Databricks Runtime 6.4 which are incompatible with the Databricks Connect client 6.4.1 and below.

Solution

Upgrade the Databricks Connect client to 6.4.2.

Follow the documentation to set up the client on your local workstation, making sure to set the databricks-connect value to 6.4.2.

pip install databricks-connect==6.4.2

Warning

Setting DEBUG_IGNORE_VERSION_MISMATCH=1 is not recommended, as it does not resolve the underlying compatibility issues.