You are using the Databricks Connect client with Databricks Runtime 6.4 and receive an error message which states that the client does not support the cluster.
Caused by: java.lang.IllegalArgumentException: The cluster is running server version `dbr-6.4` but this client only supports Set(dbr-5.5). You can find a list of client releases at https://pypi.org/project/databricks-connect/#history, and install the right client version with `pip install -U databricks-connect==<version>`. For example, to install the latest 5.1 release, use `pip install -U databricks-connect==5.1.*`. To ignore this error and continue, set DEBUG_IGNORE_VERSION_MISMATCH=1.
Improvements were made to Databricks Runtime 6.4 which are incompatible with the Databricks Connect client 6.4.1 and below.
Upgrade the Databricks Connect client to 6.4.2.
Follow the documentation to set up the client (AWS | Azure) on your local workstation, making sure to set the databricks-connect value to 6.4.2.
pip install databricks-connect==6.4.2