Problem

The latest mssql-jdbc driver is not automatically included with Databricks Runtime releases. Consequently, specific Databricks Runtime versions may not provide the most up-to-date driver by default. To use the latest driver version, you must install or configure it manually.

Cause

New JDBC driver releases can introduce breaking or behavioral changes. To minimize risk and ensure stability, these upgrades are intentionally shipped more slowly in Databricks Runtime, which means your current driver version may not align with the newest driver release.

Solution

Use the following init script to install the latest mssql-jdbc driver, then choose a source and follow the respective steps to attach the init script to a compute. 

Install the driver

Modify the following script with your desired JDBC version and execute. The script defines a version and path, creates a temp directory, downloads the latest driver jar, removes older versions of the driver if any, moves the new jar to the default JAR directory, then cleans up the temp files. 

#!/bin/bash

# Define version and path
JDBC_VERSION="<desired-jdbc-version>" #Replace with the desired version
JDBC_JAR_NAME="mssql-jdbc-${JDBC_VERSION}.jar"
DOWNLOAD_URL="https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/${JDBC_VERSION}/${JDBC_JAR_NAME}"

# Create a temporary directory
mkdir -p /tmp/sqljdbc

# Download the latest driver jar
curl -L -o /tmp/sqljdbc/$JDBC_JAR_NAME $DOWNLOAD_URL

# Remove older versions of mssql-jdbc if any (carefully!)
find /databricks/jars -name "mssql-jdbc-*.jar" -exec rm -f {} \;

# Move the new jar to the default Databricks JARs directory
cp /tmp/sqljdbc/$JDBC_JAR_NAME /databricks/jars/

# Clean up temp files
rm -rf /tmp/sqljdbc

Attach the init script to a compute

1. Create a .sh file in the workspace.
2. Copy and paste the above code into the file and save. 
3. Copy the init script path. 
4. If your source is a workspace, proceed to step 4. If your source is a volume or S3, upload the .sh file to the volume or S3 location.
5. Open the cluster. Scroll down to Advanced options > Init Scripts
6. If your source is a workspace, under Sources, choose Workspace from the dropdown menu. If your source is a volume or S3, choose Volumes or S3 respectively from the dropdown menu. 
7. Paste the init script path in File path.

For more information, refer to the What are init scripts? (AWS | Azure | GCP) documentation.