[ISOLATION_STARTUP_FAILURE] in shared clusters with Databricks Runtime 14.3 LTS - 15.4 LTS

Leverage an assigned cluster or revert to a custom Databricks Runtime release version as interim solutions.

Written by brock.baurer

Last published at: September 11th, 2024

Problem 

While working in Databricks Runtime 14.3 LTS - 15.4 LTS, you may encounter an ISOLATION_STARTUP_FAILURE error when using User-Defined Functions (UDFs) via Unity Catalog Lakeguard, in clusters with shared access mode. 

Job aborted due to stage failure: Task 36 in stage 583.0 failed 4 times, most recent failure: Lost task 36.3 in stage 583.0 (TID 109436) (<executorIpAddress> executor 16): org.apache.spark.SparkException: [ISOLATION_STARTUP_FAILURE] Failed to start isolated execution environment. Please contact Databricks support. SQLSTATE: XXKSS

Cause

Engineers are actively investigating the root cause of the error occuring on Databricks Runtime versions 14.3 LTS - 15.4 LTS when using shared access mode.

Solution

As an interim solution Databricks recommends reverting to using clusters with Assigned access mode. A fix will be rolled out as soon as possible and the timelines will be communicated later.

 

If you feel this isn't sufficient, you have two additional options. 

  1. Open a support case with Databricks Support and they will provide you with details on a custom release that is suited to your specific runtime needs.
  2. Revert to a custom Databricks Runtime release that aligns to your required release version. Find your closest Databricks Runtime version in the following sections.  For details around how to set up a custom Databricks Runtime image in a cluster please refer to the Run a custom Databricks Runtime on your cluster article.

Databricks Runtime 14.3 LTS

Leverage a custom image of Databricks Runtime 14.3.10 such as:

custom:release__14.3.x-snapshot-aarch64-photon-scala2.12__databricks-universe__14.3.10__9b6cd4f__debafb7__jenkins__c858f8a__format-3
custom:release__14.3.x-snapshot-aarch64-scala2.12__databricks-universe__14.3.10__9b6cd4f__debafb7__jenkins__ea8cc33__format-3
custom:release__14.3.x-snapshot-cpu-ml-scala2.12__databricks-universe__14.3.10__9b6cd4f__debafb7__jenkins__6b5baa7__format-3
custom:release__14.3.x-snapshot-gpu-ml-scala2.12__databricks-universe__14.3.10__9b6cd4f__debafb7__jenkins__3caf6b6__format-3
custom:release__14.3.x-snapshot-photon-scala2.12__databricks-universe__14.3.10__9b6cd4f__debafb7__jenkins__a4fe49f__format-3
custom:release__14.3.x-snapshot-scala2.12__databricks-universe__14.3.10__9b6cd4f__debafb7__jenkins__1cbb705__format-3

Databricks Runtime 15.1

Leverage a custom image of Databricks Runtime 15.1.4 such as:

custom:release__15.1.x-snapshot-aarch64-photon-scala2.12__databricks-universe__15.1.4__e4ade3f__c0005d5__jenkins__3c9f4ee__format-3
custom:release__15.1.x-snapshot-aarch64-scala2.12__databricks-universe__15.1.4__e4ade3f__c0005d5__jenkins__5b92790__format-3
custom:release__15.1.x-snapshot-cpu-ml-scala2.12__databricks-universe__15.1.4__e4ade3f__c0005d5__jenkins__a9467f5__format-3
custom:release__15.1.x-snapshot-gc-photon-scala2.12__databricks-universe__15.1.4__e4ade3f__c0005d5__jenkins__75359d9__format-3
custom:release__15.1.x-snapshot-gpu-ml-scala2.12__databricks-universe__15.1.4__e4ade3f__c0005d5__jenkins__9defed7__format-3
custom:release__15.1.x-snapshot-photon-scala2.12__databricks-universe__15.1.4__e4ade3f__c0005d5__jenkins__2ffa4da__format-3
custom:release__15.1.x-snapshot-scala2.12__databricks-universe__15.1.4__e4ade3f__c0005d5__jenkins__06d6025__format-3

Databricks Runtime 15.2

Leverage a custom image of Databricks Runtime 15.2.2 such as:

custom:release__15.2.x-snapshot-aarch64-photon-scala2.12__databricks-universe__15.2.2__b7ae9ec__fb0af7f__jenkins__eda3be1__format-3
custom:release__15.2.x-snapshot-aarch64-scala2.12__databricks-universe__15.2.2__b7ae9ec__fb0af7f__jenkins__3bf7101__format-3
custom:release__15.2.x-snapshot-cpu-ml-photon-scala2.12__databricks-universe__15.2.2__b7ae9ec__fb0af7f__jenkins__37734c0__format-3
custom:release__15.2.x-snapshot-cpu-ml-scala2.12__databricks-universe__15.2.2__b7ae9ec__fb0af7f__jenkins__4017a81__format-3
custom:release__15.2.x-snapshot-gpu-ml-scala2.12__databricks-universe__15.2.2__b7ae9ec__fb0af7f__jenkins__e4d774a__format-3
custom:release__15.2.x-snapshot-photon-scala2.12__databricks-universe__15.2.2__b7ae9ec__fb0af7f__jenkins__b8f11fd__format-3
custom:release__15.2.x-snapshot-scala2.12__databricks-universe__15.2.2__b7ae9ec__fb0af7f__jenkins__5ba15bb__format-3

Databricks Runtime 15.3

Leverage a custom image of Databricks Runtime 15.3.1 such as:

custom:release__15.3.x-snapshot-aarch64-photon-scala2.12__databricks__15.3.1__c92efd0__d4a96cf__jenkins__32a3fdb__format-3
custom:release__15.3.x-snapshot-aarch64-scala2.12__databricks__15.3.1__c92efd0__d4a96cf__jenkins__2c7332f__format-3
custom:release__15.3.x-snapshot-cpu-ml-photon-scala2.12__databricks__15.3.1__c92efd0__d4a96cf__jenkins__452cd59__format-3
custom:release__15.3.x-snapshot-cpu-ml-scala2.12__databricks__15.3.1__c92efd0__d4a96cf__jenkins__fb8cb0f__format-3
custom:release__15.3.x-snapshot-gpu-ml-scala2.12__databricks__15.3.1__c92efd0__d4a96cf__jenkins__2af35ec__format-3
custom:release__15.3.x-snapshot-photon-scala2.12__databricks__15.3.1__c92efd0__d4a96cf__jenkins__f49dfac__format-3
custom:release__15.3.x-snapshot-scala2.12__databricks__15.3.1__c92efd0__d4a96cf__jenkins__974131c__format-3

Databricks Runtime 15.4 LTS

Revert to the latest Databricks Runtime 15.3.1 custom image, such as what is listed previously in the Databricks Runtime 15.3 snippet.