Migration guidance for init scripts on DBFS

Init scripts on DBFS are end-of-life. You should migrate them to cloud storage, Unity Catalog volumes, or workspace files.

Written by Adam Pavlacka

Last published at: February 5th, 2024

This article provides migration guidance for init scripts on DBFS.

Detect End-of-Life init scripts

You can detect all init scripts stored on DBFS in your workspace by running the DBFS init script detection notebook. After you have identified any init scripts on DBFS, you should migrate them to supported storage.


Migrate End-of-Life init scripts:

The recommended migration path (AWS | Azure | GCP) for your init scripts depends on the init script type and the Databricks Runtime version you plan on using.


Legacy global init scripts on any Databricks Runtime version

  • If you are still using legacy global init scripts you should immediately migrate them to compute policies (AWS | Azure) or new global init scripts (AWS | Azure). This notebook can be used to automate the migration work to new global init scripts.


Cluster-named and cluster-scoped init scripts

Databricks Runtime 13.3 LTS and above (Unity Catalog clusters with single-user or shared access mode)

  • All cluster-named init scripts should be disabled (AWS | Azure) and converted to cluster-scoped init scripts (AWS | Azure).
  • Migrate your cluster-scoped init scripts from DBFS to Unity Catalog volumes (AWS | Azure | GCP).
  • Review the migration guides (AWS | Azure | GCP) for more information. 


Databricks Runtime 11.3 LTS and above

  • All cluster-named init scripts should be disabled (AWS | Azure) and converted to cluster-scoped init scripts (AWS | Azure).
  • Migrate your cluster-scoped init scripts from DBFS to workspace files (AWS | Azure | GCP).
  • Review the migration guides (AWS | Azure | GCP) for more information.


Databricks Runtime 10.4 LTS and below

  • All cluster-named init scripts should be disabled (AWS | Azure) and converted to cluster-scoped init scripts (AWS | Azure).
  • Migrate your cluster-scoped init scripts from DBFS to cloud object storage (AWS | Azure | GCP).
  • Review the migration guides (AWS | Azure | GCP) for more information.


Unsupported Databricks Runtime

  • Running an unsupported Databricks Runtime (AWS | Azure | GCP) is NOT recommended. You should immediately upgrade your clusters to a supported Databricks Runtime version (AWS | Azure | GCP).


Init script documentation

For more information on init scripts, please review the init script documentation (AWS | Azure | GCP).

 

https://docs.databricks.com/en/_extras/documents/aws-init-volumes.pdf

Was this article helpful?