Validate environment variable behavior

In November 2021, the way environment variables are interpreted when creating, editing, or updating clusters was changed in some workspaces.

This change will be reverted on December 3, 2021 from 01:00-03:00 UTC.

After the change is reverted, environment variables will behave as they did before the change.

This article explains how to validate the environment variable behavior on your cluster.

Behavior change examples

Use case “New” input Original input Expected value
Escape special characters ($,`,”,\) var=" var=\" "
Use $ to access other vars Not supported otherVar=1 var=$otherVar2 12
Use ‘ or escape ‘ var=te'\''st or var=te'"'"'st var=te'st te'st

Note

The “new” input behavior will no longer work after the change has been reverted.

Check the environment variable behavior

Use the following steps to determine if your workspace is using the original environment variable behavior, or if it is using the new environment variable behavior. The new behavior is scheduled to be reverted.

  1. Navigate to the Create Cluster page in your workspace.

  2. Expand Advanced options.

  3. Select Spark.

  4. Enter BEHAVIOR_CHECK=behavior_check in Environment variables.

    Update environment variables.
  5. Click Create Cluster to launch the cluster.

  6. After the cluster is successfully launched, create a notebook, and attach it to the newly launched cluster.

  7. Copy the following command and run it in a cell:

    cat /databricks/spark/conf/spark-env.sh
    
  8. If BEHAVIOR_CHECK="behavior_check" (double quotes) is included in the return value, your cluster is using the original behavior. NO ACTION IS REQUIRED

    Original behavior output.
  9. If BEHAVIOR_CHECK='behavior_check' (single quotes) is included in the return value, your cluster is using the new behavior.

    New behavior output.

If you have not made any changes to your configuration, you do not need to take any action.

If you have made changes to your configuration to account for the new environment variable behavior, you must revert those changes when the platform change is reverted on December 3, 2021 from 01:00-03:00 UTC.