How to check if a spark property is modifiable in a notebook

Learn how to modify Spark properties in a Databricks notebook.

Written by Adam Pavlacka

Last published at: May 16th, 2022

Problem

You can tune applications by setting various configurations. Some configurations must be set at the cluster level, whereas some are set inside notebooks or applications.

Solution

To check if a particular Spark configuration can be set in a notebook, run the following command in a notebook cell:

%scala

spark.conf.isModifiable("spark.databricks.preemption.enabled")

If true is returned, then the property can be set in the notebook. Otherwise, it must be set at the cluster level.