How to Check if a Spark Property is Modifiable in a Notebook

Problem

You can tune applications by setting various configurations. Some configurations must be set at the cluster level, whereas some are set inside notebooks or applications.

Solution

To check if a particular Spark configuration can be set in a notebook, run the following command in a notebook cell:

spark.conf.isModifiable("spark.databricks.preemption.enabled")

If true is returned, then the property can be set in the notebook. Otherwise, it must be set at the cluster level.