How to check if a spark property is modifiable in a notebook
Problem
You can tune applications by setting various configurations. Some configurations must be set at the cluster level, whereas some are set inside notebooks or applications.
Solution
To check if a particular Spark configuration can be set in a notebook, run the following command in a notebook cell:
spark.conf.isModifiable("spark.databricks.preemption.enabled")
If true
is returned, then the property can be set in the notebook. Otherwise, it must be set at the cluster level.