Cannot modify the value of an Apache Spark config

Problem

You are trying to SET the value of a Spark config in a notebook and get a Cannot modify the value of a Spark config error.

For example:

SET spark.serializer=org.apache.spark.serializer.KryoSerializer
Error in SQL statement: AnalysisException: Cannot modify the value of a Spark config: spark.serializer;

Cause

The SET command does not work on SparkConf entries. This is by design in Spark 3.0 and above.

Solution

You should remove SET commands for SparkConf entries from your notebook.

You can enter SparkConf values at the cluster level by entering them in the cluster’s Spark configuration and restarting the cluster.