Cannot modify the value of an Apache Spark config

You cannot modify the value of a Spark config setting within a notebook. It must be set at the cluster level.

Written by Adam Pavlacka

Last published at: May 20th, 2022

Problem

You are trying to SET the value of a Spark config in a notebook and get a Cannot modify the value of a Spark config error.

For example:

%sql

SET spark.serializer=org.apache.spark.serializer.KryoSerializer
Error in SQL statement: AnalysisException: Cannot modify the value of a Spark config: spark.serializer;

Cause

The SET command does not work on SparkConf entries. This is by design in Spark 3.0 and above.

Solution

You should remove SET commands for SparkConf entries from your notebook.

You can enter SparkConf values at the cluster level by entering them in the cluster’s Spark config (AWS | Azure | GCP) and restarting the cluster.