Problem

When you add a configuration setting by entering it in the Apache Spark config text area, the new setting replaces existing settings instead of being appended.

Version

Databricks Runtime 5.1 and below.

Cause

When the cluster restarts, the cluster reads settings from a configuration file that is created in the Clusters UI, and overwrites the default settings.

For example, when you add the following extraJavaOptions to the Spark config text area:

spark.executor.extraJavaOptions -
javaagent:/opt/prometheus_jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/prometheus
_jmx_exporter/jmx_prometheus_javaagent.yml

Then, in Spark UI > Environment > Spark Properties under spark.executor.extraJavaOptions, only the newly added configuration setting shows:

-javaagent:/opt/prometheus_jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/prometheus
_jmx_exporter/jmx_prometheus_javaagent.yml

Any existing settings are removed.

For reference, the default settings are:

-Djava.io.tmpdir=/local_disk0/tmp -XX:ReservedCodeCacheSize=256m -
XX:+UseCodeCacheFlushing -Ddatabricks.serviceName=spark-executor-1 -
Djava.security.properties=/databricks/spark/dbconf/java/extra.security -XX:+PrintFlagsFinal -
XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xss4m -
Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.Dataty
peFactoryImpl -
Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.Documen
tBuilderFactoryImpl -
Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFact
oryImpl -
Djavax.xml.validation.SchemaFactory=https://www.w3.org/2001/XMLSchema=com.sun.org.apache.xerces.internal.jaxp.validation.XMLSchemaFactory -
Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -
Dorg.w3c.dom.DOMImplementationSourceList=com.sun.org.apache.xerces.internal.dom.DOMX
SImplementationSourceImpl

Solution

To add a new configuration setting to spark.executor.extraJavaOptions without losing the default settings:

1. In Spark UI > Environment > Spark Properties, select and copy all of the properties set by default for spark.executor.extraJavaOptions.
2. Click Edit.
3. In the Spark config text area (Clusters > cluster-name > Advanced Options > Spark), paste the default settings.
4. Append the new configuration setting below the default settings.
5. Click outside the text area, then click Confirm.
6. Restart the cluster.

For example, let’s say you paste the following settings into the Spark config text area. The new configuration setting is appended to the default settings.

spark.executor.extraJavaOptions = -Djava.io.tmpdir=/local_disk0/tmp -
XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Ddatabricks.serviceName=spark-
executor-1 -Djava.security.properties=/databricks/spark/dbconf/java/extra.security -
XX:+PrintFlagsFinal -XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xss4m -
Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.Dataty
peFactoryImpl -
Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentB
uilderFactoryImpl -
Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactor
yImpl -
Djavax.xml.validation.SchemaFactory:https://www.w3.org/2001/XMLSchema=com.sun.org.apache.xer
ces.internal.jaxp.validation.XMLSchemaFactory -
Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -
Dorg.w3c.dom.DOMImplementationSourceList=com.sun.org.apache.xerces.internal.dom.DOMXSImplem
entationSourceImpl -
javaagent:/opt/prometheus_jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/prometheus_jm
x_exporter/jmx_prometheus_javaagent.yml

After you restart the cluster, the default settings and newly added configuration setting appear in Spark UI > Environment > Spark Properties.