Set executor log level

Learn how to set the log levels on Databricks executors.

Written by Adam Pavlacka

Last published at: March 4th, 2022
Delete

Warning

This article describes steps related to customer use of Log4j 1.x within a Databricks cluster. Log4j 1.x is no longer maintained and has three known CVEs (CVE-2021-4104, CVE-2020-9488, and CVE-2019-17571). If your code uses one of the affected classes (JMSAppender or SocketServer), your use may potentially be impacted by these vulnerabilities.

To set the log level on all executors, you must set it inside the JVM on each worker.

For example:

%scala

sc.parallelize(Seq("")).foreachPartition(x => {
  import org.apache.log4j.{LogManager, Level}
  import org.apache.commons.logging.LogFactory

  LogManager.getRootLogger().setLevel(Level.DEBUG)
  val log = LogFactory.getLog("EXECUTOR-LOG:")
  log.debug("START EXECUTOR DEBUG LOG LEVEL")
})

To verify that the level is set, navigate to the Spark UI, select the Executors tab, and open the stderr log for any executor: