Set executor log level

To set the log level on all executors, set it inside the JVM on each worker. Run the code below to set it:

sc.parallelize(Seq("")).foreachPartition(x => {
  import org.apache.log4j.{LogManager, Level}
  import org.apache.commons.logging.LogFactory

  LogManager.getRootLogger().setLevel(Level.DEBUG)
  val log = LogFactory.getLog("EXECUTOR-LOG:")
  log.debug("START EXECUTOR DEBUG LOG LEVEL")
})

To verify that the level is set, navigate to the Spark UI, select the Executors tab, and open the stderr log for any executor: