Set executor log level
To set the log level on all executors, you must set it inside the JVM on each worker.
For example:
sc.parallelize(Seq("")).foreachPartition(x => {
import org.apache.log4j.{LogManager, Level}
import org.apache.commons.logging.LogFactory
LogManager.getRootLogger().setLevel(Level.DEBUG)
val log = LogFactory.getLog("EXECUTOR-LOG:")
log.debug("START EXECUTOR DEBUG LOG LEVEL")
})
To verify that the level is set, navigate to the Spark UI, select the Executors tab, and open the stderr
log for any executor:
