Setting the Log Level on Executors in Apache SparkΒΆ

Settings the log level across all executors is trivial in Apache Spark. The trick is to set this inside of the JVM on each worker.

Execute the code below to set it:

sc.parallelize(Seq("")).foreachPartition(x => {
  import org.apache.log4j.{LogManager, Level}
  import org.apache.commons.logging.LogFactory

  LogManager.getRootLogger().setLevel(Level.DEBUG)
  val log = LogFactory.getLog("EXECUTOR-LOG:")
  log.debug("START EXECUTOR DEBUG LOG LEVEL")
})

Now all you need to do is navigate to the Spark UI for a given Executor and you’ll see the correct image. See the below GIF for an example for how to do that.

../../../_images/change-log-level.gif