How to use Apache Spark metrics

This article gives an example of how to monitor Apache Spark components using the Spark configurable metrics system. Specifically, it shows how to set a new source and enable a sink.

For detailed information about the Spark components available for metrics collection, including sinks supported out of the box, follow the documentation link above.

Note

There are several other ways to collect metrics to get insight into how a Spark job is performing, which are also not covered in this article:

  • SparkStatusTracker (Source, API): monitor job, stage, or task progress
  • StreamingQueryListener (Source, API): intercept streaming events
  • SparkListener: intercept events from Spark scheduler

For information about using other third-party tools to monitor Spark jobs in Databricks, see Metrics.

How does this metrics collection system work? Upon instantiation, each executor creates a connection to the driver to pass the metrics.

The first step is to write a class that extends the Source trait:

class MySource extends Source {
  override val sourceName: String = "MySource"

  override val metricRegistry: MetricRegistry = new MetricRegistry

  val FOO: Histogram = metricRegistry.histogram(MetricRegistry.name("fooHistory"))
  val FOO_COUNTER: Counter = metricRegistry.counter(MetricRegistry.name("fooCounter"))
}

The next step is to enable the sink. In this example, the metrics are printed to the console:

val spark: SparkSession = SparkSession
    .builder
    .master("local[*]")
    .appName("MySourceDemo")
    .config("spark.driver.host", "localhost")
    .config("spark.metrics.conf.*.sink.console.class", "org.apache.spark.metrics.sink.ConsoleSink")
.getOrCreate()

Note

To sink metrics to Prometheus, you can use this third-party library: https://github.com/banzaicloud/spark-metrics.

The last step is to instantiate the source and register it with SparkEnv:

val source: MySource = new MySource
SparkEnv.get.metricsSystem.registerSource(source)

You can view a complete, buildable example at https://github.com/newroyker/meter.