How to calculate the number of cores in a cluster

Learn how to calculate the number of cores in a Databricks cluster.

Written by Adam Pavlacka

Last published at: March 31st, 2023

You can view the number of cores in a Databricks cluster in the Workspace UI using the Metrics tab on the cluster details page.

Delete

Note

Azure Databricks cluster nodes must have a metrics service installed.

If the driver and executors are of the same node type, you can also determine the number of cores available in a cluster programmatically, using Scala utility code:

  1. Use sc.statusTracker.getExecutorInfos.length to get the total number of nodes. The result includes the driver node, so subtract 1.
  2. Use java.lang.Runtime.getRuntime.availableProcessors to get the number of cores per node.
  3. Multiply both results (subtracting 1 from the total number of nodes) to get the total number of cores available.

Scala example code:

java.lang.Runtime.getRuntime.availableProcessors * (sc.statusTracker.getExecutorInfos.length -1)