How to calculate the number of cores in a cluster

You can view the number of cores in a Databricks cluster in the Workspace UI using the Metrics tab on the cluster details page.

If the driver and executors are of the same node type, you can also determine the number of cores available in a cluster programmatically, using Scala utility code:

  1. Use sc.statusTracker.getExecutorInfos.length to get the total number of nodes. The result includes the driver node, so subtract 1.

  2. Use java.lang.Runtime.getRuntime.availableProcessors to get the number of cores per node.

  3. Multiply both results (subtracting 1 from the total number of nodes) to get the total number of cores available:

    java.lang.Runtime.getRuntime.availableProcessors * (sc.statusTracker.getExecutorInfos.length -1)