How to configure single-core executors to run JNI libraries

When you create a cluster, Databricks launches one executor instance per worker node, and the executor uses all of the cores on the node. In certain situations, such as if you want to run non-thread-safe JNI libraries, you might need an executor that has only one core or task slot, and does not attempt to run concurrent tasks. In this case, multiple executor instances run on a single worker node, and each executor has only one core.

If you run multiple executors, you increase the JVM overhead and decrease the overall memory available for processing.

To start single-core executors on a worker node, configure two properties:

• spark.executor.cores
• spark.executor.memory

The property spark.executor.cores specifies the number of cores per executor. Set this property to 1.

The property spark.executor.memory specifies the amount of memory to allot to each executor. To determine this amount, check the total amount of memory that is available on the worker node.

For example, an i3.xlarge node, which has 30.5 GB of memory, shows available memory at 24.9 GB. Choose a value that fits the available memory when multiplied by the number of executors. You may need to set a value that allows for some overhead. For example, set spark.executor.cores to 1 and spark.executor.memory to 6g:

The i3.xlarge instance type has 4 cores, and so 4 executors are created on the node, each with 6 GB of memory.