CPU core limit prevents cluster creation

Learn how to fix a CPU core quota limit Cloud Provider Launch error when starting a Databricks cluster.

Written by Adam Pavlacka

Last published at: December 8th, 2022

Problem

Cluster creation fails with a message about a cloud provider error when you hover over cluster state.

Cloud Provider Launch Failure: A cloud provider error was encountered while setting up the cluster.

When you view the cluster event log to get more details, you see a message about core quota limits.

Operation results in exceeding quota limits of Core. Maximum allowed: 350, Current in use: 350, Additional requested: 4.

Cause

Azure subscriptions have a CPU core quota limit which restricts the number of CPU cores you can use. This is a hard limit. If you try to start a cluster that would result in your account exceeding the CPU core quota the cluster launch will fail.

Solution

You can either free up resources or request a quota increase for your account.

  • Stop inactive clusters to free up CPU cores for use.
  • Open an Azure support case with a request to increase the CPU core quota limit for your subscription.