Problem

A Databricks notebook or Jobs API request returns the following error:

Error : {"error_code":"INVALID_STATE","message":"There were already 1000 jobs created in past 3600 seconds, exceeding rate limit: 1000 job creations per 3600 seconds."}

Cause

This error occurs because the number of jobs per hour exceeds the limit of 1000 established by Databricks to prevent API abuses and ensure quality of service.

Solution

If you cannot ensure that the number of jobs created in your workspace is less than 1000 per hour, contact Databricks Support to request a higher limit. A job rate limit increase requires at least 20 minutes of downtime.

• Databricks can increase the job limit maximumJobCreationRate up to 2000.
• Currently running jobs will be affected while the limit is being increased.