Problem

If you try to create more new jobs than the limit quota, you receive an error message. 

"Exception: Error while running create_job_api: {'error_code': 'QUOTA_EXCEEDED', 'message': 'The quota for the number of jobs has been reached"

Cause

Databricks sets a limit for the number of jobs that can be created through the UI or the Jobs API. Each cloud provider sets a resource limit (AWS | Azure | GCP). 

Limits are enforced to prevent excessive job creation that could impact system performance and resource allocation.

Solution

Confirm the amount of jobs you have in your workspace, then identify and delete the jobs you do not need. Follow your preferred identification and deletion process (whether via Databricks API or through the UI). 

Example code to confirm number of jobs

The following code uses Databricks SDK for Python (AWS | Azure | GCP).  It will output the total number of saved jobs in a notebook cell. 

from databricks.sdk import WorkspaceClient
from databricks.sdk.service import jobs
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, LongType, StringType

# Initialize Spark session
spark = SparkSession.builder.appName("DatabricksJobList").getOrCreate()

w =WorkspaceClient() # this client is for current workspace, no PAT needed

# Get the list of jobs
job_list = w.jobs.list()

# Create a list to store job details
jobs_list = []

# Extract required fields from each job and append to the list
for job in job_list:
   jobs_list.append({
"created_time": job.created_time,
"creator_user_name": job.creator_user_name,
"name": job.settings.name
})

# Define the schema for the DataFrame
schema =StructType([
StructField("created_time", LongType(), True),
StructField("creator_user_name", StringType(), True),
StructField("name", StringType(), True)
])
# Create a DataFrame from the list of job details
df = spark.createDataFrame(jobs_list, schema) #dataframe of jobs
# Display the DataFrame , uncomment below line to see the jobs list as a Dataframe
#df.show(truncate=False)

total_jobs = df.count() #

print('total number of jobs in workspace is {total_jobs}')

In general, Databricks recommends regularly monitoring and cleaning up outdated or unnecessary jobs to ensure that the workspace does not reach the quota limit.