Jobs fail with error: There are already 1000 active runs (limit: 1000).

Identify and cancel job runs causing the issue, then schedule future job runs farther apart than a few minutes.

Written by walter.camacho

Last published at: September 27th, 2024

Problem 

Jobs may fail with an error indicating that the workspace has reached the maximum number of active runs, which is set to 1000.

Run failed with error message
There are already 1000 active runs (limit: 1000).

Cause

Jobs are either creating new runs in a loop, causing the workspace to reach the 1000 limit of active runs, or multiple job runs are scheduled at the same time. 

Solution

First, identify the jobs that are causing the issue. 

  1. Navigate to Job Runs in the UI. 
  2. Check how many jobs are running. 
  3. Verify if any jobs have a high number of runs corresponding to the same job ID (evidence of creating new runs in a loop). 

Next, cancel the runs for the identified jobs using the UI or the API. 

Through the UI:  

  1. Navigate to Workflows.
  2. Select the job that has been identified as running in a loop.
  3. From the Runs tab, select the option Cancel Runs. 

To use the API, please refer to the Cancel all runs of a job documentation.

Then, check the job configuration and scheduling settings. Be sure you define schedules based on your expected timeframe, avoiding running jobs within seconds or minutes of each other. Avoid steps that call the job multiple times.

Last, monitor the workspace to ensure that the issue does not recur.