Auto termination is disabled when starting a job cluster

Auto termination policies are not supported on job clusters.

Written by navya.athiraram

Last published at: August 23rd, 2022

Problem

You are trying to start a job cluster, but the job creation fails with an error message.

Error creating job
Cluster autotermination is currently disabled.

Cause

Job clusters auto terminate once the job is completed. As a result, they do not support explicit autotermination policies.

If you include autotermination_minutes in your cluster policy JSON, you get the error on job creation.

{
 "autotermination_minutes": {
  "type": "fixed",
   "value": 30,
   "hidden": true
  }
}

Solution

Do not define autotermination_minutes in the cluster policy for job clusters.

Auto termination should only be used for all-purpose clusters.