Problem

After creating a cluster programmatically with a workload_type restriction on jobs, you try to create a job on that cluster using the API. The job creation fails with the following error. 

InvalidParameterValue: The cluster <clusterId> does not support jobs workload.

Additionally, you don’t see the cluster in the Jobs & Pipelines UI. 

Cause

When you programmatically configure a cluster with a workload_type restriction on jobs, it tells the cluster it is not allowed to run jobs. It doesn’t run the jobs you create on that cluster, and the cluster doesn’t appear as available in your job UI. 

Specifically, during cluster creation you set the workloadType.clients.jobs field to false. You can verify the setting in the cluster’s JSON. Look for the following code.

"workload_type": {
        "clients": {
            "notebooks": true,
            "jobs": false
        }
    }

Solution

Since the cluster was created programmatically, it should be updated programmatically. Use the API to update workloadType.clients.jobs to true so the cluster can be used to run jobs. Alternatively, you can use the Databricks SDK for Python.

API 

Refer to the workload_type entry in the Update cluster configuration (AWS | Azure | GCP) API documentation for details.

Databricks SDK for Python

You can use the following script to update the configurations using the SDK. 

from databricks.sdk.service.compute import WorkloadType, ClientsTypes, UpdateClusterResource

clients = ClientsTypes(notebooks=True, jobs=False)

# Define the workload type
workload_type = WorkloadType(clients=clients)

# Update the cluster
_ = w.clusters.update(
  cluster_id = clstr.cluster_id, 
  cluster = UpdateClusterResource(
    workload_type = workload_type
  ),
  update_mask = "workload_type",
).result()