How to ensure idempotency for jobs

Learn how to ensure that jobs submitted through the Databricks REST API aren't duplicated if there is a retry after a request times out.

Written by Adam Pavlacka

Last published at: May 11th, 2022

When you submit jobs through the Databricks Jobs REST API, idempotency is not guaranteed. If the client request is timed out and the client resubmits the same request, you may end up with duplicate jobs running.

To ensure job idempotency when you submit jobs through the Jobs API, you can use an idempotency token to define a unique value for a specific job run. If the same job has to be retried because the client did not receive a response due to a network error, the client can retry the job using the same idempotency token, ensuring that a duplicate job run is not triggered.

Here is an example of a REST API JSON payload for the Runs Submit API using an idempotency_token with a value of 123:

  "run_name":"my spark task",
  "new_cluster": {
    "aws_attributes": {
  "libraries": [
      "maven": {
  "spark_jar_task": {

All requests with the same idempotency token should return 200 with the same run ID.

Was this article helpful?