Databricks Help Center

Main Navigation

  • Help Center
  • Documentation
  • Knowledge Base
  • Community
  • Training
  • Feedback

Jobs (Azure)

These articles can help you with your Databricks jobs.

51 Articles in this category

  • Home
  • All articles
  • Jobs (Azure)

Distinguish active and dead jobs

Learn how to distinguish between active and dead Databricks jobs....

Last updated: May 10th, 2022 by Adam Pavlacka

Spark job fails with Driver is temporarily unavailable

Job failure due to Driver being unavailable or unresponsive....

Last updated: April 17th, 2023 by Adam Pavlacka

How to delete all jobs using the REST API

Learn how to delete all Databricks jobs using the REST API....

Last updated: May 10th, 2022 by Adam Pavlacka

Job cluster limits on notebook output

Job clusters have a maximum notebook output size of 20 MB. If the output is larger, it results in an error....

Last updated: May 10th, 2022 by Jose Gonzalez

Job fails, but Apache Spark tasks finish

Your job fails, but all of the Apache Spark tasks have completed successfully. You are using spark.stop() or System.exit(0) in your code....

Last updated: May 10th, 2022 by harikrishnan.kunhumveettil

Job fails due to job rate limit

Learn how to resolve Databricks job failures due to job rate limits....

Last updated: April 17th, 2023 by Adam Pavlacka

Create table in overwrite mode fails when interrupted

Learn how to troubleshoot failures that occur when you rerun an Apache Spark write operation by cancelling the currently running job....

Last updated: May 10th, 2022 by Adam Pavlacka

Apache Spark Jobs hang due to non-deterministic custom UDF

Learn what to do when your Apache Spark job hangs due to a non-deterministic custom UDF....

Last updated: May 10th, 2022 by Adam Pavlacka

Apache Spark job fails with Failed to parse byte string

Apache Spark job fails with a Failed to parse byte string error....

Last updated: May 10th, 2022 by noopur.nigam

Apache Spark UI shows wrong number of jobs

Apache Spark UI shows the wrong number of active jobs....

Last updated: May 11th, 2022 by ashish

Job fails with atypical errors message

Job run is throttled and fails due to observing atypical errors message....

Last updated: May 11th, 2022 by Adam Pavlacka

Apache Spark job fails with maxResultSize exception

Learn what to do when an Apache Spark job fails with a maxResultSize exception....

Last updated: May 11th, 2022 by Adam Pavlacka

Databricks job fails because library is not installed

Learn how to prevent Databricks jobs from failing due to uninstalled libraries....

Last updated: May 11th, 2022 by Adam Pavlacka

Job failure due to Azure Data Lake Storage (ADLS) CREATE limits

Learn what to do when your Databricks job fails due to Azure Data Lake Storage CREATE limits....

Last updated: May 11th, 2022 by Adam Pavlacka

Job fails with invalid access token

Jobs that run more than 48 hours fail with invalid access token error when the dbutils token expires....

Last updated: May 11th, 2022 by manjunath.swamy

How to ensure idempotency for jobs

Learn how to ensure that jobs submitted through the Databricks REST API aren't duplicated if there is a retry after a request times out....

Last updated: May 11th, 2022 by Adam Pavlacka

Streaming job has degraded performance

Streaming job has poor performance after stopping and restarting from same checkpoint....

Last updated: May 11th, 2022 by ashish

Task deserialization time is high

Configure cluster-installed libraries to install on executors at cluster launch vs executor launch to speed up your job task runs....

Last updated: February 23rd, 2023 by Adam Pavlacka

Pass arguments to a notebook as a list

Use a JSON file to temporarily store arguments that you want to use in your notebook....

Last updated: October 29th, 2022 by pallavi.gowdar

Uncommitted files causing data duplication

Partially uncommitted files from a failed write can result in apparent data duplication. Adjust VACUUM settings to resolve the issue....

Last updated: November 8th, 2022 by gopinath.chandrasekaran

Multi-task workflows using incorrect parameter values

If parallel tasks running on the same cluster use Scala companion objects the wrong values can be used due to sharing a single class in the JVM....

Last updated: December 5th, 2022 by Rajeev kannan Thangaiah

Job fails with Spark Shuffle FetchFailedException error

Disable the default Spark Shuffle service to work around a FetchFailedException error....

Last updated: December 5th, 2022 by shanmugavel.chandrakasu

Users unable to view job results when using remote Git source

Databricks does not manage permission for remote repos, so you must sync changes with a local notebook so non-admin users can view results....

Last updated: March 7th, 2023 by ravirahul.padmanabhan

Single scheduled job tries to run multiple times

Ensure your cron syntax is correct when scheduling jobs. A wildcard in the wrong space can produce unexpected results....

Last updated: January 20th, 2023 by monica.cao

Jobs failing with shuffle fetch failures

Shuffle fetch failures can happen if you have modified the Azure Databricks subnet CIDR range after deployment....

Last updated: February 23rd, 2023 by arjun.kaimaparambilrajan

Add custom tags to a Delta Live Tables pipeline

Manually edit the JSON configuration file to add custom tags....

Last updated: February 24th, 2023 by John.Lourdu

Update notification settings for jobs with the Jobs API

You can use the Jobs API to add email notifications to some, or all, of the jobs in your workspace....

Last updated: March 17th, 2023 by manoj.hegde

Spark image download failure error message

Learn how to troubleshoot the Spark image download failure error message....

Last updated: April 17th, 2023 by laila.haddad

Python kernel is unresponsive error message

Learn how to identify and troubleshoot the cause of an unresponsive Python kernel error....

Last updated: July 17th, 2023 by laila.haddad

Stop all scheduled jobs

Use the included sample code to stop all of your scheduled jobs in the workspace....

Last updated: June 7th, 2023 by simran.arora

Cluster terminates automatically even if operation or command is still executing

Consider using Databricks Jobs for long-running operations. ...

Last updated: September 12th, 2024 by raahat.varma

Permissions error when trying to run job clusters

Ensure that the service principal has the 'Service Principal User' role....

Last updated: September 12th, 2024 by dayanand.devarapalli

Idle clusters causing inefficient resource use and increased costs

Set file arrival triggers....

Last updated: September 12th, 2024 by lucas.rocha

Error when trying to create more new jobs than the limit quota

Confirm the amount of jobs you have in your workspace, then identify and delete the jobs you do not need....

Last updated: September 9th, 2024 by jairo.prado

Databricks cannot access a notebook in GitHub

Check your file type or GitHub credentials and permissions....

Last updated: September 12th, 2024 by david.vega

Unable to pass a param string value of more than 65,535 characters in a workflow using a JAR in a job

Pass the param using a text file in Workspace FileSystem (WSFS). ...

Last updated: October 14th, 2024 by shubham.bhusate

Trigger a job as a specific user with "Run As"

Use the UI or API to run a job as a specific user....

Last updated: October 18th, 2024 by simran.arora

Need to see job creator when investigating a job but can only see service principal

Leverage the API to retrieve job creator details....

Last updated: December 12th, 2024 by raahat.varma

'Cluster does not support jobs workload' error during notebook or job run

Use a cluster policy that allows the dbutils.notebooks.run API, or run the code directly within a notebook to avoid the API. ...

Last updated: December 20th, 2024 by girish.sharma

Getting NullPointerException when using dbutils.secrets.get in jar jobs

Include the necessary dependencies for dbutils....

Last updated: December 20th, 2024 by girish.sharma

Filter condition in the for each task type not filtering correctly

Use param = :Param instead. ...

Last updated: January 20th, 2025 by nikhil.jain

Jobs running longer than expected with 'Metastore_Down' events in event log

Run the VACUUM command to remove stale files, adjust the catalog update thread pool size in Databricks Runtime 14.3 LTS and above, or for read-only metastore databases, disable Delta catalog updates. ...

Last updated: February 7th, 2025 by manikandan.ganesan

Previously working jobs now failing to execute with METASTORE_DOES_NOT_EXIST error

Only use the Databricks update job API to update the job cluster....

Last updated: January 30th, 2025 by zhengxian.huang

“No module named” error for dependent libraries within a job task

Set the libraries at the compute level for all-purpose compute, share libraries between tasks in non-serverless job clusters, or select from other options provided....

Last updated: February 26th, 2025 by david.vega

Job to insert Parquet file to table fails with error (FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH)

Fix the Parquet file’s schema to match the table’s schema....

Last updated: April 8th, 2025 by nikhil.jain

String aggregation queries failing with data type mismatch error on serverless compute

Use explicit casting or disable ANSI_MODE....

Last updated: April 8th, 2025 by nikhil.jain

Job with string data failing with [CAST_OVERFLOW_IN_TABLE_INSERT] overflow error

Change the data type of the source to an equivalent type as the target....

Last updated: April 8th, 2025 by nikhil.jain

How to fetch the CREATE job JSON using an API call instead of the UI

Use the ‘settings’ key in the ‘/api/2.1/jobs/get’ API endpoint....

Last updated: April 9th, 2025 by Vidhi Khaitan

JDBC write operation fails with HiveSQLException error: The background threadpool cannot accept new task for execution

Change the spark.hive.server2.async.exec.threads, spark.hive.server2.async.exec.wait.queue.size, and spark.hive.server2.async.exec.keepalive.time configs to handle more concurrent asynchronous queries. ...

Last updated: April 28th, 2025 by manikandan.ganesan

Error “number of currently active jobs exceeds hard limit of spark.databricks.maxActiveJobs” when trying to run an API request

Optimize job design first to the extent possible, then change the spark.databricks.maxActiveJobs setting to N, depending on your needs. ...

Last updated: April 28th, 2025 by saritha.shivakumar

Can’t edit jobs created using Databricks Asset Bundles (DABs) using the UI

Disconnect the job from the source to make the job editable in the UI, or programmatically change the edit_mode field to “EDITABLE”....

Last updated: April 30th, 2025 by kevin.salas

Contact Us

If you still have questions or prefer to get help directly from an agent, please submit a request. We’ll get back to you as soon as possible.

Please enter the details of your request. A member of our support staff will respond as soon as possible.


© Databricks 2022-2025. All rights reserved. Apache, Apache Spark, Spark, and the Spark logo are trademarks of the Apache Software Foundation.

Send us feedback | Privacy Notice (Updated) | Terms of Use | Your Privacy Choices | Your California Privacy Rights Privacy Rights icon


Knowledge Base Software powered by Helpjuice

Definition by Author

0
0