Problem

You’re deploying multiple jobs using Databricks Asset Bundles (DABs), particularly notebooks that utilize Auto Loader. During the deployment process, you executing the following command.
 

databricks bundle run -t <env> <your-job>

The terminal returns the following status message. 

2025-01-13 15:58:10 "<your-job>" RUNNING

Following this message, the terminal appears to hang indefinitely. 

The terminal seeming to hang indefinitely leads to uncertainty about whether the job has successfully started or the command is waiting for the job to complete. 

Cause

By default, the databricks bundle run command performs two primary actions sequentially:

1. Initiates the deployment of the job within the specified environment.
2. Waits for the job to finish executing before returning control to the terminal.

For standard jobs that have a defined completion point, this behavior is typically acceptable. However, in scenarios with continuous jobs, which are designed to run perpetually and may have unlimited retries, there is no natural conclusion. The lack of natural conclusion leads the databricks bundle run command to wait indefinitely, causing the terminal to appear unresponsive.

Solution

Use the --no-wait flag with the databricks bundle run command. This flag alters the command's behavior to start the job and immediately return control to the terminal without waiting for the job to finish.

databricks bundle run -t <env> my_job --no-wait

Benefits of using --no-wait

• The command initiates the job and returns control, providing immediate confirmation that the job has started.
• The waiting period associated with job completion is eliminated, streamlining the deployment process.
• The deployment of multiple continuous jobs is facilitated without the risk of terminal hang-ups or prolonged wait times.
• Less ambiguity about the job's status post-deployment since the command does not linger waiting for a process that is intended to run indefinitely.