You have an automated job that requires the use of external Maven libraries.
You created a separate cluster with the libraries installed, but it incurs idle time, resulting in unnecessary costs.
To add libraries to a job cluster, follow these steps:
- Create a job in Databricks.
- Click Add next to dependent libraries.
- In the pop-up window, add the required libraries.
To reduce idle time in a job cluster, you have two options:
- Opt out of auto termination by clearing the Auto Termination checkbox.
- Specify an inactivity period of 0.
Databricks recommends running jobs on a job cluster, rather than an interactive cluster with auto termination.
Job clusters automatically terminate once the job completes, ensuring efficient resource utilization.