Problem:

You have an automated job that requires the use of external Maven libraries.

You created a separate cluster with the libraries installed, but it incurs idle time, resulting in unnecessary costs.

Solution:

To add libraries to a job cluster, follow these steps:

1. Create a job in Databricks.
2. Click Add next to dependent libraries.
3. In the pop-up window, add the required libraries.

To reduce idle time in a job cluster, you have two options:

• Opt out of auto termination by clearing the Auto Termination checkbox.
• Specify an inactivity period of 0.

Databricks recommends running jobs on a job cluster, rather than an interactive cluster with auto termination.

Job clusters automatically terminate once the job completes, ensuring efficient resource utilization.