“No module named” error for dependent libraries within a job task

Set the libraries at the compute level for all-purpose compute, share libraries between tasks in non-serverless job clusters, or select from other options provided.

Written by david.vega

Last published at: February 26th, 2025

Problem

When defining libraries for workflows using job clusters, you receive an error below despite the library being present.

 

ModuleNotFoundError: No module named '<library-name>'

 

Cause

The library is present, but it is defined in another task in the same job through the Dependent libraries field. Tasks don’t inherit libraries from other tasks.

 

Solution

If you use all-purpose compute, install libraries at the compute level rather than the job level, so that the libraries can be shared across all tasks automatically.

 

For individual job computes, in the UI, when non-serverless job clusters are used for dependencies between the same tasks, you can share the library dependencies between them. 

  1. Open the first parent task.
  2. Go to the Dependent libraries field and include all library dependencies.
  3. Move to the child/dependent tasks.
  4. File the parent task in the field Depends on. 

 

The dependent tasks now also have all libraries set at the parent task. 

Other options

  1. Use notebook-scoped libraries. For details, review the Libraries (AWS | Azure | GCP) documentation.
  2. Use Terraform, where the dependencies can be defined as variables. For details, review the Create clusters, notebooks, and jobs with Terraform (AWS | Azure | GCP) documentation. 
  3. Use Databricks Asset Bundles (DAB), which allow you to reuse the dependencies between job tasks. For details, review the Databricks Asset Bundles library dependencies (AWS | Azure | GCP) documentation. 

If you use serverless compute, you can use notebook-scoped libraries or use the Environment and Libraries field to select, edit, or add a new environment. Note this will be required in every task. For more information, review the Install Notebook dependencies (AWS | Azure | GCP) documentation.