Problem

When trying to run a job, the cluster fails. You see the following message in the job output UI. 

run failed with error message
Could not reach driver of cluster <CLUSTER-ID>.

When you check the driver logs, you see more detail on the failure. 

AttributeError: _ARRAY_API not found

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2

Cause

Databricks Runtime 14.3 LTS and 15.4 LTS come with NumPy 1.23.5 pre-installed. The library awswrangler version 3.10.0 requires NumPy 2.1.2. When installing dependencies, awswrangler forces NumPy to upgrade. The change in NumPy causes the cluster startup to fail.   

Solution

Databricks Runtime versions 14.3 LTS and 15.4 LTS are designed to work with NumPy 1.23.5. To ensure compatibility across common dependencies with the awswrangler library, use any version equal to or below 3.9.1. These versions of awswrangler work with NumPy version 1.23.5.

For more information, review the Databricks Runtime release notes versions and compatibility documentation.