Hyperopt fails with maxNumConcurrentTasks error

Do NOT install Hyperopt on a Databricks Runtime for Machine Learning cluster.

Written by chetan.kardekar

Last published at: May 16th, 2022


You are tuning machine learning parameters using Hyperopt when your job fails with a py4j.Py4JException: Method maxNumConcurrentTasks([]) does not exist error.

You are using a Databricks Runtime for Machine Learning (Databricks Runtime ML) cluster.


Databricks Runtime ML has a compatible version of Hyperopt pre-installed (AWS | Azure | GCP).

If you manually install a second version of Hyperopt, it causes a conflict.


Do not install Hyperopt on Databricks Runtime ML clusters.

Was this article helpful?