Hyperopt fails with maxNumConcurrentTasks error


You are tuning machine learning parameters using Hyperopt when your job fails with a py4j.Py4JException: Method maxNumConcurrentTasks([]) does not exist error.

You are using a Databricks Runtime for Machine Learning (Databricks Runtime ML) cluster.


Databricks Runtime ML has a compatible version of Hyperopt pre-installed.

If you manually install a second version of Hyperopt, it causes a conflict.


Do not install Hyperopt on Databricks Runtime ML clusters.