Problem

You are tuning machine learning parameters using Hyperopt when your job fails with a py4j.Py4JException: Method maxNumConcurrentTasks([]) does not exist error.

You are using a Databricks Runtime for Machine Learning (Databricks Runtime ML) cluster.

Cause

Databricks Runtime ML has a compatible version of Hyperopt pre-installed (AWS | Azure | GCP).

If you manually install a second version of Hyperopt, it causes a conflict.

Solution

Do not install Hyperopt on Databricks Runtime ML clusters.