You are tuning machine learning parameters using Hyperopt when your job fails with a
py4j.Py4JException: Method maxNumConcurrentTasks() does not exist error.
You are using a Databricks Runtime for Machine Learning (Databricks Runtime ML) cluster.
Databricks Runtime ML has a compatible version of Hyperopt pre-installed.
If you manually install a second version of Hyperopt, it causes a conflict.