Problem

You are trying to install the PyStan PyPi package on a Databricks Runtime 6.4 Extended Support cluster and get a ManagedLibraryInstallFailed error message.

java.lang.RuntimeException: ManagedLibraryInstallFailed: org.apache.spark.SparkException: Process List(/databricks/python/bin/pip, install, pystan, --disable-pip-version-check) exited with code 1.   Could not find a version that satisfies the requirement httpstan<4.5,>=4.4 (from pystan) (from versions: 0.1.0, 0.1.1, 0.2.3, 0.2.5, 0.3.0, 0.3.1, 0.4.0, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.5, 0.7.6, 0.8.0, 0.9.0, 0.10.1, 1.0.0)
No matching distribution found for httpstan<4.5,>=4.4 (from pystan)
 for library:PythonPyPiPkgId(pystan,None,None,List()),isSharedLibrary=false

Cause

When you install PyStan via PyPi, it attempts to install the latest version.

PyStan 3.0.0 and above are not compatible with Databricks Runtime 6.4 Extended Support.

Solution

You should use pystan version 2.19.1.1 on Databricks Runtime 6.4 Extended Support.

Specify pystan==2.19.1.1 when you install the library on your cluster (AWS | Azure). This is the most recent version that is compatible with Databricks Runtime 6.4 Extended Support.

If you require pystan version 3.0.0 or above, you should upgrade to Databricks Runtime 7.3 LTS or above.