Cannot set a custom PYTHONPATH

Setting a custom PYTHONPATH in an init script or in DCS is not supported.

Written by prakash.jha

Last published at: September 13th, 2022


When you are trying to set a custom PYTHONPATH environment variable in a cluster-scoped init script, but the values are overridden at driver startup. 


Setting a custom PYTHONPATH in an init scripts does not work and is not supported.

Additionally, you cannot set a custom PYTHONPATH when using Databricks Container Services.


You should not try to set a custom PYTHONPATH.

If you need to use custom Python libraries or modules, install the required files to pre-existing directories that are included in the cluster's PYTHONPATH.

This sample code lists all directories in the cluster's PYTHONPATH.


import sys

Was this article helpful?