Problem

When you are trying to set a custom PYTHONPATH environment variable in a cluster-scoped init script, but the values are overridden at driver startup. 

Cause

Setting a custom PYTHONPATH in an init scripts does not work and is not supported.

Additionally, you cannot set a custom PYTHONPATH when using Databricks Container Services.

Solution

You should not try to set a custom PYTHONPATH.

If you need to use custom Python libraries or modules, install the required files to pre-existing directories that are included in the cluster's PYTHONPATH.

This sample code lists all directories in the cluster's PYTHONPATH.

%python

import sys
print(sys.path)