After you install a Python library (via the cluster UI or by using pip), your Apache Spark jobs fail with an Environment directory not found error message.
org.apache.spark.SparkException: Environment directory not found at /local_disk0/.ephemeral_nfs/cluster_libraries/python
Libraries are installed on a Network File System (NFS) on the cluster's driver node. If any security group rules prevent the workers from communicating with the NFS server, Spark commands cannot resolve the Python executable path.
You should make sure that your security groups are configured with appropriate security rules (AWS | Azure | GCP).