Python commands fail on high concurrency clusters

Problem

You are attempting to run Python commands on a high concurrency cluster.

All Python commands fail with a WARN error message.

WARN PythonDriverWrapper: Failed to start repl ReplId-61bef-9fc33-1f8f6-2
ExitCodeException exitCode=1: chown: invalid user: ‘spark-9fcdf4d2-045d-4f3b-9293-0f’

Cause

Both spark.databricks.pyspark.enableProcessIsolation true and spark.databricks.session.share true are set in the Apache Spark configuration on the cluster.

These two Spark properties conflict with each other and prevent the cluster from running Python commands.

Solution

You can only have one of these two Spark properties enabled on your cluster at a time.

You must choose process isolation or a Spark shared session based on your needs. Disable the other option.