Databricks spark-submit jobs appear to “hang” and clusters do not auto-terminate

Problem  Databricks  spark-submit   jobs appear to “hang,” either after the user class’s main method completes (regardless of success or failure, for Java / Scala jobs), or upon a Python script exit (for Python jobs). Clusters do not auto-terminate in this case.  Cause  For Java, Scala, and Python programs, Spark does not automatically call  System....

0 min reading time