Cluster terminates automatically even if operation or command is still executing

Problem While executing a long-running operation or command within a notebook on an interactive cluster, you notice the cluster terminates automatically. This disrupts your workflows, results in incomplete processes, and requires the need to restart the operation. Cause The Workspace File System (WSFS) token for interactive sessions has a 36-hour ti...

0 min reading time

Apache Spark is configured to suppress INFO statements but they overwhelm logs anyway

Problem You receive INFO statements despite configuring the Apache Spark settings to suppress INFO and give specific WARN statements. This issue is observed even after setting the ' py4j' logger to WARN and configuring the logging in the Spark config in the Databricks UI to WARN .  The problem persists, leading to an overflow of INFO logs, which can...

0 min reading time