Problem
Notebook or job execution stops and returns either of the following errors:
Run result unavailable: job failed with error message Context ExecutionContextId(1731742567765160237) is disconnected.
Can’t attach this notebook because the cluster has reached the attached notebook limit. Detach a notebook and retry.
Cause
When you attach a notebook to a cluster, Databricks creates an execution context (AWS | Azure). If there are too many notebooks attached to a cluster or too many jobs are created, at some point the cluster reaches its maximum threshold limit of 145 execution contexts, and Databricks returns an error.
Solution
Configure context auto-eviction (AWS | Azure), which allows Databricks to remove (evict) idle execution contexts. Additionally, from the pipeline and ETL design perspective, you can avoid this issue by using:
- Fewer notebooks to reduce the number of execution contexts that are created.
- A job cluster instead of an interactive cluster. If the use case permits, submit notebooks or jars as jobs.