Problem

You’re working in a notebook on an interactive cluster that reads a file from Workspace files. When you execute a long-running operation or command in the notebook, you notice the command throws a file read error and stops your notebook.

This disrupts your workflows, results in incomplete processes, and requires the need to restart the operation.

Cause

The Workspace File System (WSFS) token for interactive sessions has a 36-hour timeout. 

Solution

Consider using Databricks jobs for long-running operations. 

Databricks jobs have a 30-day timeout, which is more suitable for extensive calculations. To create and run a Databricks job:

1. Navigate to the Databricks workspace and select the Jobs tab.
2. Click on 'Create Job' and configure the job settings, including the notebook to run and the cluster to use.
3. Set the schedule and timeout settings to accommodate the long-running calculation.
4. Save and run the job.

For detailed instructions, refer to the Create and run Databricks Jobs (AWS | Azure | GCP) documentation.