Problem

While working with the Databricks API on an active and running cluster, the last_activity_time attribute in the /api/2.0/clusters/list shows an incorrect timestamp. 

Cause

The last_activity_time attribute is deprecated and is no longer supported or documented since the 3.15 (2022) release of Databricks. 

The last_activity_time is updated on a best-effort basis and may not always reflect the real-time status of the cluster. It is calculated by the Chauffeur Service when its state is requested. If the cluster is not idle, it reports 0 milliseconds as the time since the last activity. If it is idle, the last activity time is the maximum between the lastDriverActivityTimeMs, the lastProgressUpdateTime, and the last update time from each executor. This calculation may not always be accurate, leading to the observed discrepancies.

Solution

Use the cluster auto-termination feature to manage cluster termination based on inactivity. This feature is designed to eliminate the need for manual monitoring of activity timestamps.

For more information on configuring and using the auto-termination feature, refer to the Manage compute (AWS | Azure | GCP) documentation.