Delta Live Tables (DLT) cluster keeps running after pipeline completes

Switch your DLT pipeline to production mode.

Written by mounika.tarigopula

Last published at: January 31st, 2025

Problem

Your Delta Live Tables (DLT) cluster remains active for an extended period, even after the associated DLT pipeline has completed its execution. 

 

Cause

The DLT pipeline is in development mode. Development mode configuration keeps a cluster alive for two hours after an update completes to facilitate quick iteration and testing. 

This behavior is expected and is part of the default settings for development mode in DLT pipelines.

 

Solution

Set the pipeline to Production mode, which does not keep the cluster alive after the update termination.

  1. Click Delta Live Tables in the lefthand menu. 
  2. Select the pipeline to change. 
  3. Click the Development/Production toggle on the right side of the UI to switch to Production. 

If you continue to use development mode, set the pipelines.clusterShutdown.delay parameter to your preferred duration. This allows for customization of the cluster's shutdown behavior based on specific needs.

For detailed instructions on changing DLT pipeline settings, refer to the Configure pipeline settings for Delta Live Tables (AWS | Azure | GCP) documentation.