VACUUM operations not performing even after enabling predictive optimization

Create a new deletion operation to trigger VACUUM after enabling predictive optimization.

Written by Sahil Singh

Last published at: September 12th, 2024

Problem 

Predictive optimization does not trigger VACUUM even though it is enabled and there are old unreferenced files present in the Delta table. 

Cause

Predictive optimization cannot see previously deleted data. It starts to collect metrics of unreferenced files only after predictive optimization is enabled.

Solution

Create a new deletion operation to trigger VACUUM after enabling predictive optimization. 

  1. Commit any operation that could result in unreferenced files to the table, such as DELETE or MERGE. This creates unreferenced file information to pass to predictive optimization as eligible for VACUUM.
  2. Wait for deletedFileRetentionDuration to pass. 
  3. Check that VACUUM has been executed. 

For more information, please review the Predictive optimization for Delta Lake (AWS | Azure) documentation.