EBS leaked volumes

Learn how to handle leaked EBS volumes that are not automatically deleted.

Written by Adam Pavlacka

Last published at: March 4th, 2022

Problem

After a cluster is terminated on AWS, some EBS volumes are not deleted automatically. These stray, unattached EBS volumes are often referred to as “leaked” volumes.

Cause

Databricks always sets DeletionOnTermination=true for the EBS volumes it creates when it launches clusters. Therefore, whenever a cluster instance is terminated, AWS should automatically delete all EBS volumes associated with the cluster.

However, there are two circumstances in which you might see unattached EBS volumes:

  1. There is a delay of 5 minutes between a cluster termination and the termination of its instances. Hence you might see unused volumes for a few minutes. These require no action on your part.
  2. Databricks depends on AWS to clean up EBS volumes when a cluster instance is terminated. Unfortunately, if an instance dies during start-up, the EBS volumes never get attached to the instance, and AWS never cleans up the volumes. The AWS support team has confirmed that they are working on fix for this bug, but they have not provided an ETA.

Solution

Databricks automatically logs and deletes leaked EBS volumes. You can contact Databricks customer support to request a list of leaked EBS volumes that have been deleted from your account.

Delete

Note

Databricks needs your permission to describe and delete volumes. If you created your Databricks account prior to version 2.44 (that is, before Apr 27, 2017) and want leaked EBS volumes to be deleted automatically, add volume permissions to the IAM role or keys used to create your account. In particular, you must add the permissions ec2:DeleteVolume and ec2:DescribeVolumes. For the complete list of permissions and instructions on how to update your existing IAM role or keys, see AWS Account.