ADLS and WASB writes are being throttled

Learn how to resolve a "files and folders are being created at too high a rate" ADLS or WASB storage error.

Written by Adam Pavlacka

Last published at: December 9th, 2022

Problem

When accessing data stored on Azure data Lake Storage (ADLS) Windows Azure Storage Blobs (WASB) requests start timing out. You may see an error message indicating that storage is being accessed at too high a rate.

Files and folders are being created at too high a rate

Cause

Azure storage subscriptions have a limit on how many files and folders can be accessed over time. If too many requests are made in a given time frame, your account will be subject to throttling in order to keep the requests under the subscription limit.

Solution

To resolve this issue you can either increase the storage limits on your Azure subscription or optimize your Spark code to reduce the number of files created.