Problem

When accessing data stored on Azure data Lake Storage (ADLS) Windows Azure Storage Blobs (WASB) requests start timing out. You may see an error message indicating that storage is being accessed at too high a rate.

Files and folders are being created at too high a rate

Cause

Azure storage subscriptions have a limit on how many files and folders can be accessed over time. If too many requests are made in a given time frame, your account will be subject to throttling in order to keep the requests under the subscription limit.

Solution

To resolve this issue you can either increase the storage limits on your Azure subscription or optimize your Spark code to reduce the number of files created.