Job failure due to Azure Data Lake Storage (ADLS) CREATE limits

Learn what to do when your Databricks job fails due to Azure Data Lake Storage CREATE limits.

Written by Adam Pavlacka

Last published at: May 11th, 2022

Problem

When you run a job that involves creating files in Azure Data Lake Storage (ADLS), either Gen1 or Gen2, the following exception occurs:

Caused by: java.io.IOException: CREATE failed with error 0x83090c25 (Files and folders are being created at too high a rate). [745c5836-264e-470c-9c90-c605f1c100f5] failed with error 0x83090c25 (Files and folders are being created at too high a rate). [2019-04-12T10:06:43.1117197-07:00] [ServerRequestId:745c5836-264e-470c-9c90-c605f1c100f5]
at com.microsoft.azure.datalake.store.ADLStoreClient.getRemoteException(ADLStoreClient.java:1191)
at com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1154)
at com.microsoft.azure.datalake.store.ADLStoreClient.createFile(ADLStoreClient.java:281)
at com.databricks.adl.AdlFileSystem.create(AdlFileSystem.java:348)
at com.databricks.spark.metrics.FileSystemWithMetrics.create(FileSystemWithMetrics.scala:280)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2$$anonfun$create$1$$anonfun$apply$10$$anonfun$apply$11.apply(DatabricksFileSystemV2.scala:483)

Cause

Each ADLS subscription level has a limit on the number of files that can be created per unit of time, although the limits may differ depending on whether you are using ADLS Gen1 or Gen2. When the limit is exceeded, file creation is throttled, and the job fails.

Potential causes for this error include:

  • Your application creates a large number of small files.
  • External applications create a large number of files.
  • The current limit for the subscription is too low.

Solution

If your application or an external application is generating a large number of files, then you need to optimize the application. If the limit on your current subscription is not appropriate for your use case, then contact the Microsoft Azure Team for assistance.