Problem: Job Failure Due to Azure Data Lake Storage (ADLS) CREATE Limits
Problem
When you run a job that involves creating files in Azure Data Lake Storage (ADLS), either Gen1 or Gen2, the following exception occurs:
Caused by: java.io.IOException: CREATE failed with error 0x83090c25 (Files and folders are being created at too high a rate). [745c5836-264e-470c-9c90-c605f1c100f5] failed with error 0x83090c25 (Files and folders are being created at too high a rate). [2019-04-12T10:06:43.1117197-07:00] [ServerRequestId:745c5836-264e-470c-9c90-c605f1c100f5]
at com.microsoft.azure.datalake.store.ADLStoreClient.getRemoteException(ADLStoreClient.java:1191)
at com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1154)
at com.microsoft.azure.datalake.store.ADLStoreClient.createFile(ADLStoreClient.java:281)
at com.databricks.adl.AdlFileSystem.create(AdlFileSystem.java:348)
at com.databricks.spark.metrics.FileSystemWithMetrics.create(FileSystemWithMetrics.scala:280)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2$$anonfun$create$1$$anonfun$apply$10$$anonfun$apply$11.apply(DatabricksFileSystemV2.scala:483)
Cause
Each ADLS subscription level has a limit on the number of files that can be created per unit of time, although the limits may differ depending on whether you are using ADLS Gen1 or Gen2. When the limit is exceeded, file creation is throttled, and the job fails.
Potential causes for this error include:
- Your application creates a large number of small files.
- External applications create a large number of files.
- The current limit for the subscription is too low.