Auto Loader does not pick up files for processing when uploading via an Azure function

Modify the code in the Azure function to add the 'CLOSE' parameter.

Written by potnuru.siva

Last published at: August 30th, 2024

Problem

When you upload files to a source location using an Azure function in Auto Loader, Auto Loader does not pick up the files for processing. The files are also not available in the queue, which Auto Loader sets automatically. The process does work, however, with manual intervention. 

Cause

Auto Loader listens for the ‘FlushWithClose’ event to process a file in file notification mode. The Azure function used for uploading files to a source location does not set the Close property in File Flush options, so EventBridge does not send the necessary event.  

Solution

  • Modify your Azure function to set the File Flush option with the 'CLOSE' parameter.
  • Verify that the FlushWithClose event is generated in the Azure Queue after uploading the files to the source location.
  • Use Directory listing mode until the file notification issue is resolved, then switch to file notification.
  • Monitor the diagnostic logging on the storage Blob and Queue to identify any issues with the file upload process.

 

For more information, please review the What is Auto Loader file notification mode? and DataLakeFileFlushOptions.Close Property documentation.