You attempt to read a Delta table from mounted storage and get a FileReadException error.
FileReadException: Error while reading file abfss:REDACTED@REDACTED.dfs.core.windows.net/REDACTED/REDACTED/REDACTED/REDACTED/PARTITION=REDACTED/part-00042-0725ec45-5c32-412a-ab27-5bc88c058773.c000.snappy.parquet. A file referenced in the transaction log cannot be found. This occurs when data has been manually deleted from the file system rather than using the table `DELETE` statement. For more information, see https://docs.microsoft.com/azure/databricks/delta/delta-intro#frequently-asked-questions Caused by: FileNotFoundException: Operation failed: 'The specified path does not exist.', 404, HEAD, https:// REDACTED.dfs.core.windows.net/ REDACTED/ REDACTED/REDACTED/REDACTED/PARTITION=REDACTED/part-00042-0725ec45-5c32-412a-ab27-5bc88c058773.c000.snappy.parquet?upn=false&action=getStatus&timeout=90
FileReadException errors occur when the underlying data does not exist. The most common cause is manual deletion.
If the underlying data was not manually deleted, the mount point for the storage blob was removed and recreated while the cluster was writing to the Delta table.
Delta Lake does not fail a table write if the location is removed while the data write is ongoing. Instead, a new folder is created in the default storage account of the workspace, with the same path as the removed mount. Data continues to be written in that location.
If the mount is recreated before the write operation is finished, and the Delta transaction logs are made available again, Delta updates the transaction logs and the write is considered successful. When this happens, data files written to the default storage account while the mount was deleted are not accessible, as the path currently references the mounted storage account location.
You can restore the missing data in one of two ways.
- Repair the Delta table and add the missing data back with a custom job.
Use FSCK to repair the table.
%sql FSCK REPAIR TABLE <table-name>
- Rewrite the missing data with a custom job. This option is a good choice if you can re-run the last job without risking duplicate data.
- Manually recover the missing files.
- Verify that there are no active jobs reading or writing to the mounted storage account that contains the Delta table.
- Unmount the mount path. This allows you to access the /mnt/<path-to-table>directory in the default storage account.
- Use dbutils.fs.mvto move the files located in the table path to a temporary location.
%python dbutils.fs.mv("/mnt/<path-to-table>", "/tmp/tempLocation/", True))
- Recreate the mount point.
%python dbutils.fs.mount(source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/", mount_point = "/mnt/<mount-containing-table>", extra_configs = configs)Review the Access Azure Data Lake Storage Gen2 and blob storage documentation for more information.
- Move the files from the temporary location to the updated Delta table path.
%python dbutils.fs.mv("/tmp/tempLocation", "/mnt/<path-to-table>", True))
If any jobs are reading or writing to the mount point when you attempt a manual recovery you may cause the issue to reoccur. Verify that the mount is not in use before attempting a manual repair.
- Instruct users to get approval before unmounting a storage location.
- If you must unmount a storage location, verify there are no jobs running on the cluster.
- Use dbutils.fs.updateMount to update information about the mount. Do not use unmount and mount to update the mount.
- Use diagnostic logging to identify any possible unmount issues.
- Run production jobs only on job clusters which are not affected by temporary unmount commands while running, unless they run the dbutils.fs.refreshMounts command.
- When running jobs on interactive clusters, add a verification step at the end of a job (such as a count) to check for missing data files. If any are missing an error is triggered immediately.