Problem

You are attempting to append data to a file saved on an external storage mount point and are getting an error message: OSError: [Errno 95] Operation not supported.

The error occurs when trying to append to a file from both Python and R.

Cause

Direct appends and random writes are not supported in FUSE v2, which is available in Databricks Runtime 6.0 and above. This is by design.

The underlying storage that is mounted to DBFS does not support append. This means that Databricks would have to download the data, run the append, and reupload the data in order to support the command. This works for small files, but quickly becomes an issue as file size increases. Because the DBFS mount is shared between driver and worker nodes, appending to a file from multiple nodes can cause data corruption.

Solution

As a workaround, you should run your append on a local disk, such as /tmp, and move the entire file at the end of the operation.

If you need to perform cross-session appends, please contact your account team to discuss enabling an NFS mount on your clusters.