This article explains how to resolve an error that occurs when you read large DBFS-mounted files using local Python APIs.

Problem

If you mount a folder onto dbfs:// and read a file larger than 2GB in a Python API like pandas, you will see following error:

/databricks/python/local/lib/python2.7/site-packages/pandas/parser.so in pandas.parser.TextReader.__cinit__ (pandas/parser.c:3427)()
/databricks/python/local/lib/python2.7/site-packages/pandas/parser.so in pandas.parser.TextReader._setup_parser_source (pandas/parser.c:6883)()
IOError: Initializing from file failed

Cause

The error occurs because one argument in the Python method to read a file is a signed int, the length of the file is an int, and if the object is a file larger than 2GB, the length can be larger than maximum signed int.

Solution

Move the file from dbfs:// to local file system (file://). Then read using the Python API. For example:

1. Copy the file from dbfs:// to file://:

   %fs cp dbfs:/mnt/large_file.csv file:/tmp/large_file.csv

2. Read the file in the pandasAPI:

   %python
   
   import pandas as pd
   pd.read_csv('file:/tmp/large_file.csv',).head()