Problem

When using an Apache Spark job or Structured Streaming job to read a text file containing a special ASCII character "\x1C", the job fails with the following error.

[INVALID_DELIMITER_VALUE.UNSUPPORTED_SPECIAL_CHARACTER] Invalid value for delimiter. Unsupported special character for delimiter: \x. SQLSTATE: 42602 

Cause

The error occurs because Spark treats \x as a special escape sequence in the delimiter. 

In general, CSV or text delimiters must be a single character, and Spark only supports specific single-character escapes like \t, \n, \r, \b, \f, or Unicode escapes such as \u0001. Spark does not understand \x1C. 

For more information, review the “Data Source Option” section of the CSV Files Spark documentation.

Solution

Databricks recommends using a Unicode escape "\u001C" instead of  "\x1C" when reading a file. 

Example

spark.read
    .format("csv")
    .option("header", "false")
    .option("delimiter", '\u001C')
    .load(base_path)