Recursive references in Avro schema are not allowed

Apache Avro data sources cannot have recursive references in the schema when used with Spark.

Written by saikrishna.pujari

Last published at: December 1st, 2022


Apache Spark returns an error when trying to read from an Apache Avro data source if the Avro schema has a recursive reference.

Found recursive reference in Avro schema, which can not be processed by Spark


Spark SQL does not support recursive references in an Avro data source because it is impossible to convert the schema to StructType.

Review the [SPARK-25718][SQL]Detect recursive reference in Avro schema and throw exception pull request for more information.


You must avoid using recursive references in your Avro schema.

Test for recursive references

You can test your Avro schema for recursive references with SchemaConverters.toSqlType(<avro-schema>).


import org.apache.spark.sql.avro.SchemaConverters

If the Avro schema contains recursive references, SchemaConverters.toSqlType returns an error.


  1. Create an Avro schema with a recursive reference.
    import org.apache.avro.Schema
    val schema = new Schema.Parser().parse("""{
      "type": "record",
      "name": "LongList",
      "aliases": ["LinkedLongs"],                     
      "fields" : [
        {"name": "value", "type": "long"},             
        {"name": "next", "type": ["null", "LongList"]} 
  2. Test the schema with SchemaConverters.toSqlType.
    import org.apache.spark.sql.avro.SchemaConverters
  3. It returns an IncompatibleSchemaExceptionerror.
    IncompatibleSchemaException: Found recursive reference in Avro schema, which can not be processed by Spark: {  "type" : "record",  "name" : "LongList",  "fields" : [ {    "name" : "value",    "type" : "long"  }, {    "name" : "next",    "type" : [ "null", "LongList" ]  } ],  "aliases" : [ "LinkedLongs" ] }

Was this article helpful?