Problem
When using Lakehouse Federation with Databricks Runtime 14.3 LTS or above, executing a federated query on a MySQL table that casts a TINYINT
column to another data type results in the following SQLException
error.
java.sql.SQLException: Out of range value for column 'seq': value 128 is not in class java.lang.Byte range.
Cause
In Databricks Runtime versions 14.3 LTS and above, TINYINT
is interpreted as a BYTE
, representing 1-byte signed integers. When the values in the TINYINT
column exceed the range of -128
to 127
, the query fails because it is out of range.
Solution
Set the following Apache Spark property in your compute settings to convert TINYINT
columns to SHORT
type. Using SHORT
type increases the value range so you can successfully cast TINYINT
columns.
spark.databricks.sql.mysql.mapUnsignedTinyIntToShort = true
For details on how to apply Spark configs, refer to the “Spark configuration” section of the Compute configuration reference (AWS | Azure | GCP) documentation.
Alternatively, you can run the following command in a notebook.
spark.conf.set("spark.databricks.sql.mysql.mapUnsignedTinyIntToShort", "true")