Decimal$DecimalIsFractional
assertion error
Problem
You are running a job on Databricks Runtime 7.x or above when you get a java.lang.AssertionError: assertion failed: Decimal$DecimalIsFractional
error message.
Example stack trace:
java.lang.AssertionError: assertion failed:
Decimal$DecimalIsFractional
while compiling: <notebook>
during phase: globalPhase=terminal, enteringPhase=jvm
library version: version 2.12.10
compiler version: version 2.12.10
reconstructed args: -deprecation -classpath .....
*** WARNING: skipped 126593 bytes of output ***
This error message only occurs on the first run of your notebook. Subsequent runs complete without error.
Cause
There are two common use cases that can trigger this error message.
- Cause 1: You are trying to use the
round()
function on a decimal column that contains null values in a notebook. - Cause 2: You are casting a double column to a decimal column in a notebook.
This example code can be used to reproduce the error:
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.Column
//Sample data with decimal values
val updateData = Seq(
Row(BigDecimal.decimal(123.456), 123.456),
Row(BigDecimal.decimal(123.456), 123.456))
val updateSchema = List(
StructField("amt_decimal", DecimalType(14,3), true),
StructField("amt_double", DoubleType, true))
val testDF = spark.createDataFrame(
spark.sparkContext.parallelize(updateData),
StructType(updateSchema)
)
// Cause 1:
// round() on the Decimal column reproduces the error
testDF.withColumn("round_amt_decimal",round(col("amt_decimal"),2)).show()
// Cause 2:
// CAST() on the Double column to Decimal reproduces the error
testDF.createOrReplaceTempView("dec_table")
spark.sql("select CAST(amt_double AS DECIMAL(3,3)) AS dec_col from dec_table").show()