Decimal$DecimalIsFractional assertion error

Problem

You are running a job on Databricks Runtime 7.x or above when you get a java.lang.AssertionError: assertion failed: Decimal$DecimalIsFractional error message.

Example stack trace:

java.lang.AssertionError: assertion failed:
 Decimal$DecimalIsFractional
  while compiling: <notebook>
   during phase: globalPhase=terminal, enteringPhase=jvm
  library version: version 2.12.10
 compiler version: version 2.12.10
reconstructed args: -deprecation -classpath .....
*** WARNING: skipped 126593 bytes of output ***

This error message only occurs on the first run of your notebook. Subsequent runs complete without error.

Cause

There are two common use cases that can trigger this error message.

• Cause 1: You are trying to use the round() function on a decimal column that contains null values in a notebook.
• Cause 2: You are casting a double column to a decimal column in a notebook.

This example code can be used to reproduce the error:

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.Column

//Sample data with decimal values

val updateData =  Seq(
                     Row(BigDecimal.decimal(123.456), 123.456),
                     Row(BigDecimal.decimal(123.456), 123.456))

val updateSchema = List(
                    StructField("amt_decimal", DecimalType(14,3), true),
                    StructField("amt_double", DoubleType, true))

val testDF =  spark.createDataFrame(
  spark.sparkContext.parallelize(updateData),
  StructType(updateSchema)
)

// Cause 1:
// round() on the Decimal column reproduces the error

testDF.withColumn("round_amt_decimal",round(col("amt_decimal"),2)).show()

// Cause 2:
// CAST() on the Double column to Decimal reproduces the error

testDF.createOrReplaceTempView("dec_table")
spark.sql("select CAST(amt_double AS DECIMAL(3,3)) AS dec_col from dec_table").show()

Solution

This is a known issue and can be safely ignored.

The error message does not halt the notebook run and it should not result in any data loss.