Cannot import timestamp_millis or unix_millis

Problem

You are trying to import timestamp_millis or unix_millis into a Scala notebook, but get an error message.

import org.apache.spark.sql.functions.{timestamp_millis, unix_millis}
error: value timestamp_millis is not a member of object org.apache.spark.sql.functions
import org.apache.spark.sql.functions.{timestamp_millis, unix_millis}

Cause

The functions timestamp_millis and unix_millis are not available in the Apache Spark DataFrame API.

These functions are specific to SQL and are included in Spark 3.1.1 and above.

Solution

You need to use selectExpr() with timestamp_millis or unix_millis if you want to use either one of them with a DataFrame.

selectExpr() takes a set of SQL expressions and runs them.

For example, this sample code returns an error message when run.

import sqlContext.implicits._
val df = Seq(
 (1, "First Value"),
 (2, "Second Value")
).toDF("int_column", "string_column")

import org.apache.spark.sql.functions.{unix_millis}
import org.apache.spark.sql.functions.col
df.select(unix_millis(col("int_column"))).show()
error: value unix_millis is not a member of object org.apache.spark.sql.functions
import org.apache.spark.sql.functions.{unix_millis}

While this sample code, using selectExpr(), successfully returns timestamp values.

import org.apache.spark.sql.functions._
import sqlContext.implicits._
val ndf = Seq(
 (1, "First Value"),
 (2, "Second Value")
).toDF("int_column", "string_column")

display(ndf.selectExpr("timestamp_millis(int_column)"))

timestamp_millis returns results with selectExpr.

Example notebook

Cannot import timestamp_millis or unix_millis example

Open notebook in new tab