There are multiple ways to display date and time values with Python, however not all of them are easy to read.
For example, when you collect a timestamp column from a DataFrame and save it as a Python variable, the value is stored as a datetime object. If you are not familiar with the datetime object format, it is not as easy to read as the common YYYY-MM-DD HH:MM:SS format.
If you wanted to print the date and time, or maybe use it for timestamp validation, you can convert the datetime object to a string. This automatically converts the datetime object into a common time format.
In this article, we show you how to display the timestamp as a column value, before converting it to a datetime object, and finally, a string value.
To display the current timestamp as a column value, you should call
This provides the date and time as of the moment it is called.
from pyspark.sql.functions import * display(spark.range(1).withColumn("date",current_timestamp()).select("date"))
Instead of displaying the date and time in a column, you can assign it to a variable.
mydate = spark.range(1).withColumn("date",current_timestamp()).select("date").collect()
Once this assignment is made, you can call the variable to display the stored date and time value as a datetime object.
datetime.datetime(2021, 6, 25, 11, 0, 56, 813000)
The date and time is current as of the moment it is assigned to the variable as a datetime object, but the datetime object value is static unless a new value is assigned.