Convert Python datetime object to string
There are multiple ways to display date and time values with Python, however not all of them are easy to read.
For example, when you collect a timestamp column from a DataFrame and save it as a Python variable, the value is stored as a datetime object. If you are not familiar with the datetime object format, it is not as easy to read as the common YYYY-MM-DD HH:MM:SS format.
If you wanted to print the date and time, or maybe use it for timestamp validation, you can convert the datetime object to a string. This automatically converts the datetime object into a common time format.
In this article, we show you how to display the timestamp as a column value, before converting it to a datetime object, and finally, a string value.
Display timestamp as a column value
To display the current timestamp as a column value, you should call current_timestamp().
This provides the date and time as of the moment it is called.
from pyspark.sql.functions import *
display(spark.range(1).withColumn("date",current_timestamp()).select("date"))
Sample output:
Assign timestamp to datetime object
Instead of displaying the date and time in a column, you can assign it to a variable.
mydate = spark.range(1).withColumn("date",current_timestamp()).select("date").collect()[0][0]
Once this assignment is made, you can call the variable to display the stored date and time value as a datetime object.
mydate
Sample output:
datetime.datetime(2021, 6, 25, 11, 0, 56, 813000)
Note
The date and time is current as of the moment it is assigned to the variable as a datetime object, but the datetime object value is static unless a new value is assigned.