This tutorial will explain various date/timestamp functions(Part 1) available in Pyspark which can be used to perform date/timestamp related operations.

PySpark: Dataframe Date Functions Part 1

• This tutorial will explain various date/timestamp functions(Part 1) available in Pyspark which can be used to perform date/time/timestamp related operations, click on item in the below list and it will take you to the respective section of the page(s):
  • current_timestamp
  • current_date
  • year
  • month
  • dayofmonth
  • dayofyear
  • dayofweek
  • quarter
  • hour
  • minute
  • second
  • last_day
  • next_day
  • date_add
  • add_months
  • datediff
  • months_between

Sample Data: Dataset used in the below examples can be downloaded from here.

df = spark.read.parquet("file:///path_to_files/date_format_example_file.parquet")

df.show()
+---------+----------+-------------------+
|   engine|  dt_field|           ts_field|
+---------+----------+-------------------+
|    spark|2022-11-03|2022-06-06 06:06:06|
|mapreduce|2022-10-25|2022-06-21 13:04:43|
+---------+----------+-------------------+


df.printSchema()
root
 |-- engine: string (nullable = true)
 |-- dt_field: date (nullable = true)
 |-- ts_field: timestamp (nullable = true)

➠ current_timestamp function(): current_timestamp function can be used to get current timestamp.

from pyspark.sql.functions import current_timestamp

df_update = df.select("ts_field", current_timestamp())

df_update.show(truncate=False)
+-------------------+-----------------------+
|ts_field           |current_timestamp()    |
+-------------------+-----------------------+
|2022-06-06 06:06:06|2022-06-14 19:12:20.568|
|2022-06-21 13:04:43|2022-06-14 19:12:20.568|
+-------------------+-----------------------+

➠ current_date function(): current_date function can be used to get current date.

from pyspark.sql.functions import current_date

df_update = df.select("ts_field", current_date())

df_update.show(truncate=False)
+-------------------+--------------+
|ts_field           |current_date()|
+-------------------+--------------+
|2022-06-06 06:06:06|2022-06-14    |
|2022-06-21 13:04:43|2022-06-14    |
+-------------------+--------------+

➠ year function(): year function can be used to extract year from a date or timestamp.

from pyspark.sql.functions import current_date, year

df_update = df.select("dt_field", year("dt_field").alias("year"), current_date(), year(current_date()).alias("year"))

df_update.show(truncate=False)
+----------+----+--------------+----+
|dt_field  |year|current_date()|year|
+----------+----+--------------+----+
|2022-11-03|2022|2022-06-14    |2022|
|2022-10-25|2022|2022-06-14    |2022|
+----------+----+--------------+----+

➠ month function(): month function can be used to extract month from a date or timestamp.

from pyspark.sql.functions import current_date, month

df_update = df.select("dt_field", month("dt_field").alias("month"), current_date(), month(current_date()).alias("month"))

df_update.show(truncate=False)
+----------+-----+--------------+-----+
|dt_field  |month|current_date()|month|
+----------+-----+--------------+-----+
|2022-11-03|11   |2022-06-14    |6    |
|2022-10-25|10   |2022-06-14    |6    |
+----------+-----+--------------+-----+

➠ dayofmonth function(): dayofmonth function can be used to extract day of the month from a date or timestamp.

from pyspark.sql.functions import current_date, dayofmonth

df_update = df.select("dt_field", dayofmonth("dt_field").alias("dayofmonth"), current_date(), dayofmonth(current_date()).alias("dayofmonth"))

df_update.show(truncate=False)
+----------+----------+--------------+----------+
|dt_field  |dayofmonth|current_date()|dayofmonth|
+----------+----------+--------------+----------+
|2022-11-03|3         |2022-06-14    |14        |
|2022-10-25|25        |2022-06-14    |14        |
+----------+----------+--------------+----------+

➠ dayofyear function(): dayofyear function can be used to extract day of the year from a date or timestamp.

from pyspark.sql.functions import current_date, dayofyear

df_update = df.select("ts_field", dayofyear("ts_field").alias("dayofyear"), current_date(), dayofyear(current_date()).alias("dayofyear"))

df_update.show(truncate=False)
+-------------------+---------+--------------+---------+
|ts_field           |dayofyear|current_date()|dayofyear|
+-------------------+---------+--------------+---------+
|2022-06-06 06:06:06|157      |2022-06-14    |165      |
|2022-06-21 13:04:43|172      |2022-06-14    |165      |
+-------------------+---------+--------------+---------+

➠ dayofweek function(): dayofweek function can be used to extract day of the week from a date or timestamp.

from pyspark.sql.functions import current_date, dayofweek

df_update = df.select("ts_field", dayofweek("ts_field").alias("dayofweek"), current_date(), dayofweek(current_date()).alias("dayofweek"))

df_update.show(truncate=False)
+-------------------+---------+--------------+---------+
|ts_field           |dayofweek|current_date()|dayofweek|
+-------------------+---------+--------------+---------+
|2022-06-06 06:06:06|2        |2022-06-14    |3        |
|2022-06-21 13:04:43|3        |2022-06-14    |3        |
+-------------------+---------+--------------+---------+

➠ quarter function(): quarter function can be used to extract quarter of the year from a date or timestamp.

from pyspark.sql.functions import current_date, quarter

df_update = df.select("ts_field", quarter("ts_field").alias("quarter"), current_date(), quarter(current_date()).alias("quarter"))

df_update.show(truncate=False)
+-------------------+-------+--------------+-------+
|ts_field           |quarter|current_date()|quarter|
+-------------------+-------+--------------+-------+
|2022-06-06 06:06:06|2      |2022-06-14    |2      |
|2022-06-21 13:04:43|2      |2022-06-14    |2      |
+-------------------+-------+--------------+-------+

➠ hour function(): hour function can be used to extract hour from a date or timestamp.

from pyspark.sql.functions import current_timestamp, hour

df_update = df.select("ts_field", hour("ts_field").alias("hour"), current_timestamp(), hour(current_timestamp()).alias("hour"))

df_update.show(truncate=False)
+-------------------+----+-----------------------+----+
|ts_field           |hour|current_timestamp()    |hour|
+-------------------+----+-----------------------+----+
|2022-06-06 06:06:06|6   |2022-06-14 20:35:46.406|20  |
|2022-06-21 13:04:43|13  |2022-06-14 20:35:46.406|20  |
+-------------------+----+-----------------------+----+

➠ minute function(): minute function can be used to extract minute from a date or timestamp.

from pyspark.sql.functions import current_timestamp, minute

df_update = df.select("ts_field", minute("ts_field").alias("minute"), current_timestamp(), minute(current_timestamp()).alias("minute"))

df_update.show(truncate=False)
+-------------------+------+-----------------------+------+
|ts_field           |minute|current_timestamp()    |minute|
+-------------------+------+-----------------------+------+
|2022-06-06 06:06:06|6     |2022-06-14 20:37:40.813|37    |
|2022-06-21 13:04:43|4     |2022-06-14 20:37:40.813|37    |
+-------------------+------+-----------------------+------+

➠ second function(): second function can be used to extract second from a date or timestamp.

from pyspark.sql.functions import current_timestamp, second

df_update = df.select("ts_field", second("ts_field").alias("second"), current_timestamp(), second(current_timestamp()).alias("second"))

df_update.show(truncate=False)
+-------------------+------+-----------------------+------+
|ts_field           |second|current_timestamp()    |second|
+-------------------+------+-----------------------+------+
|2022-06-06 06:06:06|6     |2022-06-14 20:38:00.812|0     |
|2022-06-21 13:04:43|43    |2022-06-14 20:38:00.812|0     |
+-------------------+------+-----------------------+------+