
        
        
            
                
                
                    df = spark.createDataFrame([["2023-04-20"], ["2023-04-22"]], ["my_date"])
df.show()
                
            
            +----------+
|   my_date|
+----------+
|2023-04-20|
|2023-04-22|
+----------+

To add 5 days to our column:


        
        
            
                
                
                    from pyspark.sql import functions as F
df.select(F.date_add("my_date", 5)).show()
                
            
            +--------------------+
|date_add(my_date, 5)|
+--------------------+
|          2023-04-25|
|          2023-04-27|
+--------------------+

Adding a column of days to a column of dates

Unfortunately, the date_add(-) method only accepts a constant for the second parameter. To add a column of days to a column of dates, we must take another approach.

To demonstrate, consider the following PySpark DataFrame:


        
        
            
                
                
                    df = spark.createDataFrame([["2023-04-20", 3], ["2023-04-22", 5]], ["my_date", "my_days"])
df.show()
                
            
            +----------+-------+
|   my_date|my_days|
+----------+-------+
|2023-04-20|      3|
|2023-04-22|      5|
+----------+-------+

To add my_days to my_date, supply the following SQL method in the F.expr() method like so:


        
        
            
                
                
                    # Cast to INT first - by default, intgers have type BIGINT (F.expr(-) with raise an error)
df = df.withColumn("my_days", df["my_days"].cast("int"))
df_new = df_new.withColumn("new_date", F.expr("date_add(my_date, my_days)"))
df_new.show()
                
            
            +----------+-------+----------+
|   my_date|my_days|  new_date|
+----------+-------+----------+
|2023-04-20|      3|2023-04-23|
|2023-04-22|      5|2023-04-27|
+----------+-------+----------+

The resulting data type of the columns is as follows:


        
        
            
                
                
                    df_new.printSchema()
                
            
            root
 |-- my_date: string (nullable = true)
 |-- days: integer (nullable = true)
 |-- new_date: date (nullable = true)

Notice how even though my_date is of type string, the resulting new_date is of type date.

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.functions.date_add.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!