
        
        
            
                
                
                    df = spark.createDataFrame([["Alex", "1995-12-16"], ["Bob", "1998-05-06"]], ["name", "birthday"])
df.show()
                
            
            +----+----------+
|name|  birthday|
+----+----------+
|Alex|1995-12-16|
| Bob|1998-05-06|
+----+----------+

To convert the date strings in the column birthday:


        
        
            
                
                
                    from pyspark.sql import functions as F
df.select(F.date_format("birthday", "dd/MM/yyyy").alias("birthday_new")).show()
                
            
            +------------+
|birthday_new|
+------------+
|  16/12/1995|
|  06/05/1998|
+------------+

Here,:

"dd/MM/yyyy" indicates a date string starting with the day, then month, then year.
alias(~) is used to give a name to the Column object returned by date_format(~).

Formatting datetime values in PySpark DataFrame

Consider the following PySpark DataFrame with some datetime values:


        
        
            
                
                
                    import datetime
df = spark.createDataFrame([["Alex", datetime.date(1995,12,16)], ["Bob", datetime.date(1995,5,9)]], ["name", "birthday"])
df.show()
                
            
            +----+----------+
|name|  birthday|
+----+----------+
|Alex|1995-12-16|
| Bob|1995-05-09|
+----+----------+

To convert the datetime values in column birthday:


        
        
            
                
                
                    df.select(F.date_format("birthday", "dd-MM-yyyy").alias("birthday_new")).show()
                
            
            +------------+
|birthday_new|
+------------+
|  16-12-1995|
|  06-05-1998|
+------------+

Here, we are using the date format "dd-MM-yyyy", which means day first, and then month followed by year. We also assign the column name "birthday_new" to the Column returned by date_format().

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.functions.date_format.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!