search
Search
Join our weekly DS/ML newsletter layers DS/ML Guides
menu
menu search toc more_vert
Robocat
Guest 0reps
Thanks for the thanks!
close
Comments
Log in or sign up
Cancel
Post
account_circle
Profile
exit_to_app
Sign out
help Ask a question
Share on Twitter
search
keyboard_voice
close
Searching Tips
Search for a recipe:
"Creating a table in MySQL"
Search for an API documentation: "@append"
Search for code: "!dataframe"
Apply a tag filter: "#python"
Useful Shortcuts
/ to open search panel
Esc to close search panel
to navigate between search results
d to clear all current filters
Enter to expand content preview
icon_star
Doc Search
icon_star
Code Search Beta
SORRY NOTHING FOUND!
mic
Start speaking...
Voice search is only supported in Safari and Chrome.
Navigate to
A
A
brightness_medium
share
arrow_backShare
Twitter
Facebook

PySpark SQL Functions | date_format method

Machine Learning
chevron_right
PySpark
chevron_right
Documentation
chevron_right
PySpark SQL Functions
schedule Jul 1, 2022
Last updated
local_offer PySpark
Tags

PySpark SQL Functions' date_format(~) method converts a date, timestamp or string into a date string with the specified format.

Parameters

1. date | Column or string

The date column - this could be of type date, timestamp or string.

2. format | string

The format of the resulting date string.

Return Value

A Column object of date strings.

Examples

Formatting date strings in PySpark DataFrame

Consider the following PySpark DataFrame with some date strings:

df = spark.createDataFrame([["Alex", "1995-12-16"], ["Bob", "1998-05-06"]], ["name", "birthday"])
df.show()
+----+----------+
|name| birthday|
+----+----------+
|Alex|1995-12-16|
| Bob|1998-05-06|
+----+----------+

To convert the date strings in the column birthday:

from pyspark.sql import functions as F
df.select(F.date_format("birthday", "dd/MM/yyyy").alias("birthday_new")).show()
+------------+
|birthday_new|
+------------+
| 16/12/1995|
| 06/05/1998|
+------------+

Here,:

  • "dd/MM/yyyy" indicates a date string starting with the day, then month, then year.

  • alias(~) is used to give a name to the Column object returned by date_format(~).

Formatting datetime values in PySpark DataFrame

Consider the following PySpark DataFrame with some datetime values:

import datetime
df = spark.createDataFrame([["Alex", datetime.date(1995,12,16)], ["Bob", datetime.date(1995,5,9)]], ["name", "birthday"])
df.show()
+----+----------+
|name| birthday|
+----+----------+
|Alex|1995-12-16|
| Bob|1995-05-09|
+----+----------+

To convert the datetime values in column birthday:

df.select(F.date_format("birthday", "dd-MM-yyyy").alias("birthday_new")).show()
+------------+
|birthday_new|
+------------+
| 16-12-1995|
| 06-05-1998|
+------------+

Here, we are using the date format "dd-MM-yyyy", which means day first, and then month followed by year. We also assign the column name "birthday_new" to the Column returned by date_format().

mail
Join our newsletter for updates on new DS/ML comprehensive guides (spam-free)
robocat
Published by Isshin Inada
Edited by 0 others
Did you find this page useful?
thumb_down
0
thumb_down
0
chat_bubble_outline
0
settings
Enjoy our search
Hit / to insta-search docs and recipes!