PySpark
keyboard_arrow_down 147 guides
chevron_leftPySpark DataFrame
Method aliasMethod coalesceMethod collectMethod colRegexMethod corrMethod countMethod covMethod describeMethod distinctMethod dropMethod dropDuplicatesMethod dropnaMethod exceptAllMethod fillnaMethod filterMethod foreachMethod groupByMethod headMethod intersectMethod intersectAllMethod joinMethod limitMethod orderByMethod printSchemaMethod randomSplitMethod repartitionMethod replaceMethod sampleMethod sampleByMethod selectMethod selectExprMethod showMethod sortMethod summaryMethod tailMethod takeMethod toDFMethod toJSONMethod toPandasMethod transformMethod unionMethod unionByNameMethod whereMethod withColumnMethod withColumnRenamedProperty columnsProperty dtypesProperty rdd
check_circle
Mark as learned thumb_up
0
thumb_down
0
chat_bubble_outline
0
Comment auto_stories Bi-column layout
settings
PySpark DataFrame | dropDuplicates method
schedule Aug 12, 2023
Last updated local_offer
Tags PySpark
tocTable of Contents
expand_more Master the mathematics behind data science with 100+ top-tier guides
Start your free 7-days trial now!
Start your free 7-days trial now!
PySpark DataFrame's dropDuplicates(~) returns a new DataFrame with duplicate rows removed. We can optionally specify columns to check for duplicates.
NOTE
dropDuplicates(~) is an alias for drop_duplicates(~).
Parameters
1. subset | string or list of string | optional
The columns by which to check for duplicates. By default, all columns will be checked.
Return Value
A new PySpark DataFrame.
Examples
Consider the following PySpark DataFrame:
df = spark.createDataFrame([["Alex", 25], ["Bob", 30], ["Bob", 30], ["Cathy", 25]], ["name", "age"])
+-----+---+| name|age|+-----+---+| Alex| 25|| Bob| 30|| Bob| 30||Cathy| 25|+-----+---+
Dropping duplicate rows in PySpark DataFrame
To drop duplicate rows:
+-----+---+| name|age|+-----+---+| Alex| 25|| Bob| 30||Cathy| 25|+-----+---+
Note the following:
only the first occurrence is kept while subsequent occurrences are removed.
a new PySpark DataFrame is returned while the original is kept intact.
Dropping duplicate rows for certain columns
To drop duplicate rows based on the age column:
+----+---+|name|age|+----+---+|Alex| 25|| Bob| 30|+----+---+
Again, only the first occurrence is kept while the latter duplicate rows are discarded.
Published by Isshin Inada
Edited by 0 others
Did you find this page useful?
thumb_up
thumb_down
Comment
Citation
Ask a question or leave a feedback...
Official PySpark Documentation
https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.dropDuplicates.html
thumb_up
0
thumb_down
0
chat_bubble_outline
0
settings
Enjoy our search
Hit / to insta-search docs and recipes!
