search
Search
Login
Math ML Join our weekly DS/ML newsletter
menu
menu search toc more_vert
Robocat
Guest 0reps
Thanks for the thanks!
close
Comments
Log in or sign up
Cancel
Post
account_circle
Profile
exit_to_app
Sign out
help Ask a question
Share on Twitter
search
keyboard_voice
close
Searching Tips
Search for a recipe:
"Creating a table in MySQL"
Search for an API documentation: "@append"
Search for code: "!dataframe"
Apply a tag filter: "#python"
Useful Shortcuts
/ to open search panel
Esc to close search panel
to navigate between search results
d to clear all current filters
Enter to expand content preview
icon_star
Doc Search
icon_star
Code Search Beta
SORRY NOTHING FOUND!
mic
Start speaking...
Voice search is only supported in Safari and Chrome.
Navigate to
A
A
brightness_medium
share
arrow_backShare
Twitter
Facebook

Counting frequency of values in PySpark DataFrame Column

Machine Learning
chevron_right
PySpark
chevron_right
Cookbooks
schedule Jul 2, 2022
Last updated
local_offer PySpark
Tags

Consider the following PySpark DataFrame:

df = spark.createDataFrame([['A'],['A'],['B']], ['col1'])
df.show()
+----+
|col1|
+----+
| A|
| A|
| B|
+----+

Counting frequency of values using aggregation (groupBy and count)

To count the frequency of values in column col1:

df.groupBy('col1').count().show()
+----+-----+
|col1|count|
+----+-----+
| A| 2|
| B| 1|
+----+-----+

Here, we are first grouping by the values in col1, and then for each group, we are counting the number of rows.

Sorting PySpark DataFrame by frequency counts

The resulting PySpark DataFrame is not sorted by any particular order by default. We can sort the DataFrame by the count column using the orderBy(~) method:

df.groupBy('col1').count().orderBy('count', ascending=False).show()
+----+-----+
|col1|count|
+----+-----+
| A| 2|
| B| 1|
+----+-----+

Here, the output is similar to Pandas' value_counts(~) method which returns the frequency counts in descending order.

Assigning label to count aggregate column

Similar to what we did with the methods groupBy(~) and count(), we can also use the agg(~) method, which takes as input an aggregate function:

import pyspark.sql.functions as F
df.groupBy('col1').agg(F.count('col1').alias('my_count')).show()
+----+--------+
|col1|my_count|
+----+--------+
| A| 2|
| B| 1|
+----+--------+

This is more verbose than the solution using groupBy(~) and count(), but the advantage is that we can use the alias(~) method to assign a name to the resulting aggregate column - here the label is my_count instead of the default count.

mail
Join our newsletter for updates on new DS/ML comprehensive guides (spam-free)
robocat
Published by Isshin Inada
Edited by 0 others
Did you find this page useful?
thumb_up
thumb_down
Ask a question or leave a feedback...
thumb_up
1
thumb_down
0
chat_bubble_outline
0
settings
Enjoy our search
Hit / to insta-search docs and recipes!