
        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 100, 200], ["Bob", 150, 50]], ["name", "salary", "bonus"])
df.show()
                
            
            +----+------+-----+
|name|salary|bonus|
+----+------+-----+
|Alex|   100|  200|
| Bob|   150|   50|
+----+------+-----+

Getting the largest value of each row in PySpark DataFrame

To get the largest value of each row in the columns salary and bonus:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.select(F.greatest("salary", "bonus")).show()
                
            
            +-----------------------+
|greatest(salary, bonus)|
+-----------------------+
|                    200|
|                    150|
+-----------------------+

To append this column to the existing PySpark DataFrame, use the withColumn(~) method:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.withColumn("my_max", F.greatest("salary", "bonus")).show()
                
            
            +----+------+-----+------+
|name|salary|bonus|my_max|
+----+------+-----+------+
|Alex|   100|  200|   200|
| Bob|   150|   50|   150|
+----+------+-----+------+

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.greatest.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!