
        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 25, True], ["Bob", 30, False]], ["name", "age", "is_married"])
df.show()
                
            
            +----+---+----------+
|name|age|is_married|
+----+---+----------+
|Alex| 25|      true|
| Bob| 30|     false|
+----+---+----------+

Dropping a single column of PySpark DataFrame

To drop the name column:


        
        
            
                
                
                    df.drop("name").show()
                
            
            +---+----------+
|age|is_married|
+---+----------+
| 25|      true|
| 30|     false|
+---+----------+

Note that the original df is kept intact.

We can also supply the column as a Column object using sql.functions:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.drop(F.col("name")).show()
                
            
            +---+----------+
|age|is_married|
+---+----------+
| 25|      true|
| 30|     false|
+---+----------+

Dropping multiple columns of PySpark DataFrame

To drop columns name and age:


        
        
            
                
                
                    df.drop("name", "age").show()
                
            
            +----------+
|is_married|
+----------+
|      true|
|     false|
+----------+

WARNING

We cannot remove columns by supplying multiple Column objects:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.drop(F.col("name"), F.col("age")).show()
                
            
            TypeError: each col in the param list should be a string

Dropping columns given a list of column labels

To drop columns given a list of column labels:


        
        
            
                
                
                    cols = ["name", "age"]
df.drop(*cols).show()
                
            
            +----------+
|is_married|
+----------+
|      true|
|     false|
+----------+

Here, *cols converts the list into positional arguments.

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.drop.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!