
        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 20], [None, None], ["Cathy", None]], ["name", "age"])
df.show()
                
            
            +-----+----+
| name| age|
+-----+----+
| Alex|  20|
| null|null|
|Cathy|null|
+-----+----+

Dropping rows with at least one missing value in PySpark DataFrame

To drop rows with at least one missing value:


        
        
            
                
                
                    df.dropna().show()
                
            
            +----+---+
|name|age|
+----+---+
|Alex| 20|
+----+---+

Dropping rows with at least n non-missing values in PySpark DataFrame

To drop rows with at least 2 non-missing values:


        
        
            
                
                
                    n_non_missing_vals = 2
df.dropna(thresh=n_non_missing_vals).show()
                
            
            +----+---+
|name|age|
+----+---+
|Alex| 20|
+----+---+

Dropping rows with at least n missing values in PySpark DataFrame

To drop rows with at least 2 missing values:


        
        
            
                
                
                    n_missing_vals = 2
df.dropna(thresh=len(df.columns)-n_missing_vals+1).show()
                
            
            +-----+----+
| name| age|
+-----+----+
| Alex|  20|
|Cathy|null|
+-----+----+

Dropping rows with all missing values in PySpark DataFrame

To drop rows with all missing values:


        
        
            
                
                
                    df.dropna(how='all').show()
                
            
            +-----+----+
| name| age|
+-----+----+
| Alex|  20|
|Cathy|null|
+-----+----+

Dropping rows where certain value is missing in PySpark DataFrame

To drop rows where the value for age is missing:


        
        
            
                
                
                    df.dropna(subset='age').show()
                
            
            +----+---+
|name|age|
+----+---+
|Alex| 20|
+----+---+

Dropping rows where certain values are missing (either) in PySpark DataFrame

To drop rows where either the name or age column value is missing:


        
        
            
                
                
                    df.dropna(subset=['name','age'], how='any').show()
                
            
            +----+---+
|name|age|
+----+---+
|Alex| 20|
+----+---+

Dropping rows where certain values are missing (all) in PySpark DataFrame

To drop rows where the name and age column values are both missing:


        
        
            
                
                
                    df.dropna(subset=['name','age'], how='all').show()
                
            
            +-----+----+
| name| age|
+-----+----+
| Alex|  20|
|Cathy|null|
+-----+----+

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.dropna.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!