
        
        
            
                
                
                    df = spark.createDataFrame([[['a','b']],[['d']]], ['vals'])
df.show()
                
            
            +------+
|  vals|
+------+
|[a, b]|
|   [d]|
+------+

Here, the column vals contains lists.

To flatten the lists in the column vals, use the explode(~) method:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.select(F.explode('vals').alias('exploded')).show()
                
            
            +--------+
|exploded|
+--------+
|       a|
|       b|
|       d|
+--------+

Here, we are using the alias(~) method to assign a label to the column returned by explode(~).

Flattening dictionaries

Consider the following PySpark DataFrame:


        
        
            
                
                
                    df = spark.createDataFrame([[{'a':'b'}],[{'c':'d','e':'f'}]], ['vals'])
df.show()
                
            
            +----------------+
|            vals|
+----------------+
|        {a -> b}|
|{e -> f, c -> d}|
+----------------+

Here, the column vals contains dictionaries.

To flatten each dictionary in column vals, use the explode(~) method:


        
        
            
                
                
                    df.select(F.explode('vals').alias('exploded_key', 'exploded_val')).show()
                
            
            +------------+------------+
|exploded_key|exploded_val|
+------------+------------+
|           a|           b|
|           e|           f|
|           c|           d|
+------------+------------+

In the case of dictionaries, the explode(~) method returns two columns - the first column contains all the keys while the second column contains all the values.

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.functions.explode.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!