
        
        
            
                
                
                    df = spark.createDataFrame([["Alex", "Wong"], ["Bob", "Marley"]], ["fname", "lname"])
df.show()
                
            
            +-----+------+
|fname| lname|
+-----+------+
| Alex|  Wong|
|  Bob|Marley|
+-----+------+

To concatenate fname and lname:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.select(F.concat("fname", "lname")).show()
                
            
            +--------------------+
|concat(fname, lname)|
+--------------------+
|            AlexWong|
|           BobMarley|
+--------------------+

If you wanted to include a space between the two columns, you can use F.lit(" ") like so:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.select(F.concat("fname", F.lit(" "), "lname")).show()
                
            
            +-----------------------+
|concat(fname,  , lname)|
+-----------------------+
|              Alex Wong|
|             Bob Marley|
+-----------------------+

F.lit(" ") is a Column object whose values are filled with " ".

You could also add an alias to the returned column like so:


        
        
            
                
                
                    df.select(F.concat("fname", "lname").alias("COMBINED NAME")).show()
                
            
            +-------------+
|COMBINED NAME|
+-------------+
|     AlexWong|
|    BobMarley|
+-------------+

You could also pass in Column objects intead of column labels:


        
        
            
                
                
                    df.select(F.concat(df.fname, F.col("lname"))).show()
                
            
            +--------------------+
|concat(fname, lname)|
+--------------------+
|            AlexWong|
|           BobMarley|
+--------------------+

Concatenating array-based columns in PySpark

Consider the following PySpark DataFrame:


        
        
            
                
                
                    df = spark.createDataFrame([[ [4,5], [6]], [ [7], [8,9] ]], ["A", "B"])
df.show()
                
            
            +------+------+
|     A|     B|
+------+------+
|[4, 5]|   [6]|
|   [7]|[8, 9]|
+------+------+

To concatenate the arrays of each column:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.select(F.concat("A", "B")).show()
                
            
            +------------+
|concat(A, B)|
+------------+
|   [4, 5, 6]|
|   [7, 8, 9]|
+------------+

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.concat.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!