Return Value

A new PySpark column.

Examples

Consider the following PySpark DataFrame:


        
        
            
                
                
                    df = spark.createDataFrame([("A#A",), ("B##B",), ("#C#C#C#",), (None,)], ["x",])
df.show()
                
            
            +-------+
|      x|
+-------+
|    A#A|
|   B##B|
|#C#C#C#|
|   null|
+-------+

Splitting strings by delimiter in PySpark Column

To split the strings in column x by "#", use the split(~) method:


        
        
            
                
                
                    df.select(F.split("x", "#")).show()
                
            
            +---------------+
|split(x, #, -1)|
+---------------+
|         [A, A]|
|       [B, , B]|
|  [, C, C, C, ]|
|           null|
+---------------+

Here, note the following:

the second delimiter parameter is actually parsed as a regular expression - we will see an example of this later.
splitting null results in null.

We can also specify the maximum number of splits to perform using the optional parameter limit:


        
        
            
                
                
                    df.select(F.split("x", "#", 2)).show()
                
            
            +--------------+
|split(x, #, 2)|
+--------------+
|        [A, A]|
|       [B, #B]|
|    [, C#C#C#]|
|          null|
+--------------+

Here, the array containing the splitted tokens can be at most length 2. This is the reason why we still see our delimiter substring "#" in there.

Splitting strings using regular expression in PySpark Column

Consider the following PySpark DataFrame:


        
        
            
                
                
                    df = spark.createDataFrame([("A#A",), ("B@B",), ("C#@C",)], ["x",])
df.show()
                
            
            +----+
|   x|
+----+
| A#A|
| B@B|
|C#@C|
+----+

To split by either the characters # or @, we can use a regular expression as the delimiter:


        
        
            
                
                
                    df.select(F.split("x", "[#@]")).show()
                
            
            +------------------+
|split(x, [#@], -1)|
+------------------+
|            [A, A]|
|            [B, B]|
|          [C, , C]|
+------------------+

Here, the regular expression [#@] denotes either # or @.

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/3.2.0/api/python/reference/api/pyspark.sql.functions.split.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!