
        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 20], ["Bob", 30], ["Cathy", 40]], ["name", "age"])
df.show()
                
            
            +-----+---+
| name|age|
+-----+---+
| Alex| 20|
|  Bob| 30|
|Cathy| 40|
+-----+---+

Replacing characters in PySpark Column

Suppose we wanted to make the following character replacements:

We can perform these character replacements using the translate(~) method:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.select(F.translate("name", "Aeo", "#@%")).show()
                
            
            +-------------------------+
|translate(name, Aeo, #@%)|
+-------------------------+
|                     #l@x|
|                      B%b|
|                    Cathy|
+-------------------------+

Note that we can obtain a new PySpark DataFrame with the translated column using the withColumn(~) method:


        
        
            
                
                
                    df_new = df.withColumn("name", F.translate("name", "Aeo", "#@%"))
df_new.show()
                
            
            +-----+---+
| name|age|
+-----+---+
| #l@x| 20|
|  B%b| 30|
|Cathy| 40|
+-----+---+

Finally, note that specifying less characters for the replace parameter will result in the removal of the corresponding characters in matching:


        
        
            
                
                
                    df.select(F.translate("name", "Aeo", "#")).show()
                
            
            +-----------------------+
|translate(name, Aeo, #)|
+-----------------------+
|                    #lx|
|                     Bb|
|                  Cathy|
+-----------------------+

Here, the characters e and o are removed, while A is replaced by #.

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/3.2.0/api/python/reference/api/pyspark.sql.functions.translate.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!