thumb_up

thumb_down

chat_bubble_outline
0
 auto_stories new 
settings

PySpark SQL Functions | trim method

Machine Learning

chevron_right

PySpark

chevron_right

Documentation

chevron_right

PySpark SQL Functions

schedule Jul 1, 2022

Last updated

local_offer PySpark

Parameters

1. col | string

The column of type string to trim.

Return Value

A new PySpark Column.

Examples

Consider the following PySpark DataFrame:


        
        
            
                
                
                    df = spark.createDataFrame([["  Alex  ", 20], ["  Bob", 30], ["Cathy    ", 40]], ["name", "age"])
df.show()
                
            
            +---------+---+
|     name|age|
+---------+---+
|   Alex  | 20|
|      Bob| 30|
|Cathy    | 40|
+---------+---+

Here, the values in the name column have leading and trailing spaces.

Trimming columns in PySpark

To trim the name column, that is, to remove the leading and trailing spaces:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.select(F.trim("name").alias("trimmed_name")).show()
                
            
            +------------+
|trimmed_name|
+------------+
|        Alex|
|         Bob|
|       Cathy|
+------------+

Here, the alias(~) method is used to assign a label to the Column returned by trim(~).

To get the original PySpark DataFrame but with the name column updated with the trimmed version, use the withColumn(~) method:


        
        
            
                
                
                    df.withColumn("name", F.trim("name").alias("trimmed_name")).show()
                
            
            +-----+---+
| name|age|
+-----+---+
| Alex| 20|
|  Bob| 30|
|Cathy| 40|
+-----+---+