The starting position. This position is inclusive and non-index, meaning the first character is in position 1. Negative position is allowed here as well - please consult the example below for clarification.

2. length | int or Column

The length of the substring to extract.

Return Value

A Column object.

Examples

Consider the following PySpark DataFrame:


        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 20], ["Bob", 30], ["Cathy", 40]], ["name", "age"])
df.show()
                
            
            +-----+---+
| name|age|
+-----+---+
| Alex| 20|
|  Bob| 30|
|Cathy| 40|
+-----+---+

Extracting substrings from column values in PySpark DataFrame

To extract substrings from column values:


        
        
            
                
                
                    from pyspark.sql import functions as F
df.select(F.col("name").substr(2,3).alias("short_name")).show()
                
            
            +----------+
|short_name|
+----------+
|       lex|
|        ob|
|       ath|
+----------+

Note the following:

the F.col("name").substr(2,3) means that we are extracting a substring starting from the 2nd character and up to a length of 3.
even if the string is too short (e.g. "Bob"), no error will be thrown.
alias(~) method is used to assign a label to our column.

Note that you could also specify a negative starting position like so:


        
        
            
                
                
                    df.select(F.col("name").substr(-3,2).alias("short_name")).show()
                
            
            +----------+
|short_name|
+----------+
|        le|
|        Bo|
|        th|
+----------+

Here, we are starting from the third character from the end (inclusive).

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.Column.endswith.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!