
        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 20], ["Bob", 30]], ["name", "age"])
df.show()
                
            
            +----+---+
|name|age|
+----+---+
|Alex| 20|
| Bob| 30|
+----+---+

Selecting a column in PySpark

To select the name column:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.select(F.col("name")).show()
                
            
            +----+
|name|
+----+
|Alex|
| Bob|
+----+

Note that we could also select the name column without the explicit use of F.col(~) like so:


        
        
            
                
                
                    df.select("name").show()
                
            
            +----+
|name|
+----+
|Alex|
| Bob|
+----+

Creating a new column

To create a new column called status whose values are dependent on the age column:


        
        
            
                
                
                    new_col = F.when(F.col("age") < 25, "junior").otherwise("senior").alias("status")
df.select("*", new_col).show()
                
            
            +----+---+------+
|name|age|status|
+----+---+------+
|Alex| 20|junior|
| Bob| 30|senior|
+----+---+------+

Note the following:

the "*" refers to all the columns of df.
we are using the when(~) and otherwise(~) pattern to fill the values of our column conditionally
we use the alias(~) method to assign a label to new column

Note F.col("age") can also be replaced by df["age"]:


        
        
            
                
                
                    new_col = F.when(df["age"] < 25, "junior").otherwise("senior").alias("status")
df.select("*", new_col).show()
                
            
            +----+---+------+
|name|age|status|
+----+---+------+
|Alex| 20|junior|
| Bob| 30|senior|
+----+---+------+

How does col know which DataFrame's column to refer to?

Notice how the col(~) method only takes in as argument the name of the column. PySpark executes our code lazily and waits until an action is invoked (e.g. show()) to run all the transformations (e.g. df.select(~)). Therefore, PySpark will have the needed context to decipher to which DataFrame's column the col(~) is referring.

For example, suppose we have the following two PySpark DataFrames with the same schema:


        
        
            
                
                
                    df1 = spark.createDataFrame([["Alex", 20], ["Bob", 30]], ["name", "age"])
df2 = spark.createDataFrame([["Cathy", 40], ["Doge", 50]], ["name", "age"])
my_col = F.col("name")

Let's select the name column from df1:


        
        
            
                
                
                    df1.select(my_col).show()
                
            
            +----+
|name|
+----+
|Alex|
| Bob|
+----+

Here, PySpark knows that we are referring to df1's name column because df1 is invoking the transformation (select(~)).

Let's now select the name column from df2:


        
        
            
                
                
                    df2.select(my_col).show()
                
            
            +-----+
| name|
+-----+
|Cathy|
| Doge|
+-----+

Again, PySpark is aware that this time the name column is referring to df2's column.

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.col.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!