for lists, key should be an integer index indicating the position of the value that you wish to extract.
for dictionaries, key should be the key of the values you wish to extract.

Return Value

A new PySpark Column.

Examples

Consider the following PySpark DataFrame:


        
        
            
                
                
                    rows = [[[5,6]], [[7,8]]]
df = spark.createDataFrame(rows, ['vals'])
df.show()
                
            
            +------+
|  vals|
+------+
|[5, 6]|
|[7, 8]|
+------+

Extracting n-th item in lists

To extract the second value from each list in the vals column:


        
        
            
                
                
                    # Assign a label to the column returned by getItem(~)
df_result = df.select(F.col('vals').getItem(1).alias('2nd val'))
df_result.show()
                
            
            +-------+
|2nd val|
+-------+
|      6|
|      8|
+-------+

Note that we could also use [~] syntax instead of getItem(~):


        
        
            
                
                
                    df_result = df.select(F.col('vals')[1].alias('2nd val'))
df_result.show()
                
            
            +-------+
|2nd val|
+-------+
|      6|
|      8|
+-------+

Specifying an index position that is out of bounds for the list will return a null value:


        
        
            
                
                
                    df_result = df.select(F.col('vals').getItem(9))
df_result.show()
                
            
            +-------+
|2nd val|
+-------+
|   null|
|   null|
+-------+

Extracting values using keys in dictionaries

Consider the following PySpark DataFrame:


        
        
            
                
                
                    rows = [[{'A':4}], [{'A':5, 'B':6}]]
df = spark.createDataFrame(rows, ['vals'])
df.show()
                
            
            +----------------+
|            vals|
+----------------+
|        {A -> 4}|
|{A -> 5, B -> 6}|
+----------------+

To extract the value where the key is 'A':


        
        
            
                
                
                    df_result = df.select(F.col('vals').getItem('A'))
df_result.show()
                
            
            +-------+
|vals[A]|
+-------+
|      4|
|      5|
+-------+

Note that referring to keys that do not exist will return null:


        
        
            
                
                
                    df_result = df.select(F.col('vals').getItem('C'))
df_result.show()
                
            
            +-------+
|vals[C]|
+-------+
|   null|
|   null|
+-------+