By default, the number of partitions will be equal to the number of partitions of the parent RDD. If the parent RDD does not have the partition count set, then the parallelism level in the PySpark configuration will be used.

3. partitionFunc | function | optional

The partitioner to use - the input is a key and return value must be the hashed value. By default, a hash partitioner will be used.

Return Value

A PySpark RDD (pyspark.rdd.PipelinedRDD).

Examples

Consider the following Pair RDD:


        
        
            
                
                
                    rdd = sc.parallelize([("A",1),("B",1),("C",1),("A",1)], numSlices=3)
rdd.collect()
                
            
            [('A', 1), ('B', 1), ('C', 1), ('A', 1)]

Here, the parallelize(~) method creates a RDD with 3 partitions.

Grouping by key in pair RDD and performing a reduction operation

To group by key and perform a summation of the values of each grouped key:


        
        
            
                
                
                    rdd.reduceByKey(lambda a, b: a+b).collect()
                
            
            [('B', 1), ('C', 1), ('A', 2)]

Setting number of partitions after reducing by key in pair RDD

By default, the number of partitions of the resulting RDD will be equal to the number of partitions of the parent RDD:


        
        
            
                
                
                    # Create a RDD using 3 partitions
rdd = sc.parallelize([("A",1),("B",1),("C",1),("A",1),("D",1)], numSlices=3)
new_rdd = rdd.reduceByKey(lambda a, b: a+b)
new_rdd.getNumPartitions()
                
            
            3

Here, rdd is the parent RDD of new_rdd.

We can set the number of partitions of the resulting RDD by setting the numPartitions parameter:


        
        
            
                
                
                    rdd = sc.parallelize([("A",1),("B",1),("C",1),("A",1),("D",1)], numSlices=3)
new_rdd = rdd.reduceByKey(lambda a, b: a+b, numPartitions=2)
new_rdd.getNumPartitions()
                
            
            2

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.RDD.reduceByKey.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!