PySpark RDD | reduceByKey method
Start your free 7-days trial now!
reduceByKey(~) method aggregates the RDD data by key, and perform a reduction operation. A reduction operation is simply one where multiple values become reduced to a single value (e.g. summation, multiplication).
The reduction function to apply.
By default, the number of partitions will be equal to the number of partitions of the parent RDD. If the parent RDD does not have the partition count set, then the parallelism level in the PySpark configuration will be used.
The partitioner to use - the input is a key and return value must be the hashed value. By default, a hash partitioner will be used.
A PySpark RDD (
Consider the following Pair RDD:
parallelize(~) method creates a RDD with 3 partitions.
Grouping by key in pair RDD and performing a reduction operation
To group by key and perform a summation of the values of each grouped key:
[('B', 1), ('C', 1), ('A', 2)]
Setting number of partitions after reducing by key in pair RDD
By default, the number of partitions of the resulting RDD will be equal to the number of partitions of the parent RDD:
rdd is the parent RDD of
We can set the number of partitions of the resulting RDD by setting the