Introduction to

PySpark RDD
B I G D ATA F U N D A M E N TA L S W I T H P Y S PA R K

Upendra Devisetty
Science Analyst, CyVerse
What is RDD?
RDD = Resilient Distributed Datasets

BIG DATA FUNDAMENTALS WITH PYSPARK

Decomposing RDDs
Resilient Distributed Datasets
Resilient: Ability to withstand failures

Distributed: Spanning across multiple machines

Datasets: Collection of partitioned data e.g, Arrays, Tables, Tuples etc.,

BIG DATA FUNDAMENTALS WITH PYSPARK

Creating RDDs. How to do it?
Parallelizing an existing collection of objects
External datasets:
Files in HDFS

Objects in Amazon S3 bucket

lines in a text file

From existing RDDs

BIG DATA FUNDAMENTALS WITH PYSPARK

Parallelized collection (parallelizing)
parallelize() for creating RDDs from python lists

numRDD = sc.parallelize([1,2,3,4])

helloRDD = sc.parallelize("Hello world")

type(helloRDD)

<class 'pyspark.rdd.PipelinedRDD'>

BIG DATA FUNDAMENTALS WITH PYSPARK

From external datasets
textFile() for creating RDDs from external datasets

fileRDD = sc.textFile("README.md")

type(fileRDD)

<class 'pyspark.rdd.PipelinedRDD'>

BIG DATA FUNDAMENTALS WITH PYSPARK

Understanding Partitioning in PySpark
A partition is a logical division of a large distributed data set
parallelize() method

numRDD = sc.parallelize(range(10), minPartitions = 6)

textFile() method

fileRDD = sc.textFile("README.md", minPartitions = 6)

The number of partitions in an RDD can be found by using getNumPartitions() method

BIG DATA FUNDAMENTALS WITH PYSPARK

 Let's practice
B I G D ATA F U N D A M E N TA L S W I T H P Y S PA R K
 RDD operations in
PySpark
B I G D ATA F U N D A M E N TA L S W I T H P Y S PA R K

Upendra Devisetty
Science Analyst, CyVerse
Overview of PySpark operations

Transformations create new RDDs

Actions perform computation on the RDDs

BIG DATA FUNDAMENTALS WITH PYSPARK

RDD Transformations
Transformations follow Lazy evaluation

Basic RDD Transformations

map() , filter() , flatMap() , and union()

BIG DATA FUNDAMENTALS WITH PYSPARK

map() Transformation
map() transformation applies a function to all elements in the RDD

RDD = sc.parallelize([1,2,3,4])
RDD_map = RDD.map(lambda x: x * x)

BIG DATA FUNDAMENTALS WITH PYSPARK

filter() Transformation
Filter transformation returns a new RDD with only the elements that pass the condition

RDD = sc.parallelize([1,2,3,4])
RDD_filter = RDD.filter(lambda x: x > 2)

BIG DATA FUNDAMENTALS WITH PYSPARK

flatMap() Transformation
flatMap() transformation returns multiple values for each element in the original RDD

RDD = sc.parallelize(["hello world", "how are you"])

RDD_flatmap = RDD.flatMap(lambda x: x.split(" "))

BIG DATA FUNDAMENTALS WITH PYSPARK

union() Transformation

inputRDD = sc.textFile("logs.txt")
errorRDD = inputRDD.filter(lambda x: "error" in x.split())
warningsRDD = inputRDD.filter(lambda x: "warnings" in x.split())
combinedRDD = errorRDD.union(warningsRDD)

BIG DATA FUNDAMENTALS WITH PYSPARK

RDD Actions
They are operations that return a value after running a computation on the RDD
Basic RDD Actions
collect()

take(N)

first()

count()

BIG DATA FUNDAMENTALS WITH PYSPARK

collect() and take() Actions
collect() return all the elements of the dataset as an array
take(N) returns an array with the first N elements of the dataset

RDD_map.collect()

[1, 4, 9, 16]

RDD_map.take(2)

[1, 4]

BIG DATA FUNDAMENTALS WITH PYSPARK

first() and count() Actions
first() prints the first element of the RDD

RDD_map.first()

[1]

count() return the number of elements in the RDD

RDD_flatmap.count()

BIG DATA FUNDAMENTALS WITH PYSPARK

 Let's practice RDD
operations
B I G D ATA F U N D A M E N TA L S W I T H P Y S PA R K
 Working with Pair
RDDs in PySpark
B I G D ATA F U N D A M E N TA L S W I T H P Y S PA R K

Upendra Devisetty
Science Analyst, CyVerse
Introduction to pair RDDs in PySpark
Real life datasets are usually key/value pairs
Each row is a key and maps to one or more values

Pair RDD is a special data structure to work with this kind of datasets

Pair RDD: Key is the identifier and value is the data

BIG DATA FUNDAMENTALS WITH PYSPARK

Creating pair RDDs
Two common ways to create pair RDDs
From a list of key-value tuple

From a regular RDD

Get the data into key/value form for paired RDD

my_tuple = [('Sam', 23), ('Mary', 34), ('Peter', 25)]

pairRDD_tuple = sc.parallelize(my_tuple)

my_list = ['Sam 23', 'Mary 34', 'Peter 25']

regularRDD = sc.parallelize(my_list)
pairRDD_RDD = regularRDD.map(lambda s: (s.split(' ')[0], s.split(' ')[1]))

BIG DATA FUNDAMENTALS WITH PYSPARK

Transformations on pair RDDs
All regular transformations work on pair RDD
Have to pass functions that operate on key value pairs rather than on individual elements

Examples of paired RDD Transformations

reduceByKey(func): Combine values with the same key

groupByKey(): Group values with the same key

sortByKey(): Return an RDD sorted by the key

join(): Join two pair RDDs based on their key

BIG DATA FUNDAMENTALS WITH PYSPARK

reduceByKey() transformation
reduceByKey() transformation combines values with the same key

It runs parallel operations for each key in the dataset

It is a transformation and not action

regularRDD = sc.parallelize([("Messi", 23), ("Ronaldo", 34),

("Neymar", 22), ("Messi", 24)])
pairRDD_reducebykey = regularRDD.reduceByKey(lambda x,y : x + y)
pairRDD_reducebykey.collect()
[('Neymar', 22), ('Ronaldo', 34), ('Messi', 47)]

BIG DATA FUNDAMENTALS WITH PYSPARK

sortByKey() transformation
sortByKey() operation orders pair RDD by key

It returns an RDD sorted by key in ascending or descending order

pairRDD_reducebykey_rev = pairRDD_reducebykey.map(lambda x: (x[1], x[0]))

pairRDD_reducebykey_rev.sortByKey(ascending=False).collect()
[(47, 'Messi'), (34, 'Ronaldo'), (22, 'Neymar')]

BIG DATA FUNDAMENTALS WITH PYSPARK

groupByKey() transformation
groupByKey() groups all the values with the same key in the pair RDD

airports = [("US", "JFK"),("UK", "LHR"),("FR", "CDG"),("US", "SFO")]

regularRDD = sc.parallelize(airports)
pairRDD_group = regularRDD.groupByKey().collect()
for cont, air in pairRDD_group:
print(cont, list(air))
FR ['CDG']
US ['JFK', 'SFO']
UK ['LHR']

BIG DATA FUNDAMENTALS WITH PYSPARK

join() transformation
join() transformation joins the two pair RDDs based on their key

RDD1 = sc.parallelize([("Messi", 34),("Ronaldo", 32),("Neymar", 24)])

RDD2 = sc.parallelize([("Ronaldo", 80),("Neymar", 120),("Messi", 100)])

RDD1.join(RDD2).collect()
[('Neymar', (24, 120)), ('Ronaldo', (32, 80)), ('Messi', (34, 100))]

BIG DATA FUNDAMENTALS WITH PYSPARK

 Let's practice
B I G D ATA F U N D A M E N TA L S W I T H P Y S PA R K
 More actions
B I G D ATA F U N D A M E N TA L S W I T H P Y S PA R K

Upendra Devisetty
Science Analyst, CyVerse
reduce() action
reduce(func) action is used for aggregating the elements of a regular RDD
The function should be commutative (changing the order of the operands does not change
the result) and associative

An example of reduce() action in PySpark

x = [1,3,4,6]
RDD = sc.parallelize(x)
RDD.reduce(lambda x, y : x + y)

14

BIG DATA FUNDAMENTALS WITH PYSPARK

saveAsTextFile() action
saveAsTextFile() action saves RDD into a text file inside a directory with each partition as
a separate file

RDD.saveAsTextFile("tempFile")

coalesce() method can be used to save RDD as a single text file

RDD.coalesce(1).saveAsTextFile("tempFile")

BIG DATA FUNDAMENTALS WITH PYSPARK

Action Operations on pair RDDs
RDD actions available for PySpark pair RDDs

Pair RDD actions leverage the key-value data

Few examples of pair RDD actions include

countByKey()

collectAsMap()

BIG DATA FUNDAMENTALS WITH PYSPARK

countByKey() action
countByKey() only available for type (K, V)

countByKey() action counts the number of elements for each key

Example of countByKey() on a simple list

rdd = sc.parallelize([("a", 1), ("b", 1), ("a", 1)])

for kee, val in rdd.countByKey().items():
print(kee, val)

('a', 2)
('b', 1)

BIG DATA FUNDAMENTALS WITH PYSPARK

collectAsMap() action
collectAsMap() return the key-value pairs in the RDD as a dictionary

Example of collectAsMap() on a simple tuple

sc.parallelize([(1, 2), (3, 4)]).collectAsMap()

{1: 2, 3: 4}

BIG DATA FUNDAMENTALS WITH PYSPARK

 Let's practice
B I G D ATA F U N D A M E N TA L S W I T H P Y S PA R K