Spark and spark streaming internals
3 likes•974 views
Spark and Spark Streaming internals allow for low latency, fault tolerance, and diverse workloads. Spark uses a Resilient Distributed Dataset (RDD) model where data is partitioned across a cluster. A directed acyclic graph (DAG) is used to schedule tasks across stages in an optimized way. Spark Streaming runs streaming computations as small deterministic batch jobs by chopping live streams into batches and processing them using RDD transformations and actions.
Spark and spark streaming internals
- 1. Spark And Spark Streaming Internals
- 2. Goals for Spark And Spark Streaming Project • Generalise the framework for diverse workloads. • Low Latency: For small jobs,latency expected is subsecond rather than waiting for few seconds for job to start • Fault Tolerance: Spark Internally should be capable of handling faults rather than depending on users to treat it as special case
- 3. Need to Understand Internals of Spark Understand Importance of Internals from perspective of performance Example: Consider a single core machine where we need to find the position of an integer in an array of integers.First intuition would be to traverse through the list sequencially and rather than randomly interating through list. This is obvious just because we know how cache works and thus sequential access is better than random. But this may not be inherently obvious in Spark just because internals of spark works little differently.
- 5. Execution model of a Job
- 6. Example Job val sc = new SparkContext(...) val file = sc.textFile(…) val errors=file.filter(…) errors.cache(…) errors.count(…) RDD Action
- 7. Resilient Distributed Dataset RDD is a read-only, partitioned collection of records. RDDs are a 'immutable resilient distributed collection of records' which can be stored in the volatile memory or in a persistent storage (HDFS, HBase etc) and can be converted into another RDD through some of the transformations. An action like count can also be applied on an RDD.
- 8. Components Program Spark Master Spark Worker Cluster Manager HDFS,HBase
- 9. DAG TaskSet Task RDD Objects Rdd1.join(rdd2) .groupBy(..) .filter(..) Build Operator DAG Splits Graph into stages Of Tasks Submit each stage as ready DAG Scheduler
- 10. What is Spark Streaming?
- 11. Overview Run a streaming computation as a series of very small, deterministic batch jobs SparkStreaming Spark - Chop up the live stream into batches of X seconds - Spark treats each batch of data as RDDs and processes them using RDD operations - Finally, the processed results of the RDD operations are returned in batches
- 12. Eg: Get hashtags from Twitter val tweets = ssc.twitterStream(<Twitter username>, <Twitter password>) val hashTags = tweets.flatMap (status => status.getText.split(" ").filter(_.startsWith("#")))) hashTags.saveAsHadoopFiles("hdfs://...") Transformation #Ebola, #India, #Mars ...
- 13. Questions