How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with Akmal Chaudri
Download as PDF, PPTX4 likes2,311 views
The document presents a comprehensive overview of Apache Ignite and its integration with Apache Spark, focusing on sharing state across Spark jobs. It details the functionalities of Ignite, including its memory-centric design, deployment modes, and the use of IgniteContext and IgniteRDD for real-time data sharing. Additionally, the document covers installation procedures and provides resources for further exploration of Ignite for Spark.
![Write code example
16#EUde9
val conf = new SparkConf().setAppName("SparkIgniteWriter")
val sc = new SparkContext(conf)
val ic = new IgniteContext(sc,
"examples/config/spark/example-shared-rdd.xml")
val sharedRDD: IgniteRDD[Int, Int] = ic.fromCache("sharedRDD")
sharedRDD.savePairs(sc.parallelize(1 to 100000, 10)
.map(i => (i, i)))](https://image.slidesharecdn.com/w29akmalchaudri-171031215427/85/How-to-Share-State-Across-Multiple-Apache-Spark-Jobs-using-Apache-Ignite-with-Akmal-Chaudri-16-320.jpg)
![Read code example
18#EUde9
val conf = new SparkConf().setAppName("SparkIgniteReader")
val sc = new SparkContext(conf)
val ic = new IgniteContext(sc,
"examples/config/spark/example-shared-rdd.xml")
val sharedRDD: IgniteRDD[Int, Int] = ic.fromCache("sharedRDD")
val greaterThanFiftyThousand = sharedRDD.filter(_._2 > 50000)
println("The count is "+greaterThanFiftyThousand.count())](https://image.slidesharecdn.com/w29akmalchaudri-171031215427/85/How-to-Share-State-Across-Multiple-Apache-Spark-Jobs-using-Apache-Ignite-with-Akmal-Chaudri-18-320.jpg)
![Shared Deployment
â˘âŻ Standalone mode
â˘âŻ Ignite nodes deployed with Spark worker nodes
â˘âŻ Add following lines to spark-env.sh
21#EUde9
IGNITE_LIBS="${IGNITE_HOME}/libs/*"
for file in ${IGNITE_HOME}/libs/*
do
if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ]; then
IGNITE_LIBS=${IGNITE_LIBS}:${file}/*
fi
done
export SPARK_CLASSPATH=$IGNITE_LIBS](https://image.slidesharecdn.com/w29akmalchaudri-171031215427/85/How-to-Share-State-Across-Multiple-Apache-Spark-Jobs-using-Apache-Ignite-with-Akmal-Chaudri-21-320.jpg)
More Related Content
Similar to How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with Akmal Chaudri (20)
How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with Akmal Chaudri
- 1. Akmal Chaudhri, GridGain Systems How to share state across multiple Spark jobs using Apache Ignite #EUde9
- 2. Agenda â˘âŻ Introduction to Apache Ignite â˘âŻ Ignite for Spark â˘âŻ IgniteContext and IgniteRDD â˘âŻ Installation and Deployment â˘âŻ Demos â˘âŻ Q&A 2#EUde9
- 3. Introduction to Apache Ignite 3#EUde9
- 4. Apache Ignite in one slide â˘âŻ Memory-centric platform â⯠that is strongly consistent â⯠and highly-available â⯠with powerful SQL â⯠key-value and processing APIs â˘âŻ Designed for â⯠Performance â⯠Scalability 4#EUde9
- 5. Apache Ignite â˘âŻ Data source agnostic â˘âŻ Fully fledged compute engine and durable storage â˘âŻ OLAP and OLTP â˘âŻ Fully ACID transactions across memory and disk â˘âŻ In-memory SQL support â˘âŻ Early ML libraries â˘âŻ Growing community 5#EUde9
- 7. Why share state in Spark? â˘âŻ Long running applications â⯠Passing state between jobs â˘âŻ Disk File System â⯠Convert RDDs to disk files and back â˘âŻ Share RDDs in-memory â⯠Native Spark API â⯠Native Spark transformations 7#EUde9
- 8. Ignite for Spark â˘âŻ Spark RDD abstraction â˘âŻ Shared in-memory view on data across different Spark jobs, workers or applications â˘âŻ Implemented as a view over a distributed Ignite cache 8#EUde9
- 9. Ignite for Spark â˘âŻ Deployment modes â⯠Share RDD across tasks on the host â⯠Share RDD across tasks in the application â⯠Share RDD globally â˘âŻ Shared state can be â⯠Standalone mode (outlives Spark application) â⯠Embedded mode (lifetime of Spark application) 9#EUde9
- 10. Ignite In-Memory File System â˘âŻ Distributed in-memory file system â˘âŻ Implements HDFS API â˘âŻ Can be transparently plugged into Hadoop or Spark deployments 10#EUde9
- 12. IgniteContext â˘âŻ Main entry-point to Spark-Ignite integration â˘âŻ SparkContext plus either one of â⯠IgniteConfiguration() â⯠Path to XML configuration file â˘âŻ Optional Boolean client argument â⯠true => Shared deployment â⯠false => Embedded deployment 12#EUde9
- 13. IgniteContext examples 13#EUde9 val igniteContext = new IgniteContext(sparkContext, () => new IgniteConfiguration()) val igniteContext = new IgniteContext(sparkContext, "examples/config/spark/example-shared-rdd.xml")
- 14. IgniteRDD â˘âŻ Implementation of Spark RDD representing a live view of an Ignite cache â˘âŻ Mutable (unlike native RDDs) â⯠All changes in Ignite cache will be visible to RDD users immediately â˘âŻ Provides partitioning information to Spark executor â˘âŻ Provides affinity information to Spark so that RDD computations can use data locality 14#EUde9
- 15. Write to Ignite â˘âŻ Ignite caches operate on key-value pairs â˘âŻ Spark tuple RDD for key-value pairs and savePairs method â⯠RDD partitioning, store values in parallel if possible â˘âŻ Value-only RDD and saveValues method â⯠IgniteRDD generates a unique affinity-local key for each value stored into the cache 15#EUde9
- 16. Write code example 16#EUde9 val conf = new SparkConf().setAppName("SparkIgniteWriter") val sc = new SparkContext(conf) val ic = new IgniteContext(sc, "examples/config/spark/example-shared-rdd.xml") val sharedRDD: IgniteRDD[Int, Int] = ic.fromCache("sharedRDD") sharedRDD.savePairs(sc.parallelize(1 to 100000, 10) .map(i => (i, i)))
- 17. Read from Ignite â˘âŻ IgniteRDD is a live view of an Ignite cache â⯠No need to explicitly load data to Spark application from Ignite â⯠All RDD methods are available to use right away after an instance of IgniteRDD is created 17#EUde9
- 18. Read code example 18#EUde9 val conf = new SparkConf().setAppName("SparkIgniteReader") val sc = new SparkContext(conf) val ic = new IgniteContext(sc, "examples/config/spark/example-shared-rdd.xml") val sharedRDD: IgniteRDD[Int, Int] = ic.fromCache("sharedRDD") val greaterThanFiftyThousand = sharedRDD.filter(_._2 > 50000) println("The count is "+greaterThanFiftyThousand.count())
- 20. Installation and Deployment â˘âŻ Shared Deployment â˘âŻ Embedded Deployment â˘âŻ Maven â˘âŻ SBT 20#EUde9
- 21. Shared Deployment â˘âŻ Standalone mode â˘âŻ Ignite nodes deployed with Spark worker nodes â˘âŻ Add following lines to spark-env.sh 21#EUde9 IGNITE_LIBS="${IGNITE_HOME}/libs/*" for file in ${IGNITE_HOME}/libs/* do if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ]; then IGNITE_LIBS=${IGNITE_LIBS}:${file}/* fi done export SPARK_CLASSPATH=$IGNITE_LIBS
- 22. Embedded Deployment â˘âŻ Ignite nodes are started inside Spark job processes and are stopped when job dies â˘âŻ Ignite code distributed to worker machines using Spark deployment mechanism â˘âŻ Ignite nodes will be started on all workers as a part of IgniteContext initialization 22#EUde9
- 23. Maven â˘âŻ Igniteâs Spark artifact hosted in Maven Central â˘âŻ Scala 2.11 example 23#EUde9 <dependency> <groupId>org.apache.ignite</groupId> <artifactId>ignite-spark</artifactId> <version>${ignite.version}</version> </dependency>
- 24. SBT â˘âŻ Igniteâs Spark artifact added to build.sbt â˘âŻ Scala 2.11 example 24#EUde9 libraryDependencies += "org.apache.ignite" % "ignite-spark" % "ignite.version"
- 25. Demos 25#EUde9
- 26. Resources â˘âŻ Ignite for Spark documentation â⯠https://apacheignite-fs.readme.io/docs/ignite-for-spark â˘âŻ Spark Data Frames Support in Apache Ignite â⯠https://issues.apache.org/jira/browse/IGNITE-3084 â˘âŻ Code examples â⯠https://github.com/apache/ignite/ => ScalarSharedRDDExample.scala â⯠https://github.com/apache/ignite/ => SharedRDDExample.java 26#EUde9
- 27. Any Questions? Thank you for joining us. Follow the conversation. http://ignite.apache.org 27#EUde9