Download as PDF, PPTX
Uploaded bySpark Summit
Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher Bradford, Open Source Connections)
Christopher Bradford presented lessons learned from using Apache Spark at the US Patent and Trademark Office to improve their process of loading data from Cassandra into Solr. The initial Spark implementation performed poorly due to opening and closing a Solr connection for each document. Optimizations like opening a single connection per partition and pushing documents in batches significantly improved performance, resulting in a solution that was 5 times faster than the original process. Future work involves further optimizing this Spark job and exploring additional uses of Spark.
More Related Content
Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher Bradford, Open Source Connections)
- 1. Lessons Learned with Sparkat the US Patent & Trademark Office Christopher Bradford Big Data Architect at OpenSource Connections
- 2.
- 3.
- 4.
- 5. EST – TechnologyStack
- 6. EST – DataLoading CSS Ingestion (CSS2C) Solr Ingestion (C2S)
- 7. EST – C2SProcess Note: some connections are omitted for clarity
- 8. EST – C2SProcess (Scaled Out) Note: some connections are omitted for clarity
- 9. EST – C2SReview Did it work? Why change it? How could we make it better?
- 11. EST – OldC2S Process Note: some connections are omitted for clarity
- 12. EST – SparkC2S Process Note: some connections are omitted for clarity
- 13. How did thiswork out? Poorly
- 14. Poor Performance joinedRDD =… joinedRDD.foreach() document = … // build document sc = new SolrConnection() sc.push(document) sc.disconnect() // Job is done
- 15. Poor Performance sc =new SolrConnection() sc.push(document) sc.disconnect()
- 16. Optimum Performance joinedRDD =… sc = new SolrConnection() joinedRDD.foreach() document = … // build document sc.push(document) sc.disconnect() // Job is done joinedRDD = … joinedRDD.foreachPartition() sc = new SolrConnection() partition.foreach() document = … // build document sc.push(document) sc.disconnect() // Job is done Almost
- 17. The Solution! joinedRDD =… joinedRDD.mapPartitions() sc = new SolrConnection() partition.foreach() document = … // build document sc.push(document) sc.close() return partition.rows .collect() joinedRDD = … joinedRDD.mapPartitions() sc = new SolrConnection() partition.foreach() document = … // build document sc.push(document) sc.close() return partitions.rows.count .collect()
- 18.
- 19.
- 20. Better Solr Indexing Note:some connections are omitted for clarity
- 21. EST – SparkC2S Process v2 Note: some connections are omitted for clarity
- 22. Success? YUP 5x faster thanthe original C2S process (with optimizations)
- 23. What’s Next? • Optimizationof the C2S Spark job • More Spark jobs • Newer version of Spark & DSE • Scala Spark jobs instead of Java