Batch Indexing & Near Real Time, keeping things fast
3 likes2,700 views
This document summarizes Marc Sturlese's presentation on Trovit's architecture for batch indexing and near real-time search. Trovit uses Hadoop and Hive for batch processing and indexing, which occurs periodically in batches and can take hours. For near real-time search, Trovit uses Storm to process documents as they arrive and index them to Solr. Storm topologies read from Kafka queues and index documents to Solr in small bulks to reduce load. Challenges include committing indexes across replicas and handling index swaps between batch and real-time layers. The architecture aims to provide low-latency search while avoiding segment merges for optimal performance.
Batch Indexing & Near Real Time, keeping things fast
- 1. Batch Indexing & Near Real Time, keeping things fast Marc Sturlese Software engineer @ Trovit Thursday, 2 May 2013
- 2. About me... • Marc Sturlese – @sturlese • Software engineer @Trovit. R&D focused • Responsible for search and scalability Thursday, 2 May 2013
- 3. Agenda • Who we are • Batch architecture. Hadoop & Hive • Near real time architecture. Storm & stuff • Putting it all together • Alternatives and Future directions • Questions Thursday, 2 May 2013
- 4. Who we are Trovit, a search engine for classifieds Thursday, 2 May 2013
- 5. Who we are Thursday, 2 May 2013
- 6. Batch Layer • Hadoop based • Documents are crunched by a pipeline of MR jobs • Hive to save stats of each phase Thursday, 2 May 2013
- 7. Batch Layer Pipeline overview Incoming data Deployment Lucene Indexes Ad Processor Diff Matching Expiration Deduplication Indexing t – 1 External Data Hive Stats Hadoop Cluster Thursday, 2 May 2013
- 8. Batch Layer The good things! • Index always built from scratch. Small number of big segments • Multicast deployment allows to send indexes to all slaves at the same time. • Backups convenient on HDFS Thursday, 2 May 2013
- 9. Batch Layer That was cool but... • Not even close to real time • Crunch documents in batch means to wait until all is processed. This can take a few hours • We want to show the user fresher results! Thursday, 2 May 2013
- 10. Near real time Layer Storm and stuff to the rescue Thursday, 2 May 2013
- 11. Near real time Layer Storm properties • Distributed real time computation system • Fault tolerance • Horizontal scalability • Low latency • Reliability Thursday, 2 May 2013
- 12. Near real time Layer Storm in action Slave Slave Solr prod replicas Slave XML feed XML feed Kafka partition Kafka partition Storm topologySources Kafka spout Kafka spout XML spout Doc Manager bolt Indexer bolt SHUFFLE GROUPING GROUPING FIELD Thursday, 2 May 2013
- 13. Near real time Layer Storm in action • Spouts just read and send • Doc Manager Bolt processes and classifies • Indexer Bolt adds documents to Solr • Replicated logic with different implementation • Careful not to overload Solr slaves... Thursday, 2 May 2013
- 14. Near real time Layer Storm in action Thursday, 2 May 2013
- 15. Near real time Layer Storm in action. But... Thursday, 2 May 2013
- 16. Near real time Layer Storm in action. But... • Now Solr has to handle user queries and storm inserts • Field grouping on Indexer Bolt for politeness • Small bulks to reduce insert requests • Committing on many cores, same host, same time can be painful Thursday, 2 May 2013
- 17. Near real time Layer Storm in action - Committing Indexer Bolt Cars US Real state UK R1 Cars US R1 Cars US R2 Jobs BR R1 Jobs BR R2 Real state ES R1 Indexer Bolt Jobs BR ZooKeeper Locker Slave 1 Slave 2 Slave N . . . Thursday, 2 May 2013
- 18. Near real time Layer Storm in action • Adding documents now is fast • Keep number of segments small • Avoid merges on big segments • Just add new docs (no deletes or updates) Thursday, 2 May 2013
- 19. Mixed Architecture Putting it all together 15 Slave Slave Solr prod replicas Slave XML feed XML feed Kafka partition Kafka partition Storm topologySources Hbase doc info Bulk add Exists? MR Pipeline zk Thursday, 2 May 2013
- 20. Mixed Architecture Swapping indexes • NRT docs might not be contained in the new batch index (even fresher than the “being built” batch index) • This can lead to inconsistencies... Thursday, 2 May 2013
- 21. Mixed Architecture Swapping indexes. Time jumps! Thursday, 2 May 2013
- 22. Mixed Architecture Swapping indexes HBase XML feed t Slave t+1 Slave t Pipeline t Pipeline t+1 XML feed t+1 XML feed t+2 NRT indexer Batch indexer Thursday, 2 May 2013
- 23. Mixed Architecture Swapping indexes HBase XML feed t Slave t+1 Slave t Pipeline t Pipeline t+1 XML feed t+1 XML feed t+2 NRT indexer Batch indexer Thursday, 2 May 2013
- 24. Mixed Architecture Swapping indexes HBase XML feed t Slave t+1 Slave t Pipeline t Pipeline t+1 XML feed t+1 XML feed t+2 NRT indexer Batch indexer NRT t+1 NRT t+2 Thursday, 2 May 2013
- 25. Mixed Architecture Swapping indexes HBase XML feed t Slave t+1 Slave t Pipeline t Pipeline t+1 XML feed t+1 XML feed t+2 NRT indexer Batch indexer NRT t+1 NRT t+2 Thursday, 2 May 2013
- 26. Mixed Architecture Swapping indexes • NRT indexed docs must be stored in a temporary storage • Fetch missing docs from the storage and add them before the next deploy • This avoids time jumps Thursday, 2 May 2013
- 27. Mixed Architecture Storm and Hadoop • Near real time inserts, low latency • Hadoop handles deletes and updates. No rush on those • No merges on big segments so optimal query response times • Tolerant to human errors • Temporary lost of accuracy on the NRT layer Thursday, 2 May 2013
- 28. Alternatives SolrCloud - Why not? • Good for the vast majority of use cases • Incremental inserts/updates/deletes oriented. Pay segment merges per real time • Need to deploy full indexes fast (faster that rsync or http replication) • Now full deploy easier with aliases Thursday, 2 May 2013
- 29. Future lines Lucene real time feature • Allows to see docs in the index before they are committed • Good but not a must right now for the use case • Very easy to integrate on the current architecture Thursday, 2 May 2013
- 30. ??Thursday, 2 May 2013
- 31. Thanks for your attention! Marc Sturlese [email protected] Lucene/Solr Revolution 2013, San Diego, May 1 2013 Thursday, 2 May 2013
- 32. CONFERENCE PARTY The Tipsy Crow: 770 5th Ave Starts after Stump The Chump Your conference badge gets you in the door TOMORROW Breakfast starts at 7:30 Keynotes start at 8:30 Thursday, 2 May 2013