Maintainable cloud architecture_of_hadoop
5 likes•4,331 views
Kai Sasaki discusses Treasure Data's architecture for maintaining Hadoop on the cloud. Some key points are using stateless services like Hive metastore and cloud storage. They also manage multiple Hadoop versions by downloading packages from S3. Regression tests on Hive queries and a REST API help ensure changes don't cause issues. An RDBMS-based queue provides persistence and scheduling across tasks. The overall aim is high maintainability through statelessness, mobility of components, and queueing of jobs.
Maintainable cloud architecture_of_hadoop
- 1. Maintainable Cloud Architecture of Hadoop Kai Sasaki Treasure Data Inc.
- 2. Who am I? • Kai Sasaki (佐々木 海) • @Lewuathe at Twitter, GitHub • Treasure Data Inc. Software Engineer • Contributing Hadoop, Spark.
- 3. Hadoop in Treasure Data
- 4. Cloud-based Data warehousing service
- 8. Hadoop is the core of Treasure Data
- 9. Hadoop on Cloud 1. Features provided by AWS, IDCF, Heroku etc 2. Fast growing reliability and integrity
- 10. Hadoop on Cloud 1. Features provided by AWS, IDCF, Heroku etc 2. Fast growing reliability and integrity Maintainability of Middleware
- 11. Agenda • Maintainability of Distributed System • Our Challenges • Stateless Hive Metastore • Cloud Storage for Hadoop • Multiple Hadoop Version Management • Regression Test for Hive Queries • REST API for Hadoop • Workflow Integration • What we should keep in mind
- 12. Maintainability We think high maintainability is achieved by… • Stateless
- 13. Maintainability We think high maintainability is achieved by… • Stateless • Mobility
- 14. Maintainability We think high maintainability is achieved by… • Stateless • Mobility • Queueing
- 15. Stateless • Stateless Hive metastore • Cloud Storage for Hadoop
- 18. Stateful Hive MS Driver Metastore MySQL
- 19. Stateful Hive MS Driver Metastore MySQL Require Maintaining RDBMS for only Meta Store
- 20. Stateless Hive MS Driver Metastore
- 21. Stateless Hive MS Driver Metastore Derby
- 22. Stateless Hive MS Driver Metastore Derby Worker Submit DDL request
- 23. Stateless Hive MS Driver Metastore Derby Worker Submit DDL request Aggregate Stateful points Treasure Data API
- 24. Cloud Storage for Hadoop
- 25. PlazmaDB Data Connector S3, Redshift, MySQL, PostgreSQL, Salesforce and more SDK iOS, Android, JavaScript Unity Bulk Import td client ...
- 26. PlazmaDB Data Connector S3, Redshift, MySQL, PostgreSQL, Salesforce and more SDK iOS, Android, JavaScript Unity Bulk Import td client ... msgpack
- 27. PlazmaDB Data Connector S3, Redshift, MySQL, PostgreSQL, Salesforce and more SDK iOS, Android, JavaScript Unity Bulk Import td client ... msgpack Hadoop
- 28. PlazmaDB Data Connector S3, Redshift, MySQL, PostgreSQL, Salesforce and more SDK iOS, Android, JavaScript Unity Bulk Import td client ... msgpack Hadoop Stateful
- 31. Mobility • Multiple Hadoop Version Management • Regression Test for Hive Queries
- 33. Multiple Version Management CDH HDP Apache
- 34. Multiple Version Management CDH HDP Apache client client client
- 35. Multiple Version Management CDH HDP Apache client client client Tough Operation
- 36. Multiple Version Management CDH HDP Apache Worker
- 37. Multiple Version Management CDH HDP Apache Worker switching
- 38. Multiple Version Management CDH HDP Apache Worker switching
- 39. Multiple Version Management CDH HDP Apache Worker CDH package HDP package Apache package switching
- 40. Multiple Version Management CDH HDP Apache Worker CDH package HDP package Apache package S3 switching
- 42. Multiple Version Management CDH package HDP package Apache package S3 /test /stable ...
- 43. Multiple Version Management CDH package HDP package Apache package S3 /test /stable ... CDH HDP Apache Worker download
- 44. Regression Test for Hive • Introducing new features, version up, migration must be done without regression • Running integration system test and regression test for Hive queries
- 50. Queueing • REST API for Hadoop • RDS based Queue management system
- 51. REST API for Hadoop
- 52. REST API for Hadoop CDH HDP Apache Worker
- 53. REST API for Hadoop CDH HDP Apache Worker PerfectQueue Hadoop Job Server REST API
- 54. REST API for Hadoop CDH HDP Apache Worker PerfectQueue Hadoop Job Server REST API Presto API
- 56. RDBMS based queue management CDH HDP Apache Worker Client Client Client PerfectQueue Hadoop Job Server
- 57. PerfectQueue • Highly available distributed queue build on RDBMS • Amazon SQS like API • Resource scheduling for multi tenancy • Graceful and Live Restarting https://github.com/treasure-data/perfectqueue
- 58. What we should keep in mind • Stateless Delegate responsibility to Cloud systems • Mobility Looking ahead for version up, migration • Queueing Make each request persistent
- 59. Recap • Maintainability of Distributed System • Our Challenges • Stateless Hive Metastore • Cloud Storage for Hadoop • Multiple Hadoop version management • Regression Test for Hive queries • REST API for Hadoop • Workflow Integration • What we should keep in mind