LINEデリマでのElasticsearchの運用と監視の話
1 like•1,665 views
The document outlines details from the Developer Meetup #27 focused on Elasticsearch, highlighting topics such as admin procedures, shard allocation, and monitoring with Prometheus and Grafana. It covers technical specifics regarding master and data nodes, thread pool configurations, and alerts. Additionally, it discusses Elasticsearch exporters and integration with monitoring tools for performance oversight.
LINEデリマでのElasticsearchの運用と監視の話
- 1. Developer Meetup #27 LINE Elasticsearch Kotaro WATANABE, Development Support 2018/01/30
- 2. Developer Meetup #27 l (Kotaro WATANABE) l (Development Support) l Infra Engineer (2016/04~) l l LINE l LINE LIVE l LINE l LINE l LINE l Clova l Etc.
- 3. Developer Meetup #27 l LINE l Elasticsearch l Elasticsearch l l
- 4. Developer Meetup #27 l LINE Elasticsearch LINE
- 5. ˆLINE
- 6. Developer Meetup #27 l LINE l l 2017 7 26 l LINE : @linedelima l 1100 l Elasticsearch LINE
- 7. Developer Meetup #27 LINE ( )
- 8. Developer Meetup #27 LINE ( )
- 10. Developer Meetup #27 l Java Client l Transport Client l Indices l block ( ) l item ( ) l shop ( ) l Dashboard l Imenezes/cerebro
- 11. ˆElasticsearch
- 12. Developer Meetup #27 LINE l Topics 1. 2. Admin Elasticsearch 3.
- 13. Developer Meetup #27 1. l l Elasticsearch l 1 4000rps l 2
- 15. Developer Meetup #27 l l Master node l l VM 3 l discovery.zen.minimum_master_nodes: 2 l Primary shard Replica shard l Primary shard: 5 (Data node ) l Replica shard: 1 l 1node 2 shard
- 17. Developer Meetup #27 l 20 l Master node l Master node Data node l discovery.zen.minimum_master_nodes l (master_eligible_nodes / 2) + 1 = 2 l batch indexing rebalance allocation l l Shard Allocation Settings l 5Mbps 20Mbps
- 18. Developer Meetup #27 2. Admin Elasticsearch l Admin Elasticsearch l Data node l Forced Awareness l /path/to/elasticsearch.yml l e.g. /etc/elasticsearch/elasticsearch.yml # for service zone node.attr.zone: service cluster.routing.allocation.awareness.force.zone.values: service,admin cluster.routing.allocation.awareness.attributes: zone # for admin zone node.attr.zone: admin cluster.routing.allocation.awareness.force.zone.values: service,admin cluster.routing.allocation.awareness.attributes: zone
- 20. Developer Meetup #27 zone l zone index l zone shard replica l Primary shard Replica shard l shard rebalance l zone index l zone Elasticsearch zone l admin
- 21. Developer Meetup #27 3. l l l 4000qps l search Thread queue
- 22. Developer Meetup #27 l l Thread Pool l int((available processors * 3) / 2) + 1 l available processors 32 l (32 * 3 / 2) + 1 = 49 ( Thread Pool ) l processors (PM : 40) l (40 * 3) / 2 + 1 = 61 l Queue (1000 → 3000) l thread_pool.search.queue_size: 3000 l ES5 ES6
- 24. Developer Meetup #27 (2018/1/2 21:00~23:00)
- 25. Developer Meetup #27 (2018/1/9 21:00~23:00)
- 26. Developer Meetup #27 l Master node Data node l service admin zone l Data Node(PM) l
- 27. ˆElasticsearch
- 28. Developer Meetup #27 l l l l IMON (Monitoring + Alerting) l Java l PMC
- 29. Developer Meetup #27 l Prometheus (Alerting) + Grafana (Monitoring) ← NEW l line/promgen l LINE Prometheus l l l l IMON (Monitoring + Alerting) l Java l PMC
- 30. Developer Meetup #27 Prometheus Architecture : https://prometheus.io/docs/introduction/overview/#architecture
- 31. Developer Meetup #27 Prometheus + Grafana + Promgen l Prometheus l Pull l agent l *_exporter l http l EXPORTERS AND INTEGRATIONS l Grafana l Prometheus DataSource l exporter Overview Dashboard l Promgen l line/promgen l LINE Prometheus
- 32. Developer Meetup #27 Elasticsearch Prometheus l elasticsearch_exporter l justwatchcom/elasticsearch_exporter l GET /_nodes/stats Prometheus exporter l Promgen Prometheus l Alertmanager push post l l e.g. elaseticsearch_cluster_health_status{color=“red”} == 1
- 35. Developer Meetup #27 l Prometheus + Grafana l l line/promgen l LINE Engineering Blog l PromCon 2017: Prometheus as an (internal) Service l Prometheus Casual Talks
- 36. ˆ
- 37. Developer Meetup #27 l Client l Transport Client High Revel REST Client l Elasticsearch l 5 6 l
- 38. ˆ
- 39. Developer Meetup #27 THANK YOU