20+ custom mackerel plugins in 20 minutes

Transcript

1. 20+ Custom mackerel Plugins in 20 minutes mackerel Meetup#7 Masahiro

   Nagano @kazeburo

2. Me • Masahiro Nagano • @kazeburo • Mercari, Inc.
 Principal

   Engineer
 Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor

3. We’re Hiring!! ΑΖ͓͘͠ئ͍͠·͢ʂ

4. Mercari Your Friendly Mobile MarketPlace JP US

5. Infrastructure ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West (Oregon)

   Region JP US

6. Monitoring (past) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West

   (Oregon) Region JP US Z ZABBIX ҟͳΔzabbixͷόʔδϣϯ.. WebUIΆͪΆͪർΕ.. ؂ࢹ௥Ճ·Ͱͷεςοϓͷଟ͞.. ؂ࢹ࿙ΕɺҟͳΔ؂ࢹ಺༰.. Z ZABBIX

7. Monitoring (now) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West

   (Oregon) Region JP US JP/USͷҰׅ؂ࢹ Service & RoleʹΑΔ؅ཧ Nagios & Sensu ϕʔεͷPlugin

8. Migration Items, Graphs Screens, Triggers Auto Discovery, UserParameters Official mackerel

   plugins Official check plugins Nagios plugins Custom mackerel Plugins Custom check Pluginss + Simple Monitoring Rules Z ZABBIX ࣜʹΑΔ ؂ࢹ จࣈྻɺ਺஋ɺ ࠩ෼ɺܭࢉ݁ՌͳͲͷ σʔλܗࣜ ΧελϜσʔλͷ औಘ

9. Custom Plugins my-ec2-tag[go], jmx-get[go], diff-detector[go], delay-checker, interval-checker, periodic-checker, check-mysql-uptime[go], check-memcached-uptime[go],

   check-conntrack-free, check-crt-expiration, check-dns-rr, check-hydra-pos, check-inode, check-iptables, check-myip, check-solr-replication, check-solr-update, check-uptime, check-mysql-msr[go], mackerel-plugin-accelmail-counter, mackerel-plugin-gaurun-usage, mackerel-plugin-linux-lite, mackerel-plugin-msr[go] mackerel-plugin-ntpq, mackerel-plugin-search, mackerel-plugin-postfix mackerel-plugin-php-and-accesslog, mackerel-plugin-php-version and MORE! and utilitiy commands

10. my-ec2-tags $ my-ec2-tag --tag Role production EC2ΠϯελϯεͷλάΛऔಘ awsίϚϯυ஗͍ͷͰGoͰ࡞੒ ޙड़ͷdiff-detectorͱ૊Έ߹Θͤͯ࢖͏ Utility

11. delay-checker $ delay-checker -c 3 -- check-tcp -p 80 ࢦఆͨ͠ճ਺Τϥʔ͕࿈ଓ͢ΔͱNG

    max_check_attempts Φϓγϣϯ͕Ͱ͖ͨ check ؂ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins

12. interval-checker $ interval-checker --interval 300 -- \ my-ec2-tag --tag Role

    ࢦఆִͨؒ͠Ͱ؂ࢹίϚϯυ࣮ߦ ࢦఆͨ࣌ؒ͠·Ͱ͸લճͷίϚϯυ࣮ߦ݁ՌΛฦ͢ check ؂ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins

13. periodic-checker $ periodic-checker --range 00:00-06:00,06:30-24:00 \ -- check-tcp -p 80

    ࢦఆͨ͠ൣғͷ͚࣌ؒͩ؂ࢹ ͦΕҎ֎͸OKΛฦ͢ Daily ϝϯςφϯεʗॏ͍batch͕͋Δͱ͖ʹ check ؂ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins

14. diff-detector $ diff-detector -- date NG: detect difference: ```@@ -1

    +1 @@ -Tue May 10 08:11:42 UTC 2016 +Tue May 10 08:11:43 UTC 2016``` ίϚϯυ࣮ߦ݁Ռͷࠩ෼͕͋ΔͱΤϥʔ ec2 tagɺ/etc/passwdϑΝΠϧͷ؂ࢹ ▶ https://github.com/kazeburo/diff-detector check ؂ࢹ

15. check-conntrack-free $ check-conntrack-free -w 10000 -c 2000 OK: free:596376 max:655360

    count:58984 `nf_conntrack_max - nf_conntrack_count` ͷ ݁Ռͷ؂ࢹɻiptables͕༗ޮͳ࣌ʹ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins

16. check-crt-expiration $ check-crt-expiration -w 7 -c 5 -f /path/cert.pem OK:

    Expiration date: 'Jun 1 10:38:57 2016 GMT'. 20 days left ূ໌ॻͷ༗ޮظݶͷ֬ೝ check-cert-file͕ެࣜʹ௥Ճ͞Εͨ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins

17. check-dns-rr $ check-dns-rr -w 3 -c 2 \ --host prod.example.service.consul

    OK: 3 dns-rr hosts found DNS-RRͷ݁Ռʹؚ·ΕΔIP਺ͷ؂ࢹ Consul/DNSͰ෼ࢄ͍ͯ͠Δ࣌ʹ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins

18. check-inode $ check-inode -w 10 -c 5
 OK: /:1%, /dev:1%,

    /dev/shm:1%, /run:1%, /sys/fs/cgroup:1%, /boot: 1%, /run/user/1037:1% ͢΂ͯͷfilesystemͷinodeΛ؂ࢹ Ұ൪গͳ͍΋ͷ͕ᮢ஋ΛԼճΔͱNG mackerel-plugin-inode ͸ `inode.percentage.*.used` ʹରͯ͠؂ࢹ͕Ͱ͖ͳ͍ͷͰ࢖͍ʹ͍͘ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins

19. check-memcached-uptime check ؂ࢹ $ check-memcached-uptime \ -p 11211 -w 110

    -c 110 memcached Uptime OK: up 240 days, 03:08:01 ▶ https://github.com/kazeburo/go-check-memcached-uptime memcachedͷಥવࢮ => ࠶ىಈΛݕ஌͢Δ check-memcached ͸keyͷऔಘΛߦ͏

20. check-mysql-uptime check ؂ࢹ $ check-mysql-uptime --host=127.0.0.1 \ -w 110 -c

    110 MySQL Uptime OK: up 71 days, 17:47:39 pingͰ͸λΠϛϯάʹΑΓҾ͔͔ͬΒͳ͍࠶ىಈΛݕ஌ MySQL 4.0.xͳͲʹ༗ޮ `check-mysql uptime` Ͱ΋Մೳ ▶ https://github.com/kazeburo/go-check-mysql-uptime

21. check-mysql-msr $ check-mysql-msr --host localhost -w 5 -c 10 MySQL

    Multi Source Replication OK: [O]foo-db=io:Yes,sql:Yes,behind:0 bar-db=io:Yes,sql:Yes,behind:0 main-db=io:Yes,sql:Yes,behind:0 baz- db=io:Yes,sql:Yes,behind:0 MySQL 5.7ͷMulti Source Replicationͷ༻ ͲΕ͔ҰͭͰ΋Replicationͷఀࢭ͋Δ͍͸ɺ஗Ԇ͕ᮢ஋Λ্ճΔͱNG mackerel-plugin-msr ΋ผʹ࡞ͬͨ inodeಉ༷ `mysql-msr.*.beind` ʹରͯ͠؂ࢹ͕Ͱ͖ͳ͍ͷͰ࢖͍ʹ͍͘ check ؂ࢹ ▶ https://github.com/kazeburo/go-check-mysql-msr

22. mackerel-plugin-linux-lite Free Memory, 100%্͕ݶͷCPU Usage CPUίΞ͋ͨΓͷϩʔυΞϕϨʔδ, ϓϩηε਺ Ұ؏ͨ͠ϝτϦΫεͷ൑அɺ؂ࢹͷᮢ஋ͷઃఆ͕Ͱ͖Δ Agent Plugin

    ▶ https://github.com/kazeburo/custom-mackerel-plugins

23. mackerel-plugin-ntpq ࣌ؒͷͣΕ(ઈର஋)ɺϦϞʔτͱͷSyncͷ༗ແ check-ntpoffset ΋͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins

24. mackerel-plugin-postfix Postfixͷqueue mackerel-plugin-mailq ΋͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins

25. mackerel-plugin-mysql-lite ίωΫγϣϯ਺/max_connectionͷׂ߹ɺϨϓϦέʔγϣϯ஗Ԇ Slave SQL Thread/IO Threadͷಈ࡞༗ແɺ εϨου਺ KuradoͰ΋ϝτϦΫεΛݟ͍ͯΔͷͰɺ؂ࢹʹඞཁͳάϥϑʹߜΔ ▶ https://github.com/kazeburo/go-mackerel-plugin-mysql-lite

    Agent Plugin

26. mackerel-plugin-php-version ݹ͍PHPΛݕ஌^^ Agent Plugin

27. Plugins & Language • ͍͔ͭ͘ͷίϚϯυͷݺͼग़͠ͷΈ
 = > Shell Script •

    ίϚϯυͷݺͼग़͠ͱ݁Ռͷॲཧ => LL (Perl) • HTTPϦΫΤετͱJSONॲཧ => LL (Perl) • AWS APIɺMySQL΁ͷ઀ଓ =>Golang ͜ΕΒΛ࢖͍׳Ε͍ͯΔ͜ͱ͕ॏཁ

28. ίʔυΛॻ͍ͯ໰୊Λղܾ͢Δͷ͸ָ͍͠!! mackerel͸࢖͍͕͍ͷ͋Δπʔϧ