GKEでは
StackDriver Loggingに
どうやってログを送っているか

Transcript

1. GKEͰ͸
 StackDriver Loggingʹ
 Ͳ͏΍ͬͯϩάΛૹ͍ͬͯΔ͔ 2017/08/08 Kubernetes Meetup Tokyo #6 


   takumi_yoshida_22 / @ytakky2014

2. ࣗݾ঺հ • ytakky2014 / ٢ా ୓࣮
 • github https://github.com/ytakky2014 •

   @y_takky2014 
 https://twitter.com/y_takky2014 • GCP / golang / PHP / CIɾCD / http://info.ytakky.tech/

3. agenda • GKE, StackDriver Loggingʹ͍ͭͯ • StackDriverʹϩάΛૹΔfluentdͷconfʹ͍ͭ ͯ • GKE্ͰfluentdͷΧελϚΠζʹ͍ͭͯ

4. GKE • GoogleCloudPlatform(GCP)ͷ
 KubernetesϑϧϚωʔδυαʔϏε • StackDriver Loggingͱ
 StackDriver MonitoringʹΑΔڧྗͳ؂ࢹ

5. StackDriverLogging • GCP্ͷϩάΛ֨ೲɾ෼ੳͰ͖ΔαʔϏε • agentΛೖΕΕ͹AWS΋ରԠ • BigQuery΁ͷΤΫεϙʔτ
 GCS΁ͷΞʔΧΠϒ 
 Cloud

   Pub/SUB΁ͷετϦʔϛϯά ͕ରԠ
6. None

7. GKE͸Ͳ͏΍ͬͯLogΛ StackDriverLoggingʹૹ͍ͬͯΔͷ͔ • A. fluentd Ͱૹ͍ͬͯΔ

8. ࣮ࡍʹͲ͏΍ͬͯऩू͢Δ͔ • daemonsetsͰશϊʔυ΁഑ஔ͍ͯ͠Δ • https://github.com/kubernetes/kubernetes/blob/ master/cluster/addons/fluentd-gcp/fluentd-gcp- ds.yaml

9. None

10. fluentdͷઃఆΛ֬ೝͯ͠ΈΔ https://github.com/kubernetes/kubernetes/blob/ master/cluster/addons/fluentd-gcp/fluentd-gcp- configmap.yaml

11. ೖྗ <source> type tail format json time_key time path /var/log/containers/*.log

    pos_file /var/log/gcp-containers.log.pos time_format %Y-%m-%dT%H:%M:%S.%N%Z tag reform.* read_from_head true </source>

12. ࣮ࡍ͸ͲͷϑΝΠϧΛऩूͯ͠ ͍Δ͔ • Nodeͷ /var/log/containers/*.logʹSTDOUTʹు͔Εͨϩά͕͋Δ • ࣮ࡍͷϑΝΠϧ໊͸
 [pod໊]_[namespace]_[containerID].log • frontend-3823415956-79vzs_default_php-

    redis-83772f44edd5ee75e921c063d45b710f5053d0a2c86a5c40c3212c b48922bb2c.log • ͜ΕʹreformΛprefixͱͯ͠෇͚Δ
 reform.frontend-3823415956-79vzs_default_php-redis- 83772f44edd5ee75e921c063d45b710f5053d0a2c86a5c40c3212cb489 22bb2c.log
13. None

14. <match reform.**> type record_reformer enable_ruby true tag raw.kubernetes.${tag_suffix[4].split('-') [0..-2].join('-')} </match>

15. <match reform.**> • tag raw.kubernetes.${tag_suffix[4].split(‘-') [0..-2].join('-')} • reform.frontend-3823415956-79vzs_default_php- redis-83772f44edd5ee75e921c063d45b710f505 3d0a2c86a5c40c3212cb48922bb2c.log

    • raw.kubernetes.frontend-3823415956-79vzs_defa ult_php-redis • ʹͳΔ

16. <match raw.kubernetes.**> @type detect_exceptions remove_tag_prefix raw message log stream stream

    multiline_flush_interval 5 max_bytes 500000 max_lines 1000 </match>

17. <match raw.kubernetes.**> • https://github.com/GoogleCloudPlatform/fluent-plugin- detect-exceptions Λ࢖͍ͬͯΔ • ྫ֎ݕग़ϓϥάΠϯɻlogstreamΛεΩϟϯ • ಉ࣌ʹraw.

    Λ֎͍ͯ͠Δ • kubernets.frontend-3823415956-79vzs_default_php- redis

18. <match kubernetes.**> @type google_cloud # Collect metrics in Prometheus registry

    about plugin activity. enable_monitoring true monitoring_type prometheus # Set the buffer type to file to improve the reliability and reduce the memory consumption buffer_type file buffer_path /var/log/fluentd-buffers/ kubernetes.containers.buffer
 ....

19. <match kubernetes.**> • @type google_cloud
 https://github.com/GoogleCloudPlatform/fluent- plugin-google-cloud • StackDriver Logging

    APIʹϩάΛૹ৴͢Δϓϥά Πϯ • ͜ͷϓϥάΠϯͰStackDriverLoggingʹૹ৴ͯ͠ ͍Δ

20. fluentdͷઃఆΛม͍͑ͨ • fluentdͷpluginΛ௥Ճ͍ͨ͠ • fluentdͷconfigͷઃఆΛมߋ͍ͨ͠

21. લ४උ • GKEͷ৔߹ Ϋϥελʔ࡞੒࣌ʹ StackDriverLoggingΛOffʹ͢Δඞཁ͕͋Δ • https://kubernetes.io/docs/tasks/debug- application-cluster/logging-stackdriver/ GKE >=

    1.6Ҏ্͕ඞཁ ※gcloud container clustersͱ͔ίϚϯυͰมߋ΋Ͱ͖ͳ͍...

22. fluentd-pluginΛ௥Ճ͍ͨ͠ • https://github.com/GoogleCloudPlatform/k8s- stackdriver/tree/master/fluentd-gcp-image • GemfileʹඞཁͳpluginΛ௥Ճͯ͠
 docker build / docker

    push • fluentd-gcp-ds.yamlͷimageΛpushͨ͠image ʹมߋͯ͠σϓϩΠ
23. None

24. fluentdͷconfigΛมߋ͢Δ • fluentd-gcp-configmap.yamlΛฤू͢Ε͹OK • 1ϑΝΠϧʹͳ͍ͬͯΔͷͰΘΓͱฤू͕ͭΒ ͍

25. ઃఆมߋޙͷσϓϩΠ • fluentd-gcp-configmap.yamlͱfluentd-gcp- ds.yamlΛ೚ҙͷnamespace΁σϓϩΠ • kubectl create -f fluentd-gcp- configmap.yaml

    • kubectl create -f fluentd-gcp-ds.yaml

26. ·ͱΊ • STDOUTʹϩάΛམͱ͓͚ͯ͠͹ɺ
 GKEͳΒ͹ࣗಈͰStackDriverLogging΁సૹ • fluentd-pluginͷ௥Ճ΍fluentd confͷมߋʹͳΔͱखؒʹ ͳΔ • StackDriverLogging͸Exportग़དྷΔͷͰfluentdͷઃఆͰͲ

    ͏ʹ͔͍ͨ͠Θ͚Ͱ͸ແ͚Ε͹ͦͪΒΛ࢖ͬͨ΄͏͕ྑ͍
 gSlack: Integrating Google Cloud Platform with Slack
 https://blog.doit-intl.com/gslack-9391be7c191a