SlideShare a Scribd company logo

Logging & Metrics

Tammo van Lessen, profile picture
Tammo van Lessen

The document discusses logging and metrics within software systems, highlighting the importance of logging for debugging and health status while cautioning against using it for system health checks. It elaborates on different types of metrics including counters, gauges, histograms, and timers, emphasizing best practices such as measuring everything and implementing retention policies. Additionally, it briefly mentions tools like Logstash, Graylog, and Kafka for managing logs and metrics.

TechnologyBusiness
1 of 47
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
Logging & Metrics Tammo van Lessen Christian Grobmeier @taval @grobmeier
log.errorſ"Houston?"ƀ;
NrɬwLɩsGnT FUCkQJɫYcJ jUQwjIaagt uVimjWTocH xɪUNWɰXZɨB dtNpSSkɬfw ADjɬɭfqlnt IOHlɫHACmU mɫJɨiɪgɫɬp wmɬommkSvh ttjWddKɰɩL VZdCOlɮSjg epqMlTntGL toɯBo00WMi NɪTzEjFɩZɪ rsɰwɪeɨZLɩ ɬXɭumEbaiu ɭCLɯHBEKɩX PihltWebCI VtuɮxhTBdf ɯHɪmhqY0ug ɰɨWvNUYCtɯ rMpEVUiwUA lɯSaAnGNto nPcoiO0rhq TYtɫpMrnrt HOUSTON!!! ɭVKJqPAnbO VɰqofvZZfq zɩlɰɮjCNqQ M0IFOjnNyg ɫMBɩbɪqcIH KXkɬypeVTɭ ZIHciɩrLyi DdLiDNuYhɬ CIhpkɯɰxEa pKSNIɫzNtB rQɬCAZSUwc MLYɨERwCɯX yZQTzyOpZb TOOrfLATEv RfLwPsLfpQ ɨcmPmjCXwɭ ɭRXɪrqEqUa UzzEmCafve BNlRGzhDhɯ ɪtNdDfhQLn TSɬKPpmIOR sntɯemAsgZ WxKMɩɭsPbt NkAhɯOFRHK rzɫjPlRɪQg OhoWɰUiOzB RBZcɫNbjCA fgjchgɮKHL pXɰGɪTɫTti
Logging & Metrics
Logging & Metrics
Logging & Metrics
MDC to the rescue? ThreadContext.putſ"loginId", loginƀ; logger.errorſ"Something bad happened!"ƀ; ThreadContext.clearſƀ; + Layout: %-ɬp: %m%n %X{loginId} Could log: ERROR: Something bad happened! @grobmeier
Enough?
Dealing with the Stream Of Logs Flume, Kafka, Logstash, Graylog2, Fluentd
Searching logs Graylog2, ElasticSearch
Having an overview
Logging & Metrics
A Logstash Cluster From the Logstash docs
May I help you? FluentD (AL 2.0) Graylog 2 (GPL) Loggly (commerical)
If you want to know... ...you need to log.
How much is healthy?
Trace is almost for free. Log4j 2 Filters help too.
Default Levels Files? Warn only. Logstash & Co? Info. Magic bugs + advanced setup? Debug, or even trace.
Start with verbs DEBUG adding user @grobmeier on node "baldur" DEBUG removing user @taval on node "mjölnir" DEBUG created entry for @taval on node "baldur" Idea taken from TJ Holowaychuk
Logging is cool. Can I use it to get a health status of my systems?
You can! But you shouldn't!
Metrics Business Metrics Application Metrics System Metrics
Gauges An instrument that measures a value.
Counters A counter is a simple incrementing and decrementing integer.
Histograms A Histogram measures the distribution of values.
Meters A meter measures the rate at which a set of events occur.
Timers
Logging & Metrics
Logging & Metrics
Logging & Metrics
Logging & Metrics
Logging & Metrics
Logging & Metrics
Logging & Metrics
Logging & Metrics
Logging & Metrics
Dashboards
Graphite
Cubism.js Mirror Offset 1 − + Credits: Michael Bostock
Logging & Metrics
Comparisons var cube = context.cubeſ"http://..."ƀ, primary = cube.metricſ"sumſrequestƀ"ƀ, secondary = primary.shiftſ-ɮ * ɩɫ * ɭ0 * ɭ0 * ɨ000ƀ;
Graphene ...
Dashing ...
Best practices Measure everything! Counters ./. Meters Metrics are cheap, but not for free. Retention Policies
Prevent the apocalypse! Logging shows events. Metrics shows state.
Thank you! We use Twitter! @taval, @innoq & @grobmeier
Credits Holzstapel CPU by mbostock Mess by Rev Stan Debt by "Images Money" (Flickr) Pay Here by Marc Falardeau Cockpit by Ronnie Rams Stream by Phil Whitehouse Magnifier by John Lodder (Flickr) Flying Saucer, Cup, and Teapot! by Mr Thinktank Ice berg by Derek Keats Gas Meters by mxmstryo (Flickr) Gauge Stock by Andrew Taylor (Flickr) Counter by Marcin Wichary (Flickr) Histogram of legos by color frequency by Jeff Boulter (Flickr) pomodoro timers by Paul Downey (Flickr) Zombie Apocalypse by pasukaru76

More Related Content

PPTX
Presenting statistics in social media 2012
University of Pittsburgh
 
PPTX
Presenting statistics in social media
University of Pittsburgh
 
PDF
Quality Assurance and Testing of Automated Business Processes
Tammo van Lessen
 
PDF
Cloud Foundry Logging and Metrics
Ed King
 
PPT
What Every Organization Should Log And Monitor
Anton Chuvakin
 
PDF
Monitoring Kubernetes with Prometheus
Tobias Schmidt
 
PDF
Metrics & more
Stefan Thies
 
PDF
Detection of Process Antipatterns: An BPEL Perspective
Francis Palma
 
Presenting statistics in social media 2012
University of Pittsburgh
 
Presenting statistics in social media
University of Pittsburgh
 
Quality Assurance and Testing of Automated Business Processes
Tammo van Lessen
 
Cloud Foundry Logging and Metrics
Ed King
 
What Every Organization Should Log And Monitor
Anton Chuvakin
 
Monitoring Kubernetes with Prometheus
Tobias Schmidt
 
Metrics & more
Stefan Thies
 
Detection of Process Antipatterns: An BPEL Perspective
Francis Palma
 

Viewers also liked (20)

PPT
Business Process Execution Language (BPEL)
Richard Claassens CIPPE
 
PPTX
Introduction to business process execution language
Patel Saunak
 
PPT
Learn Dashing Widget in 90 minutes
Larry Cai
 
PDF
Web Services Orchestration with BPEL 2.0
Tammo van Lessen
 
PDF
Formalizing Message Exchange Patterns using BPEL light
Tammo van Lessen
 
PDF
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Brian Brazil
 
PDF
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Brian Brazil
 
PDF
Business Process Management with BPMN & BPEL
Tammo van Lessen
 
PDF
Data Platform at Twitter: Enabling Real-time & Batch Analytics at Scale
Sriram Krishnan
 
PDF
OpenESB et BPEL
Lilia Sfaxi
 
PDF
Elasticsearch for Logs & Metrics - a deep dive
Sematext Group, Inc.
 
PPT
Oracle Service Bus vs. Oracle Enterprise Service Bus vs. BPEL
Guido Schmutz
 
PDF
Prometheus Overview
Brian Brazil
 
PDF
Introduction and Advanced Concepts of BPEL
Denis Weerasiri
 
PDF
Fall in Love with Graphs and Metrics using Grafana
torkelo
 
PDF
Fluentd and Kafka
N Masahiro
 
PPTX
Volta: Logging, Metrics, and Monitoring as a Service
LN Renganarayana
 
PDF
BPMN, BPEL, ESB or maybe Java? What should I use to implement my project?
Guido Schmutz
 
PPTX
Grafana and MySQL - Benefits and Challenges
Philip Wernersbach
 
PDF
Demystifying Data Engineering
nathanmarz
 
Business Process Execution Language (BPEL)
Richard Claassens CIPPE
 
Introduction to business process execution language
Patel Saunak
 
Learn Dashing Widget in 90 minutes
Larry Cai
 
Web Services Orchestration with BPEL 2.0
Tammo van Lessen
 
Formalizing Message Exchange Patterns using BPEL light
Tammo van Lessen
 
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Brian Brazil
 
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Brian Brazil
 
Business Process Management with BPMN & BPEL
Tammo van Lessen
 
Data Platform at Twitter: Enabling Real-time & Batch Analytics at Scale
Sriram Krishnan
 
OpenESB et BPEL
Lilia Sfaxi
 
Elasticsearch for Logs & Metrics - a deep dive
Sematext Group, Inc.
 
Oracle Service Bus vs. Oracle Enterprise Service Bus vs. BPEL
Guido Schmutz
 
Prometheus Overview
Brian Brazil
 
Introduction and Advanced Concepts of BPEL
Denis Weerasiri
 
Fall in Love with Graphs and Metrics using Grafana
torkelo
 
Fluentd and Kafka
N Masahiro
 
Volta: Logging, Metrics, and Monitoring as a Service
LN Renganarayana
 
BPMN, BPEL, ESB or maybe Java? What should I use to implement my project?
Guido Schmutz
 
Grafana and MySQL - Benefits and Challenges
Philip Wernersbach
 
Demystifying Data Engineering
nathanmarz
 
Ad

More from Tammo van Lessen (9)

PDF
Behavior-driven Business Process Development with BPMN
Tammo van Lessen
 
PDF
Vorsicht Schuldenfalle - Was die IT aus der Finanzwelt lernen kann
Tammo van Lessen
 
PPT
Web Service Composition mit WS-BPEL und dem Open-Source-Orchester
Tammo van Lessen
 
PDF
BPM meets Semantic Web
Tammo van Lessen
 
PDF
Virtualizing Services and Resources with ProBus: The WS-Policy-Aware Service...
Tammo van Lessen
 
PPT
SOA-based Business Integration with Eclipse BPEL and Apache ODE
Tammo van Lessen
 
PDF
Facilitating Rich Data Manipulation in BPEL using E4X
Tammo van Lessen
 
PDF
BPM meets Semantic Web
Tammo van Lessen
 
PDF
An Execution Engine For Semantic Business Processes
Tammo van Lessen
 
Behavior-driven Business Process Development with BPMN
Tammo van Lessen
 
Vorsicht Schuldenfalle - Was die IT aus der Finanzwelt lernen kann
Tammo van Lessen
 
Web Service Composition mit WS-BPEL und dem Open-Source-Orchester
Tammo van Lessen
 
BPM meets Semantic Web
Tammo van Lessen
 
Virtualizing Services and Resources with ProBus: The WS-Policy-Aware Service...
Tammo van Lessen
 
SOA-based Business Integration with Eclipse BPEL and Apache ODE
Tammo van Lessen
 
Facilitating Rich Data Manipulation in BPEL using E4X
Tammo van Lessen
 
BPM meets Semantic Web
Tammo van Lessen
 
An Execution Engine For Semantic Business Processes
Tammo van Lessen
 
Ad

Recently uploaded (20)

PDF
A comparative study of natural language inference in Swahili using monolingua...
IAESIJAI
 
PDF
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PPTX
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 
PPTX
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...
bonakiduza1024
 
PDF
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
PPTX
Chapter 5: Probability Theory and Statistics
RandyFin
 
PDF
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PDF
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
PDF
project resource management chapter-09.pdf
ssuser00e6e21
 
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
PPTX
SOPHOS-XG Firewall Administrator PPT.pptx
AgnesMunisi
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
PPTX
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
PDF
Heart disease approach using modified random forest and particle swarm optimi...
IAESIJAI
 
PDF
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
IAESIJAI
 
PDF
Getting Started with Data Integration: FME Form 101
Safe Software
 
A comparative study of natural language inference in Swahili using monolingua...
IAESIJAI
 
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Encapsulation theory and applications.pdf
gurumoop
 
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...
bonakiduza1024
 
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
Chapter 5: Probability Theory and Statistics
RandyFin
 
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
project resource management chapter-09.pdf
ssuser00e6e21
 
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
SOPHOS-XG Firewall Administrator PPT.pptx
AgnesMunisi
 
Approach and Philosophy of On baking technology
30anthuong17
 
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
Heart disease approach using modified random forest and particle swarm optimi...
IAESIJAI
 
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
IAESIJAI
 
Getting Started with Data Integration: FME Form 101
Safe Software
 

Logging & Metrics

  • 1. Logging & Metrics Tammo van Lessen Christian Grobmeier @taval @grobmeier
  • 3. NrɬwLɩsGnT FUCkQJɫYcJ jUQwjIaagt uVimjWTocH xɪUNWɰXZɨB dtNpSSkɬfw ADjɬɭfqlnt IOHlɫHACmU mɫJɨiɪgɫɬp wmɬommkSvh ttjWddKɰɩL VZdCOlɮSjg epqMlTntGL toɯBo00WMi NɪTzEjFɩZɪ rsɰwɪeɨZLɩ ɬXɭumEbaiu ɭCLɯHBEKɩX PihltWebCI VtuɮxhTBdf ɯHɪmhqY0ug ɰɨWvNUYCtɯ rMpEVUiwUA lɯSaAnGNto nPcoiO0rhq TYtɫpMrnrt HOUSTON!!! ɭVKJqPAnbO VɰqofvZZfq zɩlɰɮjCNqQ M0IFOjnNyg ɫMBɩbɪqcIH KXkɬypeVTɭ ZIHciɩrLyi DdLiDNuYhɬ CIhpkɯɰxEa pKSNIɫzNtB rQɬCAZSUwc MLYɨERwCɯX yZQTzyOpZb TOOrfLATEv RfLwPsLfpQ ɨcmPmjCXwɭ ɭRXɪrqEqUa UzzEmCafve BNlRGzhDhɯ ɪtNdDfhQLn TSɬKPpmIOR sntɯemAsgZ WxKMɩɭsPbt NkAhɯOFRHK rzɫjPlRɪQg OhoWɰUiOzB RBZcɫNbjCA fgjchgɮKHL pXɰGɪTɫTti
  • 7. MDC to the rescue? ThreadContext.putſ"loginId", loginƀ; logger.errorſ"Something bad happened!"ƀ; ThreadContext.clearſƀ; + Layout: %-ɬp: %m%n %X{loginId} Could log: ERROR: Something bad happened! @grobmeier
  • 9. Dealing with the Stream Of Logs Flume, Kafka, Logstash, Graylog2, Fluentd
  • 10. Searching logs Graylog2, ElasticSearch
  • 13. A Logstash Cluster From the Logstash docs
  • 14. May I help you? FluentD (AL 2.0) Graylog 2 (GPL) Loggly (commerical)
  • 15. If you want to know... ...you need to log.
  • 16. How much is healthy?
  • 17. Trace is almost for free. Log4j 2 Filters help too.
  • 18. Default Levels Files? Warn only. Logstash & Co? Info. Magic bugs + advanced setup? Debug, or even trace.
  • 19. Start with verbs DEBUG adding user @grobmeier on node "baldur" DEBUG removing user @taval on node "mjölnir" DEBUG created entry for @taval on node "baldur" Idea taken from TJ Holowaychuk
  • 20. Logging is cool. Can I use it to get a health status of my systems?
  • 21. You can! But you shouldn't!
  • 22. Metrics Business Metrics Application Metrics System Metrics
  • 23. Gauges An instrument that measures a value.
  • 24. Counters A counter is a simple incrementing and decrementing integer.
  • 25. Histograms A Histogram measures the distribution of values.
  • 26. Meters A meter measures the rate at which a set of events occur.
  • 39. Cubism.js Mirror Offset 1 − + Credits: Michael Bostock
  • 41. Comparisons var cube = context.cubeſ"http://..."ƀ, primary = cube.metricſ"sumſrequestƀ"ƀ, secondary = primary.shiftſ-ɮ * ɩɫ * ɭ0 * ɭ0 * ɨ000ƀ;
  • 44. Best practices Measure everything! Counters ./. Meters Metrics are cheap, but not for free. Retention Policies
  • 45. Prevent the apocalypse! Logging shows events. Metrics shows state.
  • 46. Thank you! We use Twitter! @taval, @innoq & @grobmeier
  • 47. Credits Holzstapel CPU by mbostock Mess by Rev Stan Debt by "Images Money" (Flickr) Pay Here by Marc Falardeau Cockpit by Ronnie Rams Stream by Phil Whitehouse Magnifier by John Lodder (Flickr) Flying Saucer, Cup, and Teapot! by Mr Thinktank Ice berg by Derek Keats Gas Meters by mxmstryo (Flickr) Gauge Stock by Andrew Taylor (Flickr) Counter by Marcin Wichary (Flickr) Histogram of legos by color frequency by Jeff Boulter (Flickr) pomodoro timers by Paul Downey (Flickr) Zombie Apocalypse by pasukaru76