SlideShare a Scribd company logo
Observability, Distributed Tracing,
and Open Source
The Missing Primer
2
https://laprensasa.com/culture/art-music/mozart-festival-texas-returns-uiw/
3
4
5
• DanielKhan
daniel.khan@dynatrace.com
@dkhan
• Dir. TechnologyStrategy @Dynatrace
• Everything Open Source Monitoring &
standards& our contributionsto it
• Chairof W3C Trace Context
About me
6
Why I am doing this talk
Distributed
Tracing
Observability
W3C Trace
Context
OpenCensus
OpenTracing
OpenTelemetry
Metrics
Span
Trace
7
Application
In the Beginning there was the Monolith
Presentation
Business Logic
Data Access
Database
Services
Presentation
API Gateway
Auth Inventory CartAccount
Offers Shipping CheckoutStatus
Wire
8
Developmentin a Microservices World
Cart
Dev
Preproduction
Cart Auth InventoryAccount
Offers Shopping CheckoutStatus
Push
Cart
• Latency
• Response Time
• Error Rate
• Number of queries
KPI’s
9
Metrics
Source: https://techblog.commercetools.com/adding-consistency-and-automation-to-grafana-e99eb374fe40
… containtime correlated datapoints
• Counter
Monotonously increasing values
Think: Odometer
• Gauge
Increasing and decreasing values
Think: Tachometer
• Histogram
Groups values into buckets
Think: Knock events 0-50mph, 51-100mph, …
10
Collecting and Charting Metrics
11
Error
242
Success
1302
Cart Service
12
Complecity has movedto the NetworkLayer
Client API GW Service
Service
Service
Service
Service
ServiceCart
Which requests lead to an error in our cart service?
Trace
a42b a42b
a42b
a42b
a42b
a42b
a42b
a42b
a42b = Trace Context
13
A Trace is a Tree of Spans
Trace
Span
Span
Span
Click GW API
Spans represent a single operationand containmetadatalike the HTTP method, or a databasequery, or an error code
JDBC
Span: callDB()
Span: JDBC call
14
Trace Context Propagation
Cartsa42b a42b
Extract
Inject
In process propagation
Auto Instrumentation
• Zipkin
• Sleuth
• OpenTelemetry
• Commercial
• …
15
Trace Context Header Formats
• Proprietary
• B3 Header (Zipkin)
• W3C Trace Context
What’sthe header name and what does it contain?
16
W3C Trace Context
Service A API GW Service B
Trace
Service C
OpenTelemetry AWS Zipkin OpenTelemetry
Goal: All monitoring systems and middlewaresagree on one format for trace context propagation
Span
Span
Span
Span
17
W3C Trace Context Format
traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01 tracestate:
rojo=00f067aa0ba902b7,congo=t61rcWkgMzE
Version TraceID ParentID Flags
18
Data collection
So far we just instrumentedthe code to propagatecontext but no data has been collected
Trace
Span
Span
Span
Agent Agent Agent
Click GW API
TraceContext TraceContext
MonitoringSystem Storage
19
Data Collection & PresentationSystems
Solution Agents Instrumentation Storage Presentation
Zipkin / Sleuth + + + +
Jaeger - - + +
OpenTelemetry + + - -
Commercial + + + +
20
Zipkin
21
Jaeger
22
Commercial
23
Entity Model Based Service Flow
24
Detecting Errors
25
Solving our Cart Problem
Client API GW Service
Service
Service
Service
Service
ServiceCart
Trace
Client
Service
Currency
Cart
API GW
GET: Currency=EURO
26
What we did to Solve the Problem
1. We used metrics to learn about a problem
2. We used distributedtracing to pass along a unique ID per trace
• For that,we used auto instrumentationto extract and inject the trace ID
3. We used a monitoring system and its agentsto collected traces and we could filter transactionsthatproduced an
error
4. We looked into the metadata of such a transactionto identify how it differs from succeeding ones
27
You’ve mentionedOpenTelemetry …
+ =
In early 2019 OpenCensusand OpenTracingmerged into OpenTelemetry
Metrics, Traces, Logs
28
APIs SDKs Exporters Collector
29
30
31
32
33
OpenTelemetry – Developer usecases
• Cloud nativemicroservices architectures are hard
to trace and debug during development
• In developmentOpenTelemetry can be used to
either
• manuallycreate spans to trace certain
execution paths
• use provided auto-instrumentation tomake
a system observable
• As backend and UI, Jaeger is the most popular
tool. It’s open source and solely displaystraces
34
OpenTelemetry – in Production
• Provides just a fraction of what modern tools provide
• Traces
• Metrics
• Logs
• Topology
• Behavior
• Code level visibility
• Metadata
• Manual instrumentation codeneeds to be kept up-to-date
• A backend needs to be maintained
• No support model if instrumentation breaksproductioncode
• No enterprise features (access control, throttling, scaling, …)
35
Why do Vendors Care then?
36
OpenTelemetry Company Contribution Stats
Google
Microsoft
Dynatrace
37
38
What happens when we add support for a new framework?
• Today, our engineers reverse engineer frameworks to add
instrumentationsupport to them
• Every time an update is released, the instrumentationcode is
being tested.
• In case of issues, it goes back to the developmentteam who
needs to fix it and deploy an update.
• The whole process is automated and transparent to the customer ☺
• This is costly and time consuming
39
In-process tracing
Click GW API
MonitoringSystem
Trace
Span
Span
Span
HZQ
Span: doHZQ()()
Span: HZQ call
OTEL HZQ
Wrapper
40
“We want every platform and library to
be pre-instrumented with
OpenTelemetry and we’re committed to
making this as easy as possible.”
Sergey Kanzhelev (Google)
41
What is Observability and how does it differ fromMonitoring?
1. In control theory, observability is a measure of how well internal states of a system can be inferred from
knowledgeof its external outputs.
Source: Wikipedia
2. In software development, observabilityisachieved by adding code (instrumentation)that emits telemetry
data.
3. Monitoringis the act of displayingand analyzing this telemetry data.
4. Monitoringalone can tell you that there is a problem.
E.g. ”We see that some users experience a 50% higher response time on check-out”
5. Observablityhelps finding the root cause (the why) by providingdatathat can be correlatedand analysed
freely even if this problem is completelynew to you (unknown unknowns)
E.g. “The response time of the checkout increases exponentially with the number of items in the basket,
because of a misplaced for loop that executes the same database query times the number of items for every
item in the basket”
42
Putting it all Together
• Metrics can help you to learn that there is a problem
• Distributedtracing becomes increasingly importantto understandmulti-tier execution paths and root causes
of problems
• Developersnow rely on metrics and traces to understandhow their service functionsin their microservice
architectures
• Pure Open Source solutionsare viable for pre-prod environments
• Standardization isthe only way to tackle today’scomplexity and Open Source is the key driver
• Vendorsare prepared to tap intodata collected by Open Source standard toolsto add enterprise features on
top to support web-scale workloads
43
dynatrace.com
@dkhan
daniel.khan@dynatrace.com
Thank you!

More Related Content

What's hot (20)

OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdfOSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
NETWAYS
 
Everything You wanted to Know About Distributed Tracing
Everything You wanted to Know About Distributed TracingEverything You wanted to Know About Distributed Tracing
Everything You wanted to Know About Distributed Tracing
Amuhinda Hungai
 
Introduction to Distributed Tracing
Introduction to Distributed TracingIntroduction to Distributed Tracing
Introduction to Distributed Tracing
petabridge
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)
Lucas Jellema
 
Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...
LibbySchulze
 
Prometheus and Grafana
Prometheus and GrafanaPrometheus and Grafana
Prometheus and Grafana
Lhouceine OUHAMZA
 
Opentelemetry - From frontend to backend
Opentelemetry - From frontend to backendOpentelemetry - From frontend to backend
Opentelemetry - From frontend to backend
Sebastian Poxhofer
 
Combining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observabilityCombining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observability
Elasticsearch
 
Improve monitoring and observability for kubernetes with oss tools
Improve monitoring and observability for kubernetes with oss toolsImprove monitoring and observability for kubernetes with oss tools
Improve monitoring and observability for kubernetes with oss tools
Nilesh Gule
 
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Brian Brazil
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
Grafana Labs
 
Observability in Java: Getting Started with OpenTelemetry
Observability in Java: Getting Started with OpenTelemetryObservability in Java: Getting Started with OpenTelemetry
Observability in Java: Getting Started with OpenTelemetry
DevOps.com
 
Monitoring with prometheus
Monitoring with prometheusMonitoring with prometheus
Monitoring with prometheus
Kasper Nissen
 
Getting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and GrafanaGetting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and Grafana
Syah Dwi Prihatmoko
 
Understand your system like never before with OpenTelemetry, Grafana, and Pro...
Understand your system like never before with OpenTelemetry, Grafana, and Pro...Understand your system like never before with OpenTelemetry, Grafana, and Pro...
Understand your system like never before with OpenTelemetry, Grafana, and Pro...
LibbySchulze
 
Cloud-Native Observability
Cloud-Native ObservabilityCloud-Native Observability
Cloud-Native Observability
Tyler Treat
 
Observability
Observability Observability
Observability
Enes Altınok
 
Observability
ObservabilityObservability
Observability
Maganathin Veeraragaloo
 
Grafana overview deck - Tech - 2023 May v1.pdf
Grafana overview deck  - Tech - 2023 May v1.pdfGrafana overview deck  - Tech - 2023 May v1.pdf
Grafana overview deck - Tech - 2023 May v1.pdf
BillySin5
 
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)
Brian Brazil
 
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdfOSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
NETWAYS
 
Everything You wanted to Know About Distributed Tracing
Everything You wanted to Know About Distributed TracingEverything You wanted to Know About Distributed Tracing
Everything You wanted to Know About Distributed Tracing
Amuhinda Hungai
 
Introduction to Distributed Tracing
Introduction to Distributed TracingIntroduction to Distributed Tracing
Introduction to Distributed Tracing
petabridge
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)
Lucas Jellema
 
Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...
LibbySchulze
 
Opentelemetry - From frontend to backend
Opentelemetry - From frontend to backendOpentelemetry - From frontend to backend
Opentelemetry - From frontend to backend
Sebastian Poxhofer
 
Combining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observabilityCombining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observability
Elasticsearch
 
Improve monitoring and observability for kubernetes with oss tools
Improve monitoring and observability for kubernetes with oss toolsImprove monitoring and observability for kubernetes with oss tools
Improve monitoring and observability for kubernetes with oss tools
Nilesh Gule
 
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Brian Brazil
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
Grafana Labs
 
Observability in Java: Getting Started with OpenTelemetry
Observability in Java: Getting Started with OpenTelemetryObservability in Java: Getting Started with OpenTelemetry
Observability in Java: Getting Started with OpenTelemetry
DevOps.com
 
Monitoring with prometheus
Monitoring with prometheusMonitoring with prometheus
Monitoring with prometheus
Kasper Nissen
 
Getting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and GrafanaGetting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and Grafana
Syah Dwi Prihatmoko
 
Understand your system like never before with OpenTelemetry, Grafana, and Pro...
Understand your system like never before with OpenTelemetry, Grafana, and Pro...Understand your system like never before with OpenTelemetry, Grafana, and Pro...
Understand your system like never before with OpenTelemetry, Grafana, and Pro...
LibbySchulze
 
Cloud-Native Observability
Cloud-Native ObservabilityCloud-Native Observability
Cloud-Native Observability
Tyler Treat
 
Grafana overview deck - Tech - 2023 May v1.pdf
Grafana overview deck  - Tech - 2023 May v1.pdfGrafana overview deck  - Tech - 2023 May v1.pdf
Grafana overview deck - Tech - 2023 May v1.pdf
BillySin5
 
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)
Brian Brazil
 

Similar to Observability, Distributed Tracing, and Open Source: The Missing Primer (20)

"Distributed Tracing: New DevOps Foundation" by Jayesh Ahire
"Distributed Tracing: New DevOps Foundation" by Jayesh Ahire  "Distributed Tracing: New DevOps Foundation" by Jayesh Ahire
"Distributed Tracing: New DevOps Foundation" by Jayesh Ahire
CodeOps Technologies LLP
 
Distributed Tracing: New DevOps Foundation
Distributed Tracing: New DevOps FoundationDistributed Tracing: New DevOps Foundation
Distributed Tracing: New DevOps Foundation
CodeOps Technologies LLP
 
Observability for Application Developers (1)-1.pptx
Observability for Application Developers (1)-1.pptxObservability for Application Developers (1)-1.pptx
Observability for Application Developers (1)-1.pptx
OpsTree solutions
 
Manage Microservices Chaos and Complexity with Observability
Manage Microservices Chaos and Complexity with ObservabilityManage Microservices Chaos and Complexity with Observability
Manage Microservices Chaos and Complexity with Observability
NGINX, Inc.
 
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
AgileNetwork
 
DockerCon SF 2019 - Observability Workshop
DockerCon SF 2019 - Observability WorkshopDockerCon SF 2019 - Observability Workshop
DockerCon SF 2019 - Observability Workshop
Kevin Crawley
 
What Is OpenTelemetry? A Complete Introduction
What Is OpenTelemetry? A Complete IntroductionWhat Is OpenTelemetry? A Complete Introduction
What Is OpenTelemetry? A Complete Introduction
Ciente
 
stackconf 2023 | Practical introduction to OpenTelemetry tracing by Nicolas F...
stackconf 2023 | Practical introduction to OpenTelemetry tracing by Nicolas F...stackconf 2023 | Practical introduction to OpenTelemetry tracing by Nicolas F...
stackconf 2023 | Practical introduction to OpenTelemetry tracing by Nicolas F...
NETWAYS
 
Beginner's Guide to Observability@Devoxx PL 2024
Beginner's  Guide to Observability@Devoxx PL 2024Beginner's  Guide to Observability@Devoxx PL 2024
Beginner's Guide to Observability@Devoxx PL 2024
michniczscribd
 
stackconf 2022: Open Source for Better Observability
stackconf 2022: Open Source for Better Observabilitystackconf 2022: Open Source for Better Observability
stackconf 2022: Open Source for Better Observability
NETWAYS
 
Meetup OpenTelemetry Intro
Meetup OpenTelemetry IntroMeetup OpenTelemetry Intro
Meetup OpenTelemetry Intro
DimitrisFinas1
 
ThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptx
Grace Jansen
 
Introduction to Open Telemetry as Observability Library
Introduction to Open  Telemetry as Observability LibraryIntroduction to Open  Telemetry as Observability Library
Introduction to Open Telemetry as Observability Library
Tonny Adhi Sabastian
 
Adopting Open Telemetry as Distributed Tracer on your Microservices at Kubern...
Adopting Open Telemetry as Distributed Tracer on your Microservices at Kubern...Adopting Open Telemetry as Distributed Tracer on your Microservices at Kubern...
Adopting Open Telemetry as Distributed Tracer on your Microservices at Kubern...
Tonny Adhi Sabastian
 
"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...
"Introducing Distributed Tracing in a Large Software System",  Kostiantyn Sha..."Introducing Distributed Tracing in a Large Software System",  Kostiantyn Sha...
"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...
Fwdays
 
WJAX 2019 - Taking Distributed Tracing to the next level
WJAX 2019 - Taking Distributed Tracing to the next levelWJAX 2019 - Taking Distributed Tracing to the next level
WJAX 2019 - Taking Distributed Tracing to the next level
Frank Pfleger
 
2307 - DevBCN - Otel 101_compressed.pdf
2307 - DevBCN - Otel 101_compressed.pdf2307 - DevBCN - Otel 101_compressed.pdf
2307 - DevBCN - Otel 101_compressed.pdf
DimitrisFinas1
 
ADDO Open Source Observability Tools
ADDO Open Source Observability Tools ADDO Open Source Observability Tools
ADDO Open Source Observability Tools
Mickey Boxell
 
Java il spanning services 2019
Java il   spanning services 2019Java il   spanning services 2019
Java il spanning services 2019
Yair Galler
 
Migrating from OpenTracing to OpenTelemetry - Kubernetes Community Days Munic...
Migrating from OpenTracing to OpenTelemetry - Kubernetes Community Days Munic...Migrating from OpenTracing to OpenTelemetry - Kubernetes Community Days Munic...
Migrating from OpenTracing to OpenTelemetry - Kubernetes Community Days Munic...
SonjaChevre
 
"Distributed Tracing: New DevOps Foundation" by Jayesh Ahire
"Distributed Tracing: New DevOps Foundation" by Jayesh Ahire  "Distributed Tracing: New DevOps Foundation" by Jayesh Ahire
"Distributed Tracing: New DevOps Foundation" by Jayesh Ahire
CodeOps Technologies LLP
 
Distributed Tracing: New DevOps Foundation
Distributed Tracing: New DevOps FoundationDistributed Tracing: New DevOps Foundation
Distributed Tracing: New DevOps Foundation
CodeOps Technologies LLP
 
Observability for Application Developers (1)-1.pptx
Observability for Application Developers (1)-1.pptxObservability for Application Developers (1)-1.pptx
Observability for Application Developers (1)-1.pptx
OpsTree solutions
 
Manage Microservices Chaos and Complexity with Observability
Manage Microservices Chaos and Complexity with ObservabilityManage Microservices Chaos and Complexity with Observability
Manage Microservices Chaos and Complexity with Observability
NGINX, Inc.
 
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
AgileNetwork
 
DockerCon SF 2019 - Observability Workshop
DockerCon SF 2019 - Observability WorkshopDockerCon SF 2019 - Observability Workshop
DockerCon SF 2019 - Observability Workshop
Kevin Crawley
 
What Is OpenTelemetry? A Complete Introduction
What Is OpenTelemetry? A Complete IntroductionWhat Is OpenTelemetry? A Complete Introduction
What Is OpenTelemetry? A Complete Introduction
Ciente
 
stackconf 2023 | Practical introduction to OpenTelemetry tracing by Nicolas F...
stackconf 2023 | Practical introduction to OpenTelemetry tracing by Nicolas F...stackconf 2023 | Practical introduction to OpenTelemetry tracing by Nicolas F...
stackconf 2023 | Practical introduction to OpenTelemetry tracing by Nicolas F...
NETWAYS
 
Beginner's Guide to Observability@Devoxx PL 2024
Beginner's  Guide to Observability@Devoxx PL 2024Beginner's  Guide to Observability@Devoxx PL 2024
Beginner's Guide to Observability@Devoxx PL 2024
michniczscribd
 
stackconf 2022: Open Source for Better Observability
stackconf 2022: Open Source for Better Observabilitystackconf 2022: Open Source for Better Observability
stackconf 2022: Open Source for Better Observability
NETWAYS
 
Meetup OpenTelemetry Intro
Meetup OpenTelemetry IntroMeetup OpenTelemetry Intro
Meetup OpenTelemetry Intro
DimitrisFinas1
 
ThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptx
Grace Jansen
 
Introduction to Open Telemetry as Observability Library
Introduction to Open  Telemetry as Observability LibraryIntroduction to Open  Telemetry as Observability Library
Introduction to Open Telemetry as Observability Library
Tonny Adhi Sabastian
 
Adopting Open Telemetry as Distributed Tracer on your Microservices at Kubern...
Adopting Open Telemetry as Distributed Tracer on your Microservices at Kubern...Adopting Open Telemetry as Distributed Tracer on your Microservices at Kubern...
Adopting Open Telemetry as Distributed Tracer on your Microservices at Kubern...
Tonny Adhi Sabastian
 
"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...
"Introducing Distributed Tracing in a Large Software System",  Kostiantyn Sha..."Introducing Distributed Tracing in a Large Software System",  Kostiantyn Sha...
"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...
Fwdays
 
WJAX 2019 - Taking Distributed Tracing to the next level
WJAX 2019 - Taking Distributed Tracing to the next levelWJAX 2019 - Taking Distributed Tracing to the next level
WJAX 2019 - Taking Distributed Tracing to the next level
Frank Pfleger
 
2307 - DevBCN - Otel 101_compressed.pdf
2307 - DevBCN - Otel 101_compressed.pdf2307 - DevBCN - Otel 101_compressed.pdf
2307 - DevBCN - Otel 101_compressed.pdf
DimitrisFinas1
 
ADDO Open Source Observability Tools
ADDO Open Source Observability Tools ADDO Open Source Observability Tools
ADDO Open Source Observability Tools
Mickey Boxell
 
Java il spanning services 2019
Java il   spanning services 2019Java il   spanning services 2019
Java il spanning services 2019
Yair Galler
 
Migrating from OpenTracing to OpenTelemetry - Kubernetes Community Days Munic...
Migrating from OpenTracing to OpenTelemetry - Kubernetes Community Days Munic...Migrating from OpenTracing to OpenTelemetry - Kubernetes Community Days Munic...
Migrating from OpenTracing to OpenTelemetry - Kubernetes Community Days Munic...
SonjaChevre
 
Ad

More from VMware Tanzu (20)

Spring into AI presented by Dan Vega 5/14
Spring into AI presented by Dan Vega 5/14Spring into AI presented by Dan Vega 5/14
Spring into AI presented by Dan Vega 5/14
VMware Tanzu
 
What AI Means For Your Product Strategy And What To Do About It
What AI Means For Your Product Strategy And What To Do About ItWhat AI Means For Your Product Strategy And What To Do About It
What AI Means For Your Product Strategy And What To Do About It
VMware Tanzu
 
Make the Right Thing the Obvious Thing at Cardinal Health 2023
Make the Right Thing the Obvious Thing at Cardinal Health 2023Make the Right Thing the Obvious Thing at Cardinal Health 2023
Make the Right Thing the Obvious Thing at Cardinal Health 2023
VMware Tanzu
 
Enhancing DevEx and Simplifying Operations at Scale
Enhancing DevEx and Simplifying Operations at ScaleEnhancing DevEx and Simplifying Operations at Scale
Enhancing DevEx and Simplifying Operations at Scale
VMware Tanzu
 
Spring Update | July 2023
Spring Update | July 2023Spring Update | July 2023
Spring Update | July 2023
VMware Tanzu
 
Platforms, Platform Engineering, & Platform as a Product
Platforms, Platform Engineering, & Platform as a ProductPlatforms, Platform Engineering, & Platform as a Product
Platforms, Platform Engineering, & Platform as a Product
VMware Tanzu
 
Building Cloud Ready Apps
Building Cloud Ready AppsBuilding Cloud Ready Apps
Building Cloud Ready Apps
VMware Tanzu
 
Spring Boot 3 And Beyond
Spring Boot 3 And BeyondSpring Boot 3 And Beyond
Spring Boot 3 And Beyond
VMware Tanzu
 
Spring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdf
Spring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdfSpring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdf
Spring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdf
VMware Tanzu
 
Simplify and Scale Enterprise Apps in the Cloud | Boston 2023
Simplify and Scale Enterprise Apps in the Cloud | Boston 2023Simplify and Scale Enterprise Apps in the Cloud | Boston 2023
Simplify and Scale Enterprise Apps in the Cloud | Boston 2023
VMware Tanzu
 
Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023
Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023
Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023
VMware Tanzu
 
tanzu_developer_connect.pptx
tanzu_developer_connect.pptxtanzu_developer_connect.pptx
tanzu_developer_connect.pptx
VMware Tanzu
 
Tanzu Virtual Developer Connect Workshop - French
Tanzu Virtual Developer Connect Workshop - FrenchTanzu Virtual Developer Connect Workshop - French
Tanzu Virtual Developer Connect Workshop - French
VMware Tanzu
 
Tanzu Developer Connect Workshop - English
Tanzu Developer Connect Workshop - EnglishTanzu Developer Connect Workshop - English
Tanzu Developer Connect Workshop - English
VMware Tanzu
 
Virtual Developer Connect Workshop - English
Virtual Developer Connect Workshop - EnglishVirtual Developer Connect Workshop - English
Virtual Developer Connect Workshop - English
VMware Tanzu
 
Tanzu Developer Connect - French
Tanzu Developer Connect - FrenchTanzu Developer Connect - French
Tanzu Developer Connect - French
VMware Tanzu
 
Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023
Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023
Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023
VMware Tanzu
 
SpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring Boot
SpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring BootSpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring Boot
SpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring Boot
VMware Tanzu
 
SpringOne Tour: The Influential Software Engineer
SpringOne Tour: The Influential Software EngineerSpringOne Tour: The Influential Software Engineer
SpringOne Tour: The Influential Software Engineer
VMware Tanzu
 
SpringOne Tour: Domain-Driven Design: Theory vs Practice
SpringOne Tour: Domain-Driven Design: Theory vs PracticeSpringOne Tour: Domain-Driven Design: Theory vs Practice
SpringOne Tour: Domain-Driven Design: Theory vs Practice
VMware Tanzu
 
Spring into AI presented by Dan Vega 5/14
Spring into AI presented by Dan Vega 5/14Spring into AI presented by Dan Vega 5/14
Spring into AI presented by Dan Vega 5/14
VMware Tanzu
 
What AI Means For Your Product Strategy And What To Do About It
What AI Means For Your Product Strategy And What To Do About ItWhat AI Means For Your Product Strategy And What To Do About It
What AI Means For Your Product Strategy And What To Do About It
VMware Tanzu
 
Make the Right Thing the Obvious Thing at Cardinal Health 2023
Make the Right Thing the Obvious Thing at Cardinal Health 2023Make the Right Thing the Obvious Thing at Cardinal Health 2023
Make the Right Thing the Obvious Thing at Cardinal Health 2023
VMware Tanzu
 
Enhancing DevEx and Simplifying Operations at Scale
Enhancing DevEx and Simplifying Operations at ScaleEnhancing DevEx and Simplifying Operations at Scale
Enhancing DevEx and Simplifying Operations at Scale
VMware Tanzu
 
Spring Update | July 2023
Spring Update | July 2023Spring Update | July 2023
Spring Update | July 2023
VMware Tanzu
 
Platforms, Platform Engineering, & Platform as a Product
Platforms, Platform Engineering, & Platform as a ProductPlatforms, Platform Engineering, & Platform as a Product
Platforms, Platform Engineering, & Platform as a Product
VMware Tanzu
 
Building Cloud Ready Apps
Building Cloud Ready AppsBuilding Cloud Ready Apps
Building Cloud Ready Apps
VMware Tanzu
 
Spring Boot 3 And Beyond
Spring Boot 3 And BeyondSpring Boot 3 And Beyond
Spring Boot 3 And Beyond
VMware Tanzu
 
Spring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdf
Spring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdfSpring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdf
Spring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdf
VMware Tanzu
 
Simplify and Scale Enterprise Apps in the Cloud | Boston 2023
Simplify and Scale Enterprise Apps in the Cloud | Boston 2023Simplify and Scale Enterprise Apps in the Cloud | Boston 2023
Simplify and Scale Enterprise Apps in the Cloud | Boston 2023
VMware Tanzu
 
Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023
Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023
Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023
VMware Tanzu
 
tanzu_developer_connect.pptx
tanzu_developer_connect.pptxtanzu_developer_connect.pptx
tanzu_developer_connect.pptx
VMware Tanzu
 
Tanzu Virtual Developer Connect Workshop - French
Tanzu Virtual Developer Connect Workshop - FrenchTanzu Virtual Developer Connect Workshop - French
Tanzu Virtual Developer Connect Workshop - French
VMware Tanzu
 
Tanzu Developer Connect Workshop - English
Tanzu Developer Connect Workshop - EnglishTanzu Developer Connect Workshop - English
Tanzu Developer Connect Workshop - English
VMware Tanzu
 
Virtual Developer Connect Workshop - English
Virtual Developer Connect Workshop - EnglishVirtual Developer Connect Workshop - English
Virtual Developer Connect Workshop - English
VMware Tanzu
 
Tanzu Developer Connect - French
Tanzu Developer Connect - FrenchTanzu Developer Connect - French
Tanzu Developer Connect - French
VMware Tanzu
 
Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023
Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023
Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023
VMware Tanzu
 
SpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring Boot
SpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring BootSpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring Boot
SpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring Boot
VMware Tanzu
 
SpringOne Tour: The Influential Software Engineer
SpringOne Tour: The Influential Software EngineerSpringOne Tour: The Influential Software Engineer
SpringOne Tour: The Influential Software Engineer
VMware Tanzu
 
SpringOne Tour: Domain-Driven Design: Theory vs Practice
SpringOne Tour: Domain-Driven Design: Theory vs PracticeSpringOne Tour: Domain-Driven Design: Theory vs Practice
SpringOne Tour: Domain-Driven Design: Theory vs Practice
VMware Tanzu
 
Ad

Recently uploaded (20)

Software Engineering Process, Notation & Tools Introduction - Part 3
Software Engineering Process, Notation & Tools Introduction - Part 3Software Engineering Process, Notation & Tools Introduction - Part 3
Software Engineering Process, Notation & Tools Introduction - Part 3
Gaurav Sharma
 
Design by Contract - Building Robust Software with Contract-First Development
Design by Contract - Building Robust Software with Contract-First DevelopmentDesign by Contract - Building Robust Software with Contract-First Development
Design by Contract - Building Robust Software with Contract-First Development
Par-Tec S.p.A.
 
Topic 26 Security Testing Considerations.pptx
Topic 26 Security Testing Considerations.pptxTopic 26 Security Testing Considerations.pptx
Topic 26 Security Testing Considerations.pptx
marutnand8
 
Revolutionize Your Insurance Workflow with Claims Management Software
Revolutionize Your Insurance Workflow with Claims Management SoftwareRevolutionize Your Insurance Workflow with Claims Management Software
Revolutionize Your Insurance Workflow with Claims Management Software
Insurance Tech Services
 
Online Queue Management System for Public Service Offices [Focused on Municip...
Online Queue Management System for Public Service Offices [Focused on Municip...Online Queue Management System for Public Service Offices [Focused on Municip...
Online Queue Management System for Public Service Offices [Focused on Municip...
Rishab Acharya
 
Key AI Technologies Used by Indian Artificial Intelligence Companies
Key AI Technologies Used by Indian Artificial Intelligence CompaniesKey AI Technologies Used by Indian Artificial Intelligence Companies
Key AI Technologies Used by Indian Artificial Intelligence Companies
Mypcot Infotech
 
Scalefusion Remote Access for Apple Devices
Scalefusion Remote Access for Apple DevicesScalefusion Remote Access for Apple Devices
Scalefusion Remote Access for Apple Devices
Scalefusion
 
Integration Ignited Redefining Event-Driven Architecture at Wix - EventCentric
Integration Ignited Redefining Event-Driven Architecture at Wix - EventCentricIntegration Ignited Redefining Event-Driven Architecture at Wix - EventCentric
Integration Ignited Redefining Event-Driven Architecture at Wix - EventCentric
Natan Silnitsky
 
Boost Student Engagement with Smart Attendance Software for Schools
Boost Student Engagement with Smart Attendance Software for SchoolsBoost Student Engagement with Smart Attendance Software for Schools
Boost Student Engagement with Smart Attendance Software for Schools
Visitu
 
Maintaining + Optimizing Database Health: Vendors, Orchestrations, Enrichment...
Maintaining + Optimizing Database Health: Vendors, Orchestrations, Enrichment...Maintaining + Optimizing Database Health: Vendors, Orchestrations, Enrichment...
Maintaining + Optimizing Database Health: Vendors, Orchestrations, Enrichment...
BradBedford3
 
How Insurance Policy Administration Streamlines Policy Lifecycle for Agile Op...
How Insurance Policy Administration Streamlines Policy Lifecycle for Agile Op...How Insurance Policy Administration Streamlines Policy Lifecycle for Agile Op...
How Insurance Policy Administration Streamlines Policy Lifecycle for Agile Op...
Insurance Tech Services
 
Bonk coin airdrop_ Everything You Need to Know.pdf
Bonk coin airdrop_ Everything You Need to Know.pdfBonk coin airdrop_ Everything You Need to Know.pdf
Bonk coin airdrop_ Everything You Need to Know.pdf
Herond Labs
 
Providing Better Biodiversity Through Better Data
Providing Better Biodiversity Through Better DataProviding Better Biodiversity Through Better Data
Providing Better Biodiversity Through Better Data
Safe Software
 
zOS CommServer support for the Network Express feature on z17
zOS CommServer support for the Network Express feature on z17zOS CommServer support for the Network Express feature on z17
zOS CommServer support for the Network Express feature on z17
zOSCommserver
 
Content Mate Web App Triples Content Managers‘ Productivity
Content Mate Web App Triples Content Managers‘ ProductivityContent Mate Web App Triples Content Managers‘ Productivity
Content Mate Web App Triples Content Managers‘ Productivity
Alex Vladimirovich
 
Build Smarter, Deliver Faster with Choreo - An AI Native Internal Developer P...
Build Smarter, Deliver Faster with Choreo - An AI Native Internal Developer P...Build Smarter, Deliver Faster with Choreo - An AI Native Internal Developer P...
Build Smarter, Deliver Faster with Choreo - An AI Native Internal Developer P...
WSO2
 
Leveraging Foundation Models to Infer Intents
Leveraging Foundation Models to Infer IntentsLeveraging Foundation Models to Infer Intents
Leveraging Foundation Models to Infer Intents
Keheliya Gallaba
 
How to Generate Financial Statements in QuickBooks Like a Pro (1).pdf
How to Generate Financial Statements in QuickBooks Like a Pro (1).pdfHow to Generate Financial Statements in QuickBooks Like a Pro (1).pdf
How to Generate Financial Statements in QuickBooks Like a Pro (1).pdf
QuickBooks Training
 
Why Indonesia’s $12.63B Alt-Lending Boom Needs Loan Servicing Automation & Re...
Why Indonesia’s $12.63B Alt-Lending Boom Needs Loan Servicing Automation & Re...Why Indonesia’s $12.63B Alt-Lending Boom Needs Loan Servicing Automation & Re...
Why Indonesia’s $12.63B Alt-Lending Boom Needs Loan Servicing Automation & Re...
Prachi Desai
 
How to purchase, license and subscribe to Microsoft Azure_PDF.pdf
How to purchase, license and subscribe to Microsoft Azure_PDF.pdfHow to purchase, license and subscribe to Microsoft Azure_PDF.pdf
How to purchase, license and subscribe to Microsoft Azure_PDF.pdf
victordsane
 
Software Engineering Process, Notation & Tools Introduction - Part 3
Software Engineering Process, Notation & Tools Introduction - Part 3Software Engineering Process, Notation & Tools Introduction - Part 3
Software Engineering Process, Notation & Tools Introduction - Part 3
Gaurav Sharma
 
Design by Contract - Building Robust Software with Contract-First Development
Design by Contract - Building Robust Software with Contract-First DevelopmentDesign by Contract - Building Robust Software with Contract-First Development
Design by Contract - Building Robust Software with Contract-First Development
Par-Tec S.p.A.
 
Topic 26 Security Testing Considerations.pptx
Topic 26 Security Testing Considerations.pptxTopic 26 Security Testing Considerations.pptx
Topic 26 Security Testing Considerations.pptx
marutnand8
 
Revolutionize Your Insurance Workflow with Claims Management Software
Revolutionize Your Insurance Workflow with Claims Management SoftwareRevolutionize Your Insurance Workflow with Claims Management Software
Revolutionize Your Insurance Workflow with Claims Management Software
Insurance Tech Services
 
Online Queue Management System for Public Service Offices [Focused on Municip...
Online Queue Management System for Public Service Offices [Focused on Municip...Online Queue Management System for Public Service Offices [Focused on Municip...
Online Queue Management System for Public Service Offices [Focused on Municip...
Rishab Acharya
 
Key AI Technologies Used by Indian Artificial Intelligence Companies
Key AI Technologies Used by Indian Artificial Intelligence CompaniesKey AI Technologies Used by Indian Artificial Intelligence Companies
Key AI Technologies Used by Indian Artificial Intelligence Companies
Mypcot Infotech
 
Scalefusion Remote Access for Apple Devices
Scalefusion Remote Access for Apple DevicesScalefusion Remote Access for Apple Devices
Scalefusion Remote Access for Apple Devices
Scalefusion
 
Integration Ignited Redefining Event-Driven Architecture at Wix - EventCentric
Integration Ignited Redefining Event-Driven Architecture at Wix - EventCentricIntegration Ignited Redefining Event-Driven Architecture at Wix - EventCentric
Integration Ignited Redefining Event-Driven Architecture at Wix - EventCentric
Natan Silnitsky
 
Boost Student Engagement with Smart Attendance Software for Schools
Boost Student Engagement with Smart Attendance Software for SchoolsBoost Student Engagement with Smart Attendance Software for Schools
Boost Student Engagement with Smart Attendance Software for Schools
Visitu
 
Maintaining + Optimizing Database Health: Vendors, Orchestrations, Enrichment...
Maintaining + Optimizing Database Health: Vendors, Orchestrations, Enrichment...Maintaining + Optimizing Database Health: Vendors, Orchestrations, Enrichment...
Maintaining + Optimizing Database Health: Vendors, Orchestrations, Enrichment...
BradBedford3
 
How Insurance Policy Administration Streamlines Policy Lifecycle for Agile Op...
How Insurance Policy Administration Streamlines Policy Lifecycle for Agile Op...How Insurance Policy Administration Streamlines Policy Lifecycle for Agile Op...
How Insurance Policy Administration Streamlines Policy Lifecycle for Agile Op...
Insurance Tech Services
 
Bonk coin airdrop_ Everything You Need to Know.pdf
Bonk coin airdrop_ Everything You Need to Know.pdfBonk coin airdrop_ Everything You Need to Know.pdf
Bonk coin airdrop_ Everything You Need to Know.pdf
Herond Labs
 
Providing Better Biodiversity Through Better Data
Providing Better Biodiversity Through Better DataProviding Better Biodiversity Through Better Data
Providing Better Biodiversity Through Better Data
Safe Software
 
zOS CommServer support for the Network Express feature on z17
zOS CommServer support for the Network Express feature on z17zOS CommServer support for the Network Express feature on z17
zOS CommServer support for the Network Express feature on z17
zOSCommserver
 
Content Mate Web App Triples Content Managers‘ Productivity
Content Mate Web App Triples Content Managers‘ ProductivityContent Mate Web App Triples Content Managers‘ Productivity
Content Mate Web App Triples Content Managers‘ Productivity
Alex Vladimirovich
 
Build Smarter, Deliver Faster with Choreo - An AI Native Internal Developer P...
Build Smarter, Deliver Faster with Choreo - An AI Native Internal Developer P...Build Smarter, Deliver Faster with Choreo - An AI Native Internal Developer P...
Build Smarter, Deliver Faster with Choreo - An AI Native Internal Developer P...
WSO2
 
Leveraging Foundation Models to Infer Intents
Leveraging Foundation Models to Infer IntentsLeveraging Foundation Models to Infer Intents
Leveraging Foundation Models to Infer Intents
Keheliya Gallaba
 
How to Generate Financial Statements in QuickBooks Like a Pro (1).pdf
How to Generate Financial Statements in QuickBooks Like a Pro (1).pdfHow to Generate Financial Statements in QuickBooks Like a Pro (1).pdf
How to Generate Financial Statements in QuickBooks Like a Pro (1).pdf
QuickBooks Training
 
Why Indonesia’s $12.63B Alt-Lending Boom Needs Loan Servicing Automation & Re...
Why Indonesia’s $12.63B Alt-Lending Boom Needs Loan Servicing Automation & Re...Why Indonesia’s $12.63B Alt-Lending Boom Needs Loan Servicing Automation & Re...
Why Indonesia’s $12.63B Alt-Lending Boom Needs Loan Servicing Automation & Re...
Prachi Desai
 
How to purchase, license and subscribe to Microsoft Azure_PDF.pdf
How to purchase, license and subscribe to Microsoft Azure_PDF.pdfHow to purchase, license and subscribe to Microsoft Azure_PDF.pdf
How to purchase, license and subscribe to Microsoft Azure_PDF.pdf
victordsane
 

Observability, Distributed Tracing, and Open Source: The Missing Primer

  • 1. Observability, Distributed Tracing, and Open Source The Missing Primer
  • 3. 3
  • 4. 4
  • 5. 5 • DanielKhan [email protected] @dkhan • Dir. TechnologyStrategy @Dynatrace • Everything Open Source Monitoring & standards& our contributionsto it • Chairof W3C Trace Context About me
  • 6. 6 Why I am doing this talk Distributed Tracing Observability W3C Trace Context OpenCensus OpenTracing OpenTelemetry Metrics Span Trace
  • 7. 7 Application In the Beginning there was the Monolith Presentation Business Logic Data Access Database Services Presentation API Gateway Auth Inventory CartAccount Offers Shipping CheckoutStatus Wire
  • 8. 8 Developmentin a Microservices World Cart Dev Preproduction Cart Auth InventoryAccount Offers Shopping CheckoutStatus Push Cart • Latency • Response Time • Error Rate • Number of queries KPI’s
  • 9. 9 Metrics Source: https://techblog.commercetools.com/adding-consistency-and-automation-to-grafana-e99eb374fe40 … containtime correlated datapoints • Counter Monotonously increasing values Think: Odometer • Gauge Increasing and decreasing values Think: Tachometer • Histogram Groups values into buckets Think: Knock events 0-50mph, 51-100mph, …
  • 12. 12 Complecity has movedto the NetworkLayer Client API GW Service Service Service Service Service ServiceCart Which requests lead to an error in our cart service? Trace a42b a42b a42b a42b a42b a42b a42b a42b a42b = Trace Context
  • 13. 13 A Trace is a Tree of Spans Trace Span Span Span Click GW API Spans represent a single operationand containmetadatalike the HTTP method, or a databasequery, or an error code JDBC Span: callDB() Span: JDBC call
  • 14. 14 Trace Context Propagation Cartsa42b a42b Extract Inject In process propagation Auto Instrumentation • Zipkin • Sleuth • OpenTelemetry • Commercial • …
  • 15. 15 Trace Context Header Formats • Proprietary • B3 Header (Zipkin) • W3C Trace Context What’sthe header name and what does it contain?
  • 16. 16 W3C Trace Context Service A API GW Service B Trace Service C OpenTelemetry AWS Zipkin OpenTelemetry Goal: All monitoring systems and middlewaresagree on one format for trace context propagation Span Span Span Span
  • 17. 17 W3C Trace Context Format traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01 tracestate: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE Version TraceID ParentID Flags
  • 18. 18 Data collection So far we just instrumentedthe code to propagatecontext but no data has been collected Trace Span Span Span Agent Agent Agent Click GW API TraceContext TraceContext MonitoringSystem Storage
  • 19. 19 Data Collection & PresentationSystems Solution Agents Instrumentation Storage Presentation Zipkin / Sleuth + + + + Jaeger - - + + OpenTelemetry + + - - Commercial + + + +
  • 23. 23 Entity Model Based Service Flow
  • 25. 25 Solving our Cart Problem Client API GW Service Service Service Service Service ServiceCart Trace Client Service Currency Cart API GW GET: Currency=EURO
  • 26. 26 What we did to Solve the Problem 1. We used metrics to learn about a problem 2. We used distributedtracing to pass along a unique ID per trace • For that,we used auto instrumentationto extract and inject the trace ID 3. We used a monitoring system and its agentsto collected traces and we could filter transactionsthatproduced an error 4. We looked into the metadata of such a transactionto identify how it differs from succeeding ones
  • 27. 27 You’ve mentionedOpenTelemetry … + = In early 2019 OpenCensusand OpenTracingmerged into OpenTelemetry Metrics, Traces, Logs
  • 29. 29
  • 30. 30
  • 31. 31
  • 32. 32
  • 33. 33 OpenTelemetry – Developer usecases • Cloud nativemicroservices architectures are hard to trace and debug during development • In developmentOpenTelemetry can be used to either • manuallycreate spans to trace certain execution paths • use provided auto-instrumentation tomake a system observable • As backend and UI, Jaeger is the most popular tool. It’s open source and solely displaystraces
  • 34. 34 OpenTelemetry – in Production • Provides just a fraction of what modern tools provide • Traces • Metrics • Logs • Topology • Behavior • Code level visibility • Metadata • Manual instrumentation codeneeds to be kept up-to-date • A backend needs to be maintained • No support model if instrumentation breaksproductioncode • No enterprise features (access control, throttling, scaling, …)
  • 35. 35 Why do Vendors Care then?
  • 36. 36 OpenTelemetry Company Contribution Stats Google Microsoft Dynatrace
  • 37. 37
  • 38. 38 What happens when we add support for a new framework? • Today, our engineers reverse engineer frameworks to add instrumentationsupport to them • Every time an update is released, the instrumentationcode is being tested. • In case of issues, it goes back to the developmentteam who needs to fix it and deploy an update. • The whole process is automated and transparent to the customer ☺ • This is costly and time consuming
  • 39. 39 In-process tracing Click GW API MonitoringSystem Trace Span Span Span HZQ Span: doHZQ()() Span: HZQ call OTEL HZQ Wrapper
  • 40. 40 “We want every platform and library to be pre-instrumented with OpenTelemetry and we’re committed to making this as easy as possible.” Sergey Kanzhelev (Google)
  • 41. 41 What is Observability and how does it differ fromMonitoring? 1. In control theory, observability is a measure of how well internal states of a system can be inferred from knowledgeof its external outputs. Source: Wikipedia 2. In software development, observabilityisachieved by adding code (instrumentation)that emits telemetry data. 3. Monitoringis the act of displayingand analyzing this telemetry data. 4. Monitoringalone can tell you that there is a problem. E.g. ”We see that some users experience a 50% higher response time on check-out” 5. Observablityhelps finding the root cause (the why) by providingdatathat can be correlatedand analysed freely even if this problem is completelynew to you (unknown unknowns) E.g. “The response time of the checkout increases exponentially with the number of items in the basket, because of a misplaced for loop that executes the same database query times the number of items for every item in the basket”
  • 42. 42 Putting it all Together • Metrics can help you to learn that there is a problem • Distributedtracing becomes increasingly importantto understandmulti-tier execution paths and root causes of problems • Developersnow rely on metrics and traces to understandhow their service functionsin their microservice architectures • Pure Open Source solutionsare viable for pre-prod environments • Standardization isthe only way to tackle today’scomplexity and Open Source is the key driver • Vendorsare prepared to tap intodata collected by Open Source standard toolsto add enterprise features on top to support web-scale workloads