Instrumenting the Twelve-Factor App
                                  TM
@josephruscio


Co-Founder/CTO Librato


I <3 graphs
Twelve Factor App


• Decouple app from environment
• Service Oriented Architecture
• Continuous Deployment
• Horizontal Scaling
?   How do we validate the effects of a deployment?
?   How do we detect aberrations?
?   How do we determine when to scale?
?   How do accomplish this with ephemeral processes?
Instrument during development
Instrumentation is
   Unit Testing
  for Operations
Counter


#enqueue a job
Foo.counter('mygem.jobs').increment

#complete a job
Foo.counter('mygem.jobs').decrement
Timer

# block form
Foo.timer('req.foo').time do
  process_request
end

# explicit form
t = Foo.timer('req.bar').time
process_request
t.stop
Decouple collection from reporting
Observer effect
Math is involved
Logs as streams
Logs


• logplex
• l2met
• logstash, greylog2
• Splunk, Papertrail, et al
Process Reporter

   Request   Metrics
   Thread    Registry




             Reporter
              Thread

 Process
Process Reporter


• CodaHale Metrics (JVM)
• metriks (Ruby)
• folsom (Erlang)
• librato-rails, librato-rack
UDP

    Nginx         Unicorn




                  StatsD


Front-End
Store metrics in a common repository
Anti-Pattern



         Storage               Storage                    Storage


                                           ...
•   Ping              •   CPU
•   CPU               •   Interface              •   Custom Stats
•   Memory            •   Memory                 •   MySQL threads
•   Disks             •   Ping                   •   VMstat
•   SNMP Service      •   Battery charge         •   ....
•   ....              •   ....




     Project X               Project Y                 Project Z
Arbitrary correlations
TM
Significant events
TM
TM
Time Series Datastores


• RRDTool
• Graphite (Whisper)
• OpenTSDB (HBase)
• Librato et. al (SaaS)
Define per-service operating ranges
Trigger alerts
Apply backpressure
TM
Collaborate through Dashboards
“No aberration detection
mechanism is more sophisticated
than your VP Marketing staring at
a dashboard whilst drinking
coffee.”
TM
Iterate and Refine
Signal
------------
   Noise
• Instrument during development
• Decouple collection from reporting
• Store metrics in a common repository
• Define per-service thresholds
• Collaborate through dashboards
• Iterate and refine
fin

Librato's Joseph Ruscio at Heroku's 2013: Instrumenting 12-Factor Apps