Downloaded 20 times
Uploaded byMatthew Boeckman
Top 10 Practices of Highly Successful DevOps Incident Management Teams
This document discusses incident management best practices. It outlines the five phases of incident management: detection, response, remediation, analysis, and readiness. Key recommendations include using runbooks to guide remediation, implementing infrastructure as code, conducting blameless postmortems to drive learning, and organizing response teams with defined roles to improve MTTR (mean time to repair) and enable continuous learning. The goal is to create a closed-loop incident management process that detects issues early, responds quickly, remedies problems efficiently, analyzes root causes, and applies lessons to reduce future outages.
More Related Content
Similar to Top 10 Practices of Highly Successful DevOps Incident Management Teams
Top 10 Practices of Highly Successful DevOps Incident Management Teams
- 2. Matthew Boeckman Developer Advocate @matthewboeckman Background ●18 years on-call Ops ● 15 years w/software teams ● Startup junkie ● DevOps enthusiast
- 3. 3 What is VictorOps? VictorOpsingests all of your alerts from your current monitoring tools and becomes the logical layer between your alerts and the people who receive them.
- 4. 4 Five Phases ofIncident Management Detection ● monitoring ● metrics ● thresholds Response ● alerting ● on-call ● escalation Remediation ● fixes ● tickets ● deployment Analysis ● postmortem ● how or why ● understand Readiness ● improve ● game days ● learning 1 2 3 4 5
- 5.
- 6. 6 MTTR - OurCore Metric Detection Response Remediation Analysis R e a d i n e s s Time to Repair (MTTR)
- 7.
- 8. 8 Detection - Response- Remediation - Analysis - Readiness Detection
- 9. 9 #1 Blended approachto Detection#1 Blended Approach to Detection1
- 10. 10 #1 Blended Approachto Detection ● Synthetic Testing ● Time-series data ● Application Monitoring ● Log Analytics
- 11. 1 Detection - SyntheticTesting Existing User? Synthetic monitoring leverages scripted web interactions to validate critical path user interactions (or system interactions). Landing Page Registration Welcome Stream User Home User Home InteractionLogin
- 12. 12 Detection - TimeSeries Systems are not static. Why treat measurements as if they were?
- 13. 13 Detection - Rateof ChangeDetection - Rate of Change
- 14. 14 Detection - ApplicationPerformance (APM) RUM Transactions Page Performance Thread Profiling Application internals Timings and Counters Microservices Transactional path 3rd party calls Dependency Management Circuit Breakers APM monitors complex application behaviors and reports or alerts on deviation from norms. UX Runtime Tracing Internals
- 15. 15 Detection - LogAnalytics Log Analysis opens new detection avenues with insight particularly into security and compliance concerns.
- 16. 16 #2 Focus onBusiness Outcomes#2 Focus on Business Outcomes2
- 17. These datapoints becomeDetectable conditions for actionable alerting. 17 #2 Focus on Business Outcomes Business Activity Monitoring maps key metrics to data or flows within the IT environment.
- 18. Email Deliverability SEM/Referral Volume Channel Activity 18 Business Objectives Revenue (SDLW, YoY) Transactions User Registration Conversion NPSScores Social Media Reviews KPI 3rd Party Indirect
- 19.
- 20. 20 Detection - Response- Remediation - Analysis - Readiness Response
- 21. 21 #3 Alerts areActionable#3 Alerts are Actionable3
- 22. 22 Alert Fatigue isa Leading Cause of Burnout “I get paged for issues that I can’t resolve; most of my time is either researching a problem that is transient or non reproducible, or contacting a vendor.” “I’m on call for everything from infrastructure issues to, much more often, application issues for software I didn’t write and don’t own.”
- 23. 23 Actionability Exists onTwo Dimensions The alert must be actionable, and differentiated from something that’s merely informational.
- 24. Alerts must routeto someone who has the access, permission, and skills to adequately perform said action. 24 Actionability Exists on Two Dimensions
- 25. Ensure you’re measuringthe right thing, at the proper precision. 25 Alert Fidelity
- 26. 26 #4 ChatOps isan area of focus#4 ChatOps is an Area of Focus4
- 27. 27 Providing a CentralCollaboration Channel
- 28. 28 Integrating with Informationand Workflow
- 29.
- 30.
- 31. 31 Detection - Response- Remediation - Analysis - Readiness Remediation
- 32. #5 Runbooks arecentral to Remediation efforts 32 #5 Runbooks are Central to Remediation Efforts5
- 33. 33 Great Runbooks... ...provide clear explanationsof system metrics and alert thresholds
- 34.
- 35. 35 Great Runbooks... ...track incidenthistory ...list known failure conditions
- 36. 36 Great Runbooks... ...are accessibleto all ...are routinely updated
- 37. 37 #6 Configuration andInfrastructure as Code#6 Configuration and Infrastructure as Code6
- 38.
- 39. 39 Infrastructure as Code ●Empower Dev and Ops to manage systems ● Create adaptable patterns ● Enable quicker iterations ● Ensure auditable change management ● Common deployment pipelines ● Instrument everything!
- 40. 40 Detection - Response- Remediation - Analysis - Readiness Analysis
- 41. 41 #7 Data drivesinvestigations#7 Data Drives Investigations7
- 42. 42 Analysis of Systems *ArchonMagnusWikipedia
- 43.
- 44. 44 There is noRoot Cause
- 45.
- 46. 46 #8 Postmortems areblameless#8 Postmortems are Blameless8
- 47. 47 Great Postmortems... Focus onlearning Focus on data Require objectivity Encourage participation
- 48. 48 Detection - Response- Remediation - Analysis - Readiness Readiness
- 49. 49 #9 Postmortems populateBacklogs#9 Postmortems Populate Backlogs9
- 50. 50 #9 Postmortems PopulateBacklogs Call Postmortem Overview of Incident Review timeline Review Remediation Review Response Discuss improvements Add to Backlog Involve PM
- 51. 51 #10 Organize teamresponses#1 Organize Team Responses10
- 52. 52 Organization of theSwarm Formal escalation policies including a catch-all escalation
- 53. 53 Who Ya GonnaCall? Formal communication plan
- 54. 54 Defined Roles Incident Commander(Quarterback) - directs and dedupes team efforts. Calls the ball on action
- 55. 55 Defined Roles Incident Communicator(Scribe) - handles all async communication, records team actions
- 56. 56 MTTR - OurCore Metric Detection Response Remediation Analysis R e a d i n e s s Time to Repair (MTTR)
- 57. 57 Conclusion D e t e c t i o n R e s p o n s e Remediation Analysis Readiness Time toRepair (MTTR) Time to Learn (TTL)
- 58. THANK YOU! @matthewboeckman Slides ondevops.com & slideshare.com