Downloaded 112 times
Uploaded byAcquia
A Crash Course in Building Site Reliability
The document outlines the principles and practices of Site Reliability Engineering (SRE) as implemented at Acquia, detailing its evolution and integration with DevOps initiatives. It emphasizes the role of automation in operations, the importance of service level objectives, and the need for a cultural shift toward shared responsibility in operations. Additionally, it highlights the challenges faced during the implementation and retrospective insights gained over one year of SRE practices.
More Related Content
![Webinar: Vodafone and The Connected Customer Journey [10.19.2017]](https://cdn.slidesharecdn.com/ss_thumbnails/vodafonewebinar-2017-oct-19-171024212241-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to A Crash Course in Building Site Reliability
![Site-Reliability-Engineering-v2[6241].pdf](https://cdn.slidesharecdn.com/ss_thumbnails/site-reliability-engineering-v26241-221023035909-82e9559b-thumbnail.jpg?width=640&height=640&fit=bounds)
![Devfest Harare 2025 Slides [GDG Harare] - Combined Slide Deck](https://cdn.slidesharecdn.com/ss_thumbnails/ilovepdfmerged12-251029161756-3b829161-thumbnail.jpg?width=640&height=640&fit=bounds)
A Crash Course in Building Site Reliability
- 1. Building Site Reliability Engineering: ACrash Course Amin Astaneh, Acquia Inc.
- 2. Who am I? ●Senior Manager, SRE at Acquia ● Was in Operations Team from Dec 2010 - Nov 2015 ● Built and Lead the Site Reliability Engineering Team
- 3. Agenda ● What isSRE? ● Why Do SRE? ● Acquia, Pre-SRE ● How Acquia Does SRE ● Building an SRE Competency ● How to Hire SREs? ● 1-Year Retrospective
- 4.
- 5. What is SRE? “Whathappens when a software engineer is tasked with what used to be called operations.” - Ben Treynor, Google
- 6. What is SRE? SREtakes the manual processes associated with Operations..
- 7. What is SRE? ..andreplaces them with automation using software engineering.
- 8. What is SRE? Theyalso use a set of methodologies and best practices that help engineering teams create a mature and sustainable process for service ownership.
- 9. How Does ThisRelate to DevOps? DevOps is a set of values, tools, and processes that allow teams to best deliver value to the customer. Therefore, SRE can be considered a specific implementation of DevOps.
- 10. SRE Practices (according toGoogle)
- 11.
- 12. 2) Have SLO(s)for your service.
- 13. What are SLOs? ●SLI: Service Level Indicators (What to Measure) ● SLOs: Service Level Objectives (Targets for Measurements) ● SLAs: Service Level Agreements (Consequences for Missing Targets)
- 14. 3) Measure andreport performance against the SLO(s).
- 15. 4) Use ErrorBudgets and gate launches on them.
- 16. 5) Have acommon staffing pool for SRE and developers.
- 17. 6) Cap SREoperational load at 50%.
- 18. 7) Have excessOps work overflow to the Dev Team.
- 19. 8) Share 5%of Ops work with the Dev Team.
- 20. 9) Oncall teamsshould have at least eight people at one location, or 6 people at each of multiple locations.
- 21. 10) Aim fora maximum of two events per oncall shift.
- 22. 11) Do apostmortem for every event.
- 23. 12) Postmortems areblameless and focus on process and technology, not people.
- 24.
- 25.
- 26. Improve Employees’ Qualityof Life
- 27.
- 28.
- 33. Things We TriedFirst ● Implemented Kanban for Ops to make work visible and maximize throughput ● Did ‘Tier 2 Sprints’ to build automation for the team ● Generated team metrics to influence decision-making “People Metrics: How to Use Team Data to Produce Positive Change” https://events.drupal.org/dublin2016/sessions/people-metrics
- 34.
- 35. How Acquia DoesSRE Acquia SRE was commissioned as the driving force of our DevOps Initiative, which has the following core values: ● Eliminate Toil ● No Capes ● Deliver With Empathy ● Own Your Service ● Own Your Business ● Own Customer Success
- 36. Acquia SRE vsGoogle SRE ● We embed engineers on teams, rather than build teams that run services on behalf of engineers ● The entire engineering team (plus the SRE) is expected to ‘own their service’, with the SRE providing leadership on how to best handle those responsibilities ● The SRE identifies risk as part of their day-to-day and brings improvement opportunities directly to the Product Manager for prioritization
- 37. Acquia SRE vsGoogle SRE ● We evaluate with Engineering and Product what the most critical projects are on a quarterly basis, and allocate the team to best meet the present need ● We still reserve the right to remove engineers if an engagement becomes untenable, though it has not yet been necessary ● We have a heavy focus on time tracking to aid in toil reduction
- 38. 8) Share 5%of Ops work with the Dev Team.
- 39. 8) Share 5%of Ops work with the Dev Team.
- 40. 8) Ops workIS the responsibility of the Dev Team.
- 45. Building A SRECompetency
- 46.
- 47. SRE Won’t WorkWithout Two Things ● Authority to stop releases when the error budget has been exhausted ● Authority to overflow operational work to the dev team when operational load > 50% This must be given from lead of engineering/product efforts. DO NOT CONTINUE UNLESS YOU HAVE THESE!
- 48. How Do YouGet Buy-In?
- 49. Establish a Senseof Urgency! https://events.drupal.org/baltimore2017/sessions/%C2%A1viva-la-revoluci%C3%B3n-how- start-devops-transformation-your-workplace
- 50.
- 51. SRE Operational LoadDashboard
- 52.
- 53. Operational Responsibility Assessment ●Based on the Capability Maturity Model (https://en.wikipedia.org/wiki/Capability_Maturity_Model) ● Evaluates the following responsibilities: ○ Routine Tasks ○ Emergency Response ○ Monitoring and Metrics ○ Capacity Planning ○ Change Management ○ New Product Introduction and Removal ○ Service Deploy and Decommissioning ○ Performance and Efficiency ○ Information Security
- 54. Operational Responsibility Assessment Eachresponsibility is scored from 1-5: 1. Initial: Chaotic. Undocumented, ad-hoc, and require individual heroics. 2. Repeatable: Documented sufficiently so they can be repeated with the same results. 3. Defined: Roles and responsibilities for the process are defined and confirmed. 4. Managed: The process is quantitatively managed in accordance with agreed- upon metrics. 5. Optimizing: Process management includes deliberate process
- 56. Operational Responsibility Assessment ●Assess your services often! (we suggest quarterly) ● Take findings/risks and create tasks for improvement ● Publish your results and share them with your organization ● Do not tie ORA results to KPIs, incentives, etc
- 57.
- 58.
- 59. Blameless Post Mortems ●Document timeline of the incident ● With the team, determine: ○ What went well ○ What didn’t go well (process failures, technical root cause) ○ What was lucky (or circumstantial) ● For each thing that didn’t go well or was circumstantial: ○ File an action item to address it ○ Make sure they have clear acceptance criteria/requirements (grooming) ○ Make sure they have a clear level of effort (sizing) ○ Prioritize in the backlog based on relative risk ● Openly share the post-mortem with the rest of the company ● Review with the team periodically
- 60.
- 61. What is LaunchReadiness Criteria? ● A set of guidelines that represent the minimum standard of what a new product launch requires from an operational standpoint ● Expressed in terms of the Operational Responsibility Assessment ● Intended to address the major forms of risk without introducing needless roadblocks into the product launch process ● A living document that is continuously maintained and kept relevant ● Inspired by: https://landing.google.com/sre/book/chapters/reliable-product- launches.html
- 62.
- 63.
- 64.
- 65.
- 66.
- 67.
- 68. Example Post Mortem/RCATemplate
- 69. Create an OnboardingProcess
- 70. Create an OnboardingProcess ● Implement an Incident Response Process ○ On-Call Rotation ○ Documentation for stakeholders on how to get help ○ Fundamentals: production access credentials, runbooks ● Perform/Publish an Operational Responsibility Assessment ● Define/Publish Service Level Objectives ● Create Monitoring/Alerting against SLOs ● Create Dashboards For SLO performance and remaining error budget
- 71.
- 72. How To HireSREs?
- 73.
- 74.
- 75.
- 76.
- 77. What Makes aGood SRE? ● It’s complicated ● You want someone with the ability to contribute to a software engineering project.. ● Yet is motivated by operational concerns and understands the subject matter (Linux, TCP/IP, monitoring, performance, config management..) ● Is willing to be on-call ● Knowledge of agile practices as a method to suggest improvements ● ‘SRE Temperament’: can communicate their opinions on something in a way that is persuasive and data-driven
- 78. Selling Points forProspective SREs ● Toil capped at 50%, that means 50%+ project work at all times! ● Authority to stop flow of releases when service is too unreliable ● There is oncall, but responsibility is shared with the whole team ● Root causes of outages are tracked, prioritized, and addressed These Create A Work Environment That Respects The SRE
- 79.
- 80.
- 81. What Went Well ●Launch Readiness Criteria is now a corporate standard ● Teams are independently performing their own blameless post mortems ● Teams are independently performing their own ORAs ● SRE influenced a grassroots reorg of Cloud Engineering around SOA ● More and more teams are taking an active role in on-call responsibilities ● Weekly Office Hours has been an effective tool for sharing ideas
- 82. What Didn’t GoWell
- 83. What Didn’t GoWell ● We struggled with getting SLOs and error budgets established for all services ● We didn’t get Launch Readiness out the door fast enough for new services
- 84.
- 85. Current Improvements ● SREengagements now require the onboarding process before any other work can take place: ○ Establish Incident Response Process ○ Perform Operational Responsibility Assessment ○ Defining Service Level Objectives ○ Establishing Monitoring and Alerting Against SLOs ○ Create Dashboards Displaying SLOs and Error Budgets ● Operational Stories are required to be prioritized proportional to the SRE presence on an engineering team.
- 86. “When we werein Ops, it was simple, because our purpose was to simply address the incident. Our purpose now is to address the problems of the business. We are the vehicle of change. That’s hard work, but we can do it.”
- 87.
- 88.