Download to read offline
More Related Content
Similar to So you-want-to-go-faster
![Governance, Deployment & Methodologies for Agentic Automation [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day2-251112135038-a386476d-thumbnail.jpg?width=640&height=640&fit=bounds)
So you-want-to-go-faster
- 1. So You WantTo Go Faster? A Journey to Continuous Deployment
- 2. • 10+ yearssoftware development experience • Work at Excella Consulting (~5 years) • I love testing, all things agile, cats • If any of those things interest you… About Me Dan Davis
- 3. Agile + CatsCats that can test
- 4. More About Me Wedeploy to production about 4-5 times per day!
- 7. How do yousignal to turn right?What’s a turn signal?
- 8. How do Iget to Continuous Deployment?
- 10. How do Iget to Continuous Deployment? • Tools / Techniques • Cultural Changes • Warning: I’m going to gloss over Tools / Techniques
- 11.
- 12. Continuous Deployment “Every committriggers a deployment to production”
- 13.
- 14. • Works, buthow fast? • Top Speed: limited by… – how frequently I check the repository – how quickly I can perform the deployment • Probably not a great process IMHO Not really Continuous Deployment Code Repository Deploy to Production
- 15. • Need somekind of automation How can we go faster? Code Repository Deploy to Production Bottlenecks
- 16. Automation Tools • 3basic characteristics • Execute a script / code (in a task/job) • Trigger task/job – On demand – On a timer – On an event, like a code commit • Start other jobs / tasks when completed – Avoid one big script, create logical separations – Assembly line, but for code!
- 17.
- 18. +Automation, good processyet? Code Repository Deploy to Production Should we just let any commit go to production? (on commit)
- 19. Deploy to Production Approval Process Whatinforms the approval process? Approved? Code Repository
- 20. Add in Testing Deployto Production Post-deploy Testing Approved?
- 21. Non-production Environment Deploy to Production Deployto non- production Post-deploy Testing Approved? Let’s Rearrange…
- 22. Pipeline Outline Deploy to non- production Acceptance Test Deployto production How do I get servers to run this code?
- 23. Setting up Infrastructure Howdo I keep track of what’s been approved? Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Deploy to production
- 24. Creating Build Artifacts CanI test sooner than post-deploy? Artifact Repo Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Create Build Artifacts
- 25. Run Unit Tests Provision Non-prod server Deployto non- production Acceptance Test Provision Production Deploy to production Create Build Artifacts Unit Tests / Static Analysis How do I know it continues to work after I deploy?
- 26. Monitoring Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production CreateBuild Artifacts Unit Tests / Static Analysis Monitoring
- 27. Generic Pipeline • Thisis the basic machinery • Speed is an important variable – Depending on speed we may choose different tools • Continuous Deployment is a speed, not a process or tool – To achieve that speed, we have to work up to it – Think of it like shifting gears Unit Test / Static Analysis Create a build artifact Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Monitoring
- 28. • Once PerMonth • Once Per Sprint (2 weeks) • Once Per Week • Every 2-3 days • Every Commit (multiple times per day) 5 Gears of Production Deployment
- 29. 1st Gear (Deploying oncea month)
- 30. • Basic ContinuousIntegration – Run unit tests when code is committed • Manual Elements – Acceptance Testing performed by a QA team – Deployments performed by a deployment engineer – Servers are manually provisioned as needed 1st Gear Teams Unit Test / Static Analysis Create a build artifact Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Monitoring
- 31. • Deployments proneto failure – Knowledge siloed in handful of people • Deployments require downtime – Must be done on weekends or late at night (with little support) • Lack of consistency between environments – Leads to deployment problems and bugs 1st Gear Pain Points
- 32. Do You WantTo Go Faster? (Deploying Every Sprint)
- 33. • Agile! Scrum! –Potentially Shippable Product Increment! Shifting to Sprintly Deployments
- 34. • Main LimitingFactor: Deployment failure rate – High failure rate will prevent you from speeding up… • Focus on – Standardizing the deployment process (though automation) – Building out a pipeline – Learning from build failures 2nd Gear Focus Areas Unit Test / Static Analysis Create a build artifact Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Monitoring
- 35. Automating Deployments Fabric Powerful Configuration ManagementTools Deployment Scripting Frameworks
- 36. • Basic SSHwrappers around shell commands Fabric / Capistrano Server Command Command Command Manual Execution Server Command Command CommandFabric / Capistrano File Version Control
- 37. Benefits • Simple approach,one step above shell scripts • Low learning curve • Does just enough… – Multi-host deployment – Organize into roles • Great for lightweight automation
- 38. • Now wecan establish a basic pipeline • Next Challenge: only deploy from pipeline • When pipeline breaks, prevents deployments until fixed • FAQ: Why should I care about failures in the pipeline? 2nd Gear Cultural Changes Unit Test / Static Analysis Create a build artifact Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Monitoring
- 39. • When yousee the check engine light, what do you do? – Take it to mechanic – Don’t keep driving! – Letting it fester will only make it worse! • Treat your pipeline like your car! – When it breaks fix it with a sense of urgency – Minimize the possible changes that caused it to break in the first place Pipeline Failures
- 40. • Now havea somewhat reliable build process – Our build engineers should be a lot happier – Easier to recruit new people for deployments • Lots of manual steps, but should be fast enough to deploy every sprint… The power of 2nd gear Unit Test / Static Analysis Create a build artifact Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Monitoring
- 41. So You WantTo Go Faster? (Deploying every week)
- 42. • Main LimitingFactor: Environments – Deploy once per week: start thinking about impact of downtime • Focus On – Improving consistency between environments – Standing up infrastructure – Eliminating reasons to SSH to servers Shifting into 3rd gear Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Create Build Artifacts Unit Tests / Static Analysis Monitoring
- 43. • Deployments oftenrequire service interruption – Stop processes/services – Drain queues – Update packages • Can try to reduce deployment time, but not worth it • Better Approach: Blue/Green Deployment Why Downtime Matters
- 44. Blue/Green Deployment Server A ServerA Server A V1 Code Load Balancer The internet V2 Code
- 45. Blue/Green Deployment Server A ServerA Server A V1 Code Load Balancer The internet Server B Server B Server B V2 Code
- 46. Blue/Green Deployment Server A ServerA Server C V3 Code Load Balancer The internet Server B Server B Server B V2 Code
- 47.
- 48.
- 49. Configuration Management Tools MutableInfrastructure • Apply changes to a running server • Bring a server into compliance Base Image + Configurations
- 50. • Shift inmindset: Infrastructure as Code – Virtualization means no more owning physical servers – No more dedicated hardware Culture Changes Yes, this is a thing you can buy…
- 51. • Orchestrator +Config mgmt + Deployment • Reliable deployments with no downtime! – Consistency between environments – Blue/Green deployments • Can even start deploying to non-prod more frequently • Team should settle into a pretty good groove with weekly deployments Power of 3rd Gear! Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Create Build Artifacts Unit Tests / Static Analysis Monitoring
- 52. Do you wantto go faster? (Deploying every 2-3 days)
- 53. • Ironically, oneof the hardest gears • Traditionally manual processes become unsustainable • More cultural battling than technology Shifting into 4th gear
- 54. • Main LimitingFactor: Unready / Unfinished code – Barely enough time to develop – No time to manually test • Focus On – Improving confidence in automated testing – Eliminating manual testing – Using Feature Toggles Shifting into 4th gear Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Create Build Artifacts Unit Tests / Static Analysis Monitoring
- 55. One Week DeploymentCycle Coding Coding Coding Testing / Bug Fixing Deployment / Happy Hour Day 1 Day 2 Day 3 Day 4 Day 5 (Tu)(Wed)
- 56. 2 – 3Day Deployment Cycle Coding Testing / Bug Fixing Deploy ment? Day 1 Day 2 Day 3 Bottleneck We must eliminate the “code freeze” period before deployment
- 57. • Acceptance Testing(i.e. New Functionality) • Regression Testing (i.e. Old Functionality) • Both are highly predictable – Great candidates for automation Cutting down testing time
- 58. Tools for AutomatingAcceptance Tests “Web Driver” • Mimics the actions of a user in a browser
- 59. • Who writesthe automation code? – Testers don’t typically have the skillset – Developers are better suited for it • Developers must now write feature code AND test code? – Is that fair? Culture
- 60. • Engineers Day •Deeper appreciation for what you build • Encourage thoughtfulness about how to design parts Automotive Engineers
- 61. • Developers shouldhave a stake in the quality of the final product – Developers should learn what types of bugs get through and how to prevent them • Encourage mindfulness of code – Hard to test? Find ways to make it easier • Promote a better collaboration with Product Owner – Work closer with acceptance criteria Developers Responsible for Quality
- 62. Feedback Cycle Developers Write Code Deploy Code QA Team TestsCode No longer going to have an explicit “Testing” Phase
- 63. 2 – 3Day Deployment Cycle Coding / Continuous Testing Deploy ment? Day 1 Day 2 Day 3 Is 2.5 days enough time to develop a useful feature?
- 64. • Can’t completein everything 2-3 days • Catch 22: Can’t merge it until all stories are completed • Maybe we should slow down… Useful Functionality Useful Feature Story 1 Story 2 Story 3 Story 4
- 65. • A littlebit of conditional logic that can enable/disable functionality • Often stored in a cache or database Feature Toggles
- 66.
- 67. Deploying With FeatureToggles Non-Prod Production Story 1 Story 2 Story 3 Story 4 Story 1 Story 2 Story 3 Story 4
- 68. • Decoupling codedeployment with releasing functionality – At previous speeds, deploying to production meant turning on functionality – Replaced with turning on a feature toggle • Culture Change: Deploying to production is an unremarkable event – The same scrutiny is no longer needed – “Rollbacks” are now effortless, just un-toggle Culture Change
- 69. • Automated ourtesting process to reduce testing time to minutes • Using feature toggles allows our team to work extremely efficiently – Ship what’s ready, keep working on what’s not • Should be able to comfortably deploy every 2-3 days – There’s very little in the way of going faster… The Power of 4th Gear Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Create Build Artifacts Unit Tests / Static Analysis Monitoring
- 70. Do you wantto go faster? (Deploying on Every Commit!) the most fast??? ^
- 71. • Main LimitingFactors: Efficiency – Creating new infrastructure is slow – Push button deploys are getting in the way • Focus On – Shore up testing with monitoring and alerting – Immutable Infrastructure – Flip the switch to automatically deploy to production Shifting into 5th gear Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production Create Build Artifacts Unit Tests / Static Analysis Monitoring
- 72. • Spending alot of time configuring servers Immutable Infrastructure All Configs Base OS New Server Code Image
- 73. • Spending alot of time configuring servers Immutable Infrastructure All Configs Base OS New Server Image Code Image w/ Code (v1.0)
- 74. Immutable Infrastructure Image w/ code(v1.0) Non-Prod Production Very Fast
- 75. • New codemeans building a new image – Creates a “build artifact” • Rather than updating, just destroy and recreate Immutable Infrastructure
- 76. • Even withperfect deployments, gap in our testing • What happens after we deploy? – A flood of users might hit the server all at once – A memory leak could start sapping performance – Networking could fail Monitoring / Alerting
- 77. • Alerts: whencertain conditions are met, send a notification • If we see a sudden spike in errors – Alert the developers – Let the Product Owner know – CC the security team • Be proactive! – Don’t just wait for users to tell you it’s broken Monitoring / Alerting
- 78. What’s left? Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deployto production Create Build Artifacts Unit Tests / Static Analysis Monitoring
- 79. • Cultural Challenge:Is it ok to automatically deploy to production?? Culture Change
- 80. • Continuous Deploymenttakes Bravery – Trust that your automated tests will catch show stopper bugs – Dedicate yourself to fixing bugs quickly (because you can) • Continuous Deployment is a speed, there are trade-offs – Only achievable with a dedicated and competent team • You’re not the first – I run a team at Excella that is doing this – We built on the shoulders of others before us Culture Change
- 81. So take adeep breath…
- 82. Provision Non-prod server Deploy to non- production Acceptance Test Provision Production Deploy to production CreateBuild Artifacts Unit Tests / Static Analysis Monitoring
- 83.
- 84. Do you wantto go faster?
- 86. • Once everycommit deploys to production, you may start to redline • Limited by – # of commits per day – Longest step in pipeline • Most teams won’t hit upper limit – ~5 min / step == 100 deployments (8 hour day) • Consider breaking into microservices Going Faster
- 87.
- 88. • Fastest Route •Scenic Route • Drive the Speed Limit • Speed like crazy • How you get there is up to you… Final Thoughts
- 89.