Downloaded 12 times
![Developing on the Trunk
• Use feature toggle flags
• Each dev has a full system - update, compile, run it all, test-it
• Write backwards compatible code
– Assume some servers/services do not have the new feature/code
yet
– DB fields may be missing
– Mark older APIs with [deprecated/obsolete]
• Dynamic Configuration
– Central config, dynamically loaded
– Plan for no app/system restarts
Thor n
Fr ee?!?
• Schema changes are added to code as DB versioning scripts
• Larger features that require a branch are rare!
• Education: Do not break the trunk!
2/15/2014
© All rights reserved
19](https://image.slidesharecdn.com/agilesaturdaycontinousdeployment140215slideshare-140215162436-phpapp02/75/Making-the-Agile-Leap-to-Continuous-Deployment-18-2048.jpg)
Uploaded byEthan Ram
Making the Agile Leap to Continuous Deployment
The document is a presentation about making the transition to continuous deployment. It discusses setting goals for continuous deployment, techniques for deployments and development, how to make the transition, and some unexpected positive results. The presentation encourages moving from large releases to smaller frequent releases through techniques like feature flags, database versioning, and deploying new features to a test environment first.
More Related Content
Similar to Making the Agile Leap to Continuous Deployment
![Governance, Deployment & Methodologies for Agentic Automation [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day2-251112135038-a386476d-thumbnail.jpg?width=640&height=640&fit=bounds)
Making the Agile Leap to Continuous Deployment
- 1. © All rightsreserved Want to use slides from this presentation? Ask for permission: [email protected] Making the Agile Leap to Continuous Deployment Ethan Ram / 15 Feb 2014 V1.1
- 2. Ethan Ram -Bio • Chief architect of Videobet • Leading Videobet’s R&D and OPs teams Agility efforts • As VP R&D of GameGround transformed the R&D to run in Continuous Deployment environment • Hacker in soul and a Windows Internals hobbyist 2/15/2014 © All rights reserved Find me here: • [email protected] • LinkedIn • The Null Terminator blog 2
- 3. Videobet • A Playtechdivision • Innovative Gaming Platform Provider – Established 9 years ago – 150 employees in Tallinn and Kiev – Operate in 9 countries – 60% of UK market • R&D – Front-end teams (C++) – Server teams (Java) – Game development teams 2/15/2014 © All rights reserved 3
- 4. Agenda #1 inciple Pr anifesto Agile M y to satisf tyi s a nd st priori ur highe r through early le O e ab e custom elivery of valu th us d ontinuo c e. softwar 2/15/2014 • • • • • • • © All rights reserved Why Continuous Deployment Setting the goals Deployment technics Development technics Making the leap Some unexpected results Q&A 4
- 5. Goal: Continuous Deployment Wix“From Jan ‘13 to Jun ’13 we did 1500 At SlideShare, instead of big launches, we Deployments, 470 A/B Tests, We have been doing continuous deployment release code all the time... it reduces risk. 200 Feature Toggles” at Plex since our inception in 1995. We Since we are putting out small changes at a Yoav Avrahami formalized the process with our own “deploy”time, if it breaks we know what it is and can button in 2003, and hit the 25k deployment roll back. mark less than two years thereafter. We Rashmi currently average between 50-100 deployments per day to our massive online Flickr "10+ Deploys Per Day" ERP system. WordPress.com - we’ve averaged about 16 Conover Steve of the exciting things… about Digg is Jerry Foster One product releases a day, every day for theOne of the exciting things… about Digg is last continuous deployment --when developers fix continuous deployment when developers fix four and a half years… Everyone in our athat or add new feature, there's no need bug company has access to a deploy button abug or add aanew feature, there's no need to wait for scheduled release… the to releases the latest checked in code to about wait for aascheduled release… the turnaround time for aachange drops turnaround time for change drops 400 production servers in our web tier in less dramatically. dramatically. than 30 seconds (across 3 data centers). Andrew Bayer Andrew Bayer Toni Schneider 2/15/2014 © All rights reserved 5
- 6. Major Release 2/15/2014 © Allrights reserved 6
- 7. No. I Can’tDo That • “We’re starting this campaign next week and I need the product to do X” • “We need to stop showing this feature to non-US customers” • “I think this idea would work. When can we start A/B testing live?” • “I need to know if users are clicking on this link and if they see this error message” • “Forget about the other features. We have to go live by the end of the month!” 2/15/2014 © All rights reserved 7
- 8. 2/15/2014 Continuous Deployment: Yes WeCan! © All rights reserved 8
- 9. How to StreamlineDeployment? • Speed-up Dev-ops • Lower risk of deploying something broken • Streamline feature implementation 2/15/2014 © All rights reserved ??? 9
- 10. Monitoring & Rollback •Must haves – real-time – Monitor that features are working, not just servers – Monitor the business results – Visualize and put it on the big screen – One-click rollback • Rollback: the ultimate risk mitigation – Prepare to fail! • Will the rollback work? – Rollback drills 2/15/2014 © All rights reserved 10
- 11. Deployment & ServersSetup Go from – 8 hrs downtime on upgrade • Physical machines • IP based network topology • Layered servers: web-apps, backend, cache, db • Feature servers • Each team has a different deployment setup and scripts • Complex database schema upgrades To – 8 mins deploy, no downtime • Virtual machines • DNS based network topology • All servers on each VM – DB too!!! • Same VM setup on all environments – With different config – With different data-set • One deployment script run on dev machine >> production – Target tag and environment name as input I T’S EASY! 2/15/2014 • Versioned mini schema migrations © All rights reserved 11
- 12. DB Versioning WithFlyway DB Flyway DB makes you Agile! • Mini schema migrations kept with source code – Applying migrations added between v1 to v2 is easy – Roll-back of migrations supported built-in • Each developer has her own DB • Merging team-branch migrations is even simpler than merging code • Integrative with Maven/Ant/scripts and Java APIs – Server can upgrade/lower DB schema by itself • Data for unit-testing can be added and removed on the fly 2/15/2014 © All rights reserved 12
- 13. Staging on Production •Get real users to use new version ASAP – Call it Beta, Early Availability, Version Trial, Testbed • Deploy new version to part of the servers and direct it traffic of part of the user-base – – – – – Deploy to a subset of servers Run manual tests & automated tests Direct some traffic to new version Monitor! Measure! Deploy to the rest after some time • Not only for consumer-based internet products! 2/15/2014 © All rights reserved 13
- 14. Quality Assurance • IntegrationTest Automation – Focus first on covering main data flows – More main features, less details • Part of the Continuous Integration – Build > Unit-test > Integration Tests – All tests must pass to allow build promotion … not so easy… • Run the same tests on all environments – Developers’ & production environments too • Don’t give-up on manual testing • ‘Test frenzy’ as version signs-off – One hour session, structured, lead by QA – ALL employees join. The secretary too. 2/15/2014 © All rights reserved 14
- 15. R&D Kanban Board:CD Workflow BACKLOG Backlog SELECTED Selected • Bugs • Improvements. • Daily prioritization 2/15/2014 IN In PROGRESS Progress •Feature owner • Coding • Unit testing assigned •Task • Integration testing Breakdown •User stories •Design/gfx •Test plan RESOLVED Resolved VERIFIED Verified • Need verification • Drop build to QA • Automated integration testing • Manual testing TESTBED Testbed • Verified • Ready to be installed on TestBed • Optional step: install on TestBed • Regression Testing • Enforce WIP limits • One owner for each column • Feature owner responsible for feature end-to-end… • QA plan and test completion (no need for bugs!) • Unit testing and integration testing • Use DBA/HTML/FLASH and QA resources as needed • Test it on testbed/production © All rights reserved DONE Done • Running on production • Done / Not relevant 16
- 16. Developing on Trunk 2/15/2014 ©All rights reserved 17
- 17. Use feature toggleflag 2/15/2014 © All rights reserved 18
- 18. Developing on theTrunk • Use feature toggle flags • Each dev has a full system - update, compile, run it all, test-it • Write backwards compatible code – Assume some servers/services do not have the new feature/code yet – DB fields may be missing – Mark older APIs with [deprecated/obsolete] • Dynamic Configuration – Central config, dynamically loaded – Plan for no app/system restarts Thor n Fr ee?!? • Schema changes are added to code as DB versioning scripts • Larger features that require a branch are rare! • Education: Do not break the trunk! 2/15/2014 © All rights reserved 19
- 19. Product: From Revolutionto Evolution V2.2 V2.3 / R403 R417 2/15/2014 © All rights reserved 20
- 20. Leaping into CD •Initiative must come from high mgmt. – It’s a large investment… and risky money t ime • It’s not all or nothing >>> – First milestone: weekly release – Set a deadline and commit: “we start releasing once a week right after the next major release in September” • It’s a project to deliver – Multi-function task-force: devs, QA, deployment – Full time! – Plan a lot – detailed task list etc. • Training to all teams: process, tools • Decide on a CD weekly schedule – Tuesday afternoon is best time for a release • Make results visible early! – And celebrate success! 2/15/2014 © All rights reserved 21
- 21. The first CDrelease felt a bit like this… ..Just Go For It! 2/15/2014 © All rights reserved 22
- 22. Some Unexpected Results •… the sky did not fall on us • Excellent process transparency – – – – R&D stopped fighting with Product Mgmt. Team leaders stopped blaming each other Devs thought it’s much more fun Everyone got more involved • We managed to attract much better engineers • R&D managers thought its awesome! 2/15/2014 © All rights reserved 23
- 23. Want to useslides from this presentation? Ask for permission: [email protected] Q&A Thanks for contributing! Yoav Avrahami @ Wix Mike Mcgarr @ Excella Thanks! 2/15/2014 © All rights reserved 24
Editor's Notes
- #4 Expand project mode in VB slide; expand on Playtech
- #8 Credits: These questions were taken from ThoughtWorks CD marketing decks
- #16 Feature owner responsible for feature development including… QA plan and test completion (no need for bugs!) Unit testing and integration testing Use DBA/HTML/FLASH and QA resources as needed QA per feature Unit testing - Test automation! Verify the feature before and after commit Sanity on TestBed Feature enable flag Enable/disable flag for each feature in config Early Availability group to test features first
- #17 Feature owner responsible for feature development including… QA plan and test completion (no need for bugs!) Unit testing and integration testing Use DBA/HTML/FLASH and QA resources as needed QA per feature Unit testing - Test automation! Verify the feature before and after commit Sanity on TestBed Feature enable flag Enable/disable flag for each feature in config Early Availability group to test features first
- #19 One of the key components to successful CD
- #23 We missed the CD deadlines only twice in We never had a to press the rollback