SanDiego_DevOps_Meetup_9212016
- 1. Mitchell CI/CD Journey Continues… Richard Fong (Raj) Rajwinder Singh Makkar
- 2. Introduction
- 3. Mitchell International • Founded in 1946, with 70 years of experience in Property, Casualty claims, and Collision Repair industries • 50 million transactions annually • 300 insurance companies/claims payers • Over 30,000 collision repair facilities • 2,000 associates
- 4. Raj Makkar Senior Configuration Engineer @ Mitchell Developing solutions around software release automation, dependency management and continuous integration for various software products in Mitchell. Currently promoting continuous integration and continuous delivery solutions to increase efficiency within Mitchell’s software delivery pipeline. Before Mitchell use to work for GE.
- 5. Richard Fong CICD enthusiast with experience creating service oriented infrastructure to efficiently and rapidly build, integrate, and deliver quality software product for large enterprise. Richard’s has worked with many known companies such as Yahoo, Intuit and Qualcomm in architecting their full CICD tool stacks and evangelized CICD for those organization.
- 6. Journey
- 7. Keep in mind it’s a journey … Insert Screenshots
- 8. 7 Years Ago • Componentized • Developer and SCM Team maintain the Build • Developer maintain the Build of Materials • Developer decide when to Release • Central QA Team run Test Manually Issues • Dependency management is a pain • Custom deployment for each component • Multi days planning just for release coordination
- 10. 4 Years Ago • Big Build • System Team maintains the Build • System Team maintain the Build of Materials • System Team decide when to Release • Central QA Team try to automate test Issues • The Build is slow • Dependency and Deployment are bottleneck by a single team • Developer cannot run local big build
- 11. “Left Is the New Right”
- 12. Today • Componentized • Developer maintain the Build • Developer maintain the Build of Materials • Developer decide when to Release • Central QA Team run a mix of Manual and Automated Tests Benefits • Componentized with Standard Packaging • Component is Developer’s responsibility • Build, Deploy, Test on check-in with Immediate Feedback
- 13. How the Journey Begin Componentized Who control build dependen cies Who maintain packaging Who decide on what to release State of QA 7 years ago Yes Dev+scm team N/A Dev Central QA team 4 years ago Big build and no component build System team System team System team Being Agile ;-) + partial workflow QA team Today Yes Dev Dev System team Central QA team
- 14. “Build Highway, Not Traffic Light”
- 15. Lots of Terms • CICD • DevOps • TDD • BDD • Don’t break the build • Unit Test • Pair programming
- 16. The Mission!!! Is not about the terminology, but our Mission to “Deliver Quality Product FAST”
- 17. Tools
- 18. VM Image Layer Application OS Baseline Platform Baseline Mitchell Baseline Environment Configuration Image Baseline Application Deployable Artifacts Environment Configuration
- 19. From of Application Artifact Application OS Baseline Environment Configuration ISSS Baseline SCM Baseline Assembled Application Application Layer • The Delta • Reproducible • Flexible • Simple to use
- 20. Shipping Application by Packages Release Tools App.jar Runtime Client Lib Other Things N-th Package Application Packages
- 22. Vision on Shipping Enterprise Application Package Creation Code Creation Application Creation System Deployment System Creation System Promotion
- 23. ? Windows RHEL One Tool to Rule Them All
- 24. Deployment Standard • Package to manage files • State to manage packages • System to manage States
- 25. Isolation Env Integration Env CI Build Farm Mitchell Env(s) Continuous Delivery Pipeline Application OS Baseline Mitchell Baseline Other Baseline Promote Integration Verification Auto Deploy Build + Unit Test + Package Service Verification Promote Application Env Configuration Test Agents Test Agents Test Agents
- 26. Many Technologies are Used
- 27. Use Jenkins/Artifactory • From 1 Big Build to 1000 Component Build, Deploy, Test jobs • Do about 1400 job runs / day • Average life of slave is 12 hours • Use of Docker and AWS to scale Jenkins Slaves • Use of push model rather then polling for check-ins ( loosely coupled) • New version of artifact for each check-in • Artifactory serve as the centralized artifact repository • 800+ GB in about 12 months
- 28. Same Data for Reporting Java .Net Python Javascript
- 29. Pipeline as Code • Too many builds to manage manually • Pipeline defined as JSON configuration • Let Developer maintain their own pipelines • Ability to create a Pipeline for Branching • Job template to allow • Re-usable for different projects • Flexible in adopting new technology
- 30. Processes
- 31. Category of CI/CD Capabilities Build Deploy Test Promote Continuous building software and validate new source code Continuous deploying software and validate new compiled code Continuous integration testing for software cohesive functionalities Continuous promotion of software for feature availability to market
- 32. Product Taxonomy • Systems • Services • Components • Libraries • Ownerships Product
- 33. 10/4/2016 | Page 33 Capabilities and Adoption Build Deploy Test Promote Product-A Product-D Product-C Product-B Capabilities and Adoption CI/CD Capabilities CI/CDAdoption CI/CD Execution Grid
- 34. Software Technology Not Yet as a Service Platform Infrastructure JBoss IIS Apache Oracle SQL Server Network System or VM Storage Backup Monitoring
- 35. Software a.a.S Whose “aaS” do you kick? Platform a.a.S Infrastructure a.a.S System Admin SCM System Team Network Team System Admin Storage Team Monitoring Team IT Developer QA Application Operator
- 36. Dev, QA, Ops, SCM, IT – Fragmented pipeline with walls Dev - We completed our story and task given by PO QA - We tested what we think we need to deliver to customer OPS - I hope Dev did well and QA tested it well SCM - Code is in version control , we can track it. IT - We need to keep hardware up
- 37. Left Is the New Right • Dev -> QA -> Ops -> SCM -> IT • Pushing ownership to Developer • You build it, you own
- 38. Test Pyramid Martin Fowler: Test Pyramid Reference: http://martinfowler.com/bliki/TestPyramid.html
- 40. Pyramid on Top of Pyramid Unit Service Int/UI Unit Service Int/UI The Super System Tests System A System B
- 41. Software Quality Anti-Pattern • High level integration test is better than no test • Manual QA transitioning to Automation does not work • No emphasis on Unit Test • Big Build cannot be run on local machine
- 42. Software Quality Done Right • Emphasis on Unit Test • BDD to connect feature scenario sign off and test • Service testing to eliminate external dependency • Leverage local machine and VMs for functional validation • Test Category • Run all your tests
- 43. “Pipeline, Bucket of Sewage Water without Test”
- 44. Where Developer Spent Their Time • Development Work • Build fixes • Deployment • Support • Meetings • Design • Test Support
- 45. We Don’t Support BS • Do the right thing, not just automating current process • Is the process event right • People focus on current process and forgot about the reason
- 46. Culture
- 47. Some aspects … CICD and Quality is CEO’s Priority Dealing with Merger and Acquisition Acceleration to cloud related efforts No big blue print designs before starting
- 48. Speak the same language across BUs Java .Net Python Javascript
- 49. Should We Break the Build • Builds are broken, and no one is looking at it • Don’t break the build • With TDD, break the build, then fix the build
- 50. Hiring Challenges • Hiring DevOps? You want a Developer? Or an Operation Expert? • Developer with no Ops/System/Network experience • System Admin with no Development experience • SCM in the middle but weak on both ends • DevOps is a Culture and it is in all of us
- 52. Everything is continuous … including challenges ;-) • Exploring capabilities of TFS in diff areas • Multiple platforms support for development teams • How cloud fit in this whole picture • Exploring options for information radiators • Exploring options end to end orchestration • Promote “culture of reuse” for our chef recipes
- 53. Training Within • Cross pollination (other wants to learn) • Internal Operational Manuals for other teams • AWS Study Group and Certification
- 54. One Team Dev System Admin SCM Network QA OPS Business Dev System Admin SCM Network QA Ops Business
- 55. • Raj Makkar – [email protected] – LinkedIn: https://www.linkedin.com/in/rajwinder-singh-83166718 • Richard Fong – [email protected] – LinkedIn: https://www.linkedin.com/in/richardfong1