Download to read offline
Reflections on18monthfederaldevopstransformation2015
- 1. Reflections on an18-Month Federal DevOps Transformation August 4, 2015 Dan Craig Director, Agile Services Agile2015
- 2. A Bit AboutMe… • 2001 Started working agile projects – Exclusively commercial sector – Held almost every role, but project management fit best • 2009 First DevOps transformation work – First work with Federal Government (DISA) – Served as agile & process re-engineering SME • 2013 Started my current engagement – Large, Highly-Visible Office in Commerce Department – Solidify agile practices; initiate release automation
- 3. Obligatory DevOps Definition •DevOps is the natural extension of 15 years of agile thinking • DevOps is primarily about: – Communication/Collaboration – Automation • Advent of tools enables organizations to become DevOps: – Cloud Technology – Puppet / Ansible / Chef
- 4. A Great Picture •Businesses Demand Change – Fast time to market – Innovation – Differentiation • Businesses Need Stability – Perfect up-Time – High quality – Bullet-proof security These two demands are necessary, but cause conflict using traditional delivery mechanisms!
- 5. DevOps in FederalSector
- 6. The Organization August 13,2015 6 • Large Federal Office • Forward-thinking CIO • commercial background • history of innovation • Leadership that already worked well with each other • 170 Internal/External Systems
- 7. The Situation • Becamevisibile at height of Recession • Congressional funding of NextGen • Delays in NextGen delivery • Visible production outages • Issues – Manual CM – No Automation
- 8. The Team • SmallDigital Services team – 7 cross-functional team members – 1 Government PM • Strong Leadership Support • Access to All Layers of Organization • Flexible Contract Vehicle!
- 9. Initial Engagement Objectives •Create CICM Platform – Leverage 100% Open Source Software – Enable external user access • Migrate 3 NextGen Programs onto CICM – Institute CM – Automated builds – Automated Deploys – Nightly testing • Support Agile Practices – Training/Mentoring – Documented best practices • Prepare for Rollout to all NextGen Our Plan • Months 1-3 • setup CICM • migrate systems • Months 4-6 • document learning • “harden” mechanisms
- 10. Our Planned Approachto DevOps Transformation Continuous Delivery DevOps Practices Application Release Automation (ARA) Agile Development Configuration Management We want to get here > then solidify things!
- 11. The Initial Platform August13, 2015 11 Jenkins Continuous Integration (CI) tool that automates and manages building and testing of software Subversion Version control system that manages files and directories and tracks changes made to them over time. Nexus Sonatype Repository Management tool stores binary software components used during development, build and deployment SonarQube Reporting Dashboard that publishes Continuous Inspection metrics gathered during build and analysis of software
- 12. How the PlatformWorked Delivery Teams CICM Platform SIT FQT PVT Publish Commit Deploy Shell Test Selenium PROD Checkout NextGen Dependencies Artifacts
- 13. Early Transformation • Migrated3 large programs • Modernized CM practices • Automated builds & SIT deployments • Proof-of-concept Selenium testing • Focused teams on Sonar metrics • Passed several internal audits • Established initial development standards • Starting include more organizations in the effort Showed the organization that this was actually possible. Confidence was building! August 13, 2015 13
- 14. Early Problems • Notwo projects looked the same • Operations not participating in automation • Deployments not addressing middleware, system or DB • Test organization in “analysis paralysis” on test tools • Audit organization not convinced of robust traceability • Procurement saying automation demands out-of-scope August 13, 2015 14
- 15. The “Perfect Storm”Upsets our Path • Increasing rumors of “silver bullet” solution on CICM project • Two upgrade-related outages on critical Legacy system costing millions • Demands from business for more focus on Legacy vice NextGen August 13, 2015 15 • Pressure from Congress to move faster on NextGen • CIO desire to eliminate license costs
- 16. A Critical DecisionPoint… • Option #1 - Stay the Course – Continue focus on NextGen – Spend 3-6 months “hardening gains” – Preparing for enterprise use - OR – • Option #2 - Full Steam Ahead – Expand focus to include Legacy – Defer “hardening” in favor of onboarding – “Build the plane as we fly” August 13, 2015 16 What do you think they chose to do?
- 17. Going into Battle! Newobjectives included: • Begin migration of all projects onto new platform • Assess their viability for automation • Ensure limited deployment- related issues • Fix problems with testing • Sunset IBM Rational Suite • Support new operations puppet initiative August 13, 2015 17 Oh Yeah, and… No additional staff!!
- 18. Enhance & Extendthe Platform Delivery Teams CICM Platform SIT FQT PVT Publish Commit Deploy Ansible Test Selenium, TestComplete, SoapUI, LoadRunner, WebInspect PROD 50+ Jenkins Slaves Checkout OSS ALM Admin Features NextGen Legacy • Automated Upgrades • Jenkins Templates • LDAP Groups • Labels
- 19. Our Ansible Decision Ansiblebest suited our need for application deployments: • Agentless • Intended for non-Developers/System Admins • Shallower learning curve • Provides full Orchestration • Clean Division of Interests: – Engineering maintains Playbook (the “what”) – PSB maintains Inventory (the “where”) – CICM maintains Platform (the “how”)
- 20. Creating the Transformation“Playbook” How to handle flood of projects: • Formalize platform processes • Sure-up agile processes • Align organization • Get the word out
- 21. Focus on Process •Migration to the platform – Vetting Projects for “fit” – On-boarding • Administration of Platform – Enhancement Requests – Frequency of Deployments • Knowledge transfer – Audits to ensure continued maturity – Internal “certification” August 13, 2015 21
- 22. Fine-Tune the ReleaseProcess • Introduced Bundling – “Build Once – Deploy Anywhere” (BODA) – All config defined up front • Orchestration of non-automated components – DB – Middleware – System • Defined metrics to gather & retention policy August 13, 2015 22
- 23. Sure-up Agile Practices •Mandate continuous integration • Report on & review Sonar metrics • Focus on Testing – Unit – Functional • Tailored best practices
- 24. Align the Organization •Vision alignment – Quarterly leadership – Monthly stakeholder – Weekly platform team • Metrics-based reviews to reinforce organizational vision • Project maturity scale • Updating requirements for new procurements August 13, 2015 24
- 25. Get the WordOut • Marketing Campaigns – Newsletter – Platform Knowledge Center – DevOps Days etc. • Education – Brown Bags – Classroom & Online Training – Targeted Audit Reviews • Procurement – New Demands on Contractors – Measurable Results August 13, 2015 25
- 26. Starting to MakeSerious Impact August 13, 2015 26 • 70+ projects migrated to the Platform – All portfolios represented – Heterogeneous technologies • Clear Transition from Pilot to Enterprise – 1500+ Automated Builds Daily – 10 Projects with Ansible Deployments • No Reported Issues with CM or Deployments from Platform • Cutting costs moving onto Open Source Platform But Operations Still Not Comfortable with Automated Deployments to Post- Development Environments!
- 27. A Second PerfectStorm?? • Legacy outage forces Operations to recover using Ansible • First ever “organic” DevOps deployment occurs • Growing pressure from Business and Engineering to leverage new automation • Gene Kim and other thought leaders come speak about taking risks in DevOps Permission granted to automated deployments to production!
- 28. Its been 1Year Since That Breakthrough… • All 170 Active Systems Under CM • Most popular technologies supported • Standards & best practices defined • Application Release Automation (ARA) in place Production deployments aren’t continuous – but they are common! • DevOps Culture taking root • Production outages increasingly rare • Organization nearly able to staff CICM roles as O&M not high-end consulting
- 29. What We AreWorking on Now Platform – Introducing tools to mature platform – Introduction of a Pipeline View – Central Administration – Self-provisioning of cloud resources – Audit reporting – CMDB
- 30. What We AreWorking on Now • People – Work with Procurement on Agile contracting – Continued training; focus on Government employees – Focused mentoring specific to functional test quality & regression • Process – Reinventing release process • Orchestrate hybrid Puppet & Ansible deployments • Addressing database – Institutionalize metrics-based reviews (“virtual ORR”) – Satifsy audit & oversight requirements automatically – Determine “how far” we want to go towards CD
- 31. What You ShouldExpect on Your DevOps Journey • DevOps Is Hard! – Particularly hard in Federal sector – Takes vision, courage & endurance – You’ll have to be a salesman – Everybody is bought in…until it impacts them • You will have problems with: – Automated Testing – Getting Operations to take that first leap • Contracts & outside SMEs can help, but nothing works without strong, consistent support of leadership • You need the proper foundation before DevOps works • What works today will NOT work tomorrow…Plan for it. • People, Process & Tools must mature in Lockstep • ARA with good DevOps may be the finish line for Federal Enterprises
- 32.
Editor's Notes
- #3 Just so you know who you are listening to: been working on commercial agile projects since the early 2000s. - In fact, In 2000 I left AOL to join a small startup and live live “firsthand” the new light-weight process I had heard about (XP) - I had worked on many projects before I came across books like “extreme programming explained” by Kent Beck and “Agile Software Development” by Robert Martin - Exited to see like-minded people in the industry and it began to put a name and framework around what we were doing. - I saw the community grow and find it exciting that it now is well accepted everywhere (nearly) In 2009 I got a call from a friend who had just started a company called Steel Thread. He wanted some help at DISA to do some cutting edge automated deployments - It was here that I first became acclimated with tools such as puppet and began working towards DevOps practices (though I didn’t know the term at the time) - In 2010 the Continuous Delivery book by Jez Humble/David Farley came out and put a name to what we were trying to do. I felt the same feeling as I did 10 years earlier reading Beck and Martin. I knew I had found the next major shift in how IT should work. In 2013 I accepted a role to transform a large, innovative office within Commerce Department - Not a professional speaker you would see on these types of circuits, but I thought it worthwhile to speak about my learning as it is 1) large enterprise and 2) Federal Sector
- #4 Before I jump in, Everyone loves to give their definition of DevOps these days and I’m no different! In General, I believe DevOps is about COMMUNICATION & AUTOMATION. Devops is a concept that shines a light on communication issues much the same as agile did in the early 2000’s. Back then, the effort was to get the business talking with the Development/testing team. Today we are trying to bridge the communication gap on that “last mile” between Dev and Ops. Just as with the agile movement, I think that the rise of certain tools and technologies has greatly aided this communication. While the appearance and maturation of tools such as Hudson and Subversion, certainly helped bridge the gap between business-development-test and allowed a metrics based conversation on agile teams, so too is the advent of Cloud Technologies, Puppet, Ansible giving a common language to our DevOps discussion.
- #5 But what IS the communication problem? This is a Great picture I saw in an article by Niek Bartholomeus. Historically, Business owners push development teams for innovation, speed and agility. Revenue and upside depends upon this. Years of pressure to move fast and deliver is baked into the DNA of development teams They also require stability and quality. 5 9’s, no security breaches, no risk or embarassing hacks. Security and stability are mandatory. Years of pressure to move with caution and protect assets is baked into ops teams If I were to whisper in the horses ear deliver fast and tell the other move stable and safe. What happens when I say GO? This is what we do with our businesses and its only through great effort and heros that things stay afloat. DevOps – to me – is an acknowledgement of these contrary demands for stability and speed. It’s understanding that the only way you can get both is to get teams heading in the same direction and at the same cadence. Things must be predictable and repeatable. I’ve come to believe it is even harder in the federal world though….
- #6 Though they share the same name “DevOps”, the landscape of the Federal Government makes growing a DevOps culture a very different problem They have all the problems of a commercial organization: * Finding quality staff (unicorns) * Breaking down silos Years of finger pointing creating a “great divide” between dev and ops Concerns over control, security & compliance But the Federal Government – or at least where I have served -- has some unique issues that make it a very different problem: Federal Procurement Cycle Contracted Staff (stuck in roles, transient, self-serving) Unions amplify resistence from disbelievers layers of oversight Shifting administrations and funding priorities People Contracted staff locked into long term contracts Low labor rates = body shop vs. unicorns Contractors can often act as a bottleneck or hamper devops initiatives as they often focus on profitability and “protect their turf” as aggressively as any employee Labor Unions often support cries from employees who disagree with the changes and the new roles that they bring Regular shifts in administration brings unpredictable funding for large, innovative initiatives. Process Procurement Law Layers of Oversight (e.g., office, agency, DHS, OMB…) Tools Antiquated Systems (like a walk through technology museum) Licensing vendor “lock” Slow adoption of Cloud Technologies SO the main takeaway from this talk is DevOps is hard anywhere…but particularly in the Federal Government. Even under the best of cases. My current engagement –- by my estimation – IS the best possible case and we’ve learned PLENTY of lessos. I want to take you on a walk through our first 18 months at our Federal client. Describe our approach and highlight the bumps we had along the way. I think there is value to conversations like this, so we in the IT world can gain the same “pattern recgnition” we gained in the early days of agile. We can help evolve a DevOps concept which is now an artform…into something that has commonly accepted standards and best practices.
- #7 The engagement occurred at a very visible office in the Commerce Department Particularly visible at the height of the Recession when Obama administration named it as a key to getting businesses moving again Quite a bit of new funding (and Congressional oversight!) went that direction to create the next generation of customer facing and internal applications The CIO was very forward thinking and had experience with modern tools and technologies in the commercial sector 170 systems in some form of active development - 140 legacy applications with a very wide variety of technologies and platforms - 30 NextGen applications fairly homogenized to Java/Maven/Jboss, but quite a few differences in the middleware utilized
- #8 So what was the situation before we engaged? Initial small contract to do an assessment and to develop a 3 year road map for continuous delivery When presented to the CIO, he said “this is nice”, but how are you going to stop my hemorrhaging NOW. So here was the situation… not all that uncommon in the federal space – Main driver - Highly Visible Legacy Blow-Ups (CM, Deployment). Took days to get back online. Couldn’t immediately reproduce it. No chaos monkey to test with…. yet Pressure to Deliver Suite of NextGen Applications – Legacy modernization initiatives Manual infrastructure configuration Various methodologies/technologies No central source for CM, Dev, Test or Deployment of Systems Silo’d organization – conflicting goals and performance objectives – very important . I’ll talk more about this later In terms of process maturity… Some nextgen development teams were “closet using” Jenkins and SVN, but most had never used those tools Some Scrum with waterfall testing and release processes – infrequent releases 1x to 2x per year. Ops teams generally supportive - has help fast track processes to get us the resources we need A lot of infrastructure variances do to manual processes
- #9 Established a small Digital Services team – CICM 7 Steel Thread SMEs of various talents – PM, Architect Sys Admin, Test Automation, Agile Coach, 2 automation engineers 1 Govt PM and our Tack Order Manager. These two roles to proved absolutely critical to our success. Must stress the role of a strong, seasoned PM in the early days. Ours was fantastic at navigating the existing organization and process and wasn’t afraid to ““break some dishes” as she puts it. While our team has grown a bit since the initial pilot – both on Steel Thread and Govy side – ths initial small size really worked Also must stress the need for High level of executive buy-in and attention. In fact, in early days we had regular (semi-weekly) meetings with executives to discuss organizational, policy, process roadblocks. During these meetings we gave them 1-2 “homework” assignments (clearing the path) and alerted them to which organizations were about to complain (based on what we were doing) A quick note on the initial contract engagement: We had very good collaboration going into the contract award. Not knowing exactly what the outcomes would be at the start, the contract was written in such a way that allowed us to pivot, but also provided objective results based on continuous planning with the Govt stakeholders. Even though this was first attempt at this type of contract, it worked quite well. Contract was for 2 years. Ended up doing completing in a little over 1 year Phase 1 – 1 year – pilot several projects Phase 2 – 1 year – onboard all nextgen and some number of legacy
- #10 3 projects – 1 mistake was the selection of 3 of the largest projects from the most visible portfolios ~15 projects/~50 components One project drove the financial interaction with general public so particularly sensitive to audit and traceability pilot mentality agile development with nightly pushes to SIT for automated testing platform to be fully open source access to outside CI – establish the automation and then increase build frequency and unit test coverage CD – move testing to the left and increase functional test coverage, perform nightly regression, load and security testing Team enablement - so the theme was technology was needed, but it’s not a tool problem. We had to change the way developer and testers and PMs for that matter worked. We had very good collaboration going into the contract award. Not knowing exactly what the outcomes would be at the start, the contract was written in such a way that allowed us to pivot, but also provided objective results based on continuous planning with the Govt stakeholders. Even though this was first attempt at this type of contract, it worked quite well.
- #11 Our approach to DevOps & CD is seeped in our background as software developers configuration management - common source management - standardization around commit, branching and tagging practices Agile Development - SCRUM Management framework and XP developer practices - Build Automation > Continuous Integration & Inspection Application Release Automation (ARA) - packaging/bundling for deployment - Shift testing and deployment to the left - release to post-development environment (system, middleware, application, db) DevOps Practices - it takes cycles to get good at DevOps - With initial ARA in place, can really exercise teamwork and communication required. Continuous Delivery Requires extremely mature processes and tools We do this in our in house development on very small teams, but very different at scale True CD (e.g., release trains, features toggles and automation all the way to production) is rarely the goal within Government organizations due to policy and oversight restrictions. This may change, but it doesn’t appear to have changed yet.
- #12 Subversion considered GIT, but too advanced wanted to standardize entire organization Jenkins Optimized for Java/Maven via plugin To be leveraged for builds, deploys and tests Nexus Manage dependencies & store build artifacts Follow Maven build process using Snapshot & Release Repos Sonar Custom dashboard highlighting a few important metrics (unit test, SA) Leverage out-of-box quality profile All used LDAP for authentication/Access Control All managed individually – would turn into a nightmare later!
- #13 Explain Process Deployments were simple SSH In using Jenkins System User with access to all SIT boxes Deployments Application Only; no middleware, System or DB Tests were written by our team as example of how they should work No attention yet to performance, security, api or other forms of testing Ops Disallowing deployments to post-SIT environments (even though we had access to do it)
- #14 There was a lot of good news and our stock was high in the organization.
- #15 no two projects looked the same Each with nuanced differences in their POM structure causing different build Two using custom shell deploys; one using a jenkins plugin approach One respecting the “BODA” concept, but the other two requiring distinct builds per environment Branching and Tagging using different convention Operations Shell scripts not transferable between projects …also not transferable across environments Didn’t want liability of automation Test Govies not deep in testing; esp tools Over 130 contracted testers; but little automation experience In assessment of various licensed tools not well suited for CD Automation (looking for easiest one) Audit didn’t like “Jenkins System User” concept procurement some teams complaining about burden; oos of existing contracts Though there were LOTS of issues to work through; we have months to get things rolling before more projects migrate to the platform. But…
- #16 Silver bullet Lots of myth in halls spent ½ time selling; ½ time talking down euphoria CIO on outage can cicm handle it?
- #17 At the end of the pilot initiative, we had learned a lot and had many opportunities to close gaps At that point, howeve,r there was a convergence of several events that shifted our direction For better or worse, we needed to take on Legacy Project
- #18 Not exactly true, they allowed us to claim victory on our 1 year goal and pull some funding in from year 2. We also got help from legacy CM team who moved projects across for us.
- #19 Introduced & Integrated ALM Capabilities Rally for NextGen teams Open Source ALM for Legacy Teams Integrated with Subversion/GIT to provide traceability of commits to stories/requirements Expanded SCM Capabilities Added GIT to support Puppet Based Dev activities of Operations teams Added SCM Manager to provide convenient RBAC and Administrative control Extended & Improved Jenkins & Sonar Support additional technologies (.Net, NodeJS, C#, Javascript, PHP) via plugin Added concept of Portfolios + Projects to improve user experience and maintainability Added over 50 Jenkins slaves to serve traffic + execute TestComplete/LoadRunner etc which not well suited for CI Automation Retrofit existing deploys to leverage Ansible Shell not scaling and no easy standard Puppet required coding apptitude and not easily understood by Ops team Ansible: Agentless Orchestration right out of the box Recognized division of labor (ops created inventory file, engineering creates playbook) Still leveraging Puppet for system deployments Added Custom Administrative features Monitoring Labeling & automatic deployment of slaves using swarm plugin Move towards LDAP groups for scale scans for standards on job definition Still not allowed to deploy past SIT due to discomfort with the concept of automation!!!
- #21 With the decision made to We had spent time with mature teams and focused primarily on tools. It was time to get serious about the People and Processes Involved with this major culture shift When we started, about 10 total people know what CICM was, so we had a huge sales and marketing effort to do. So lot’s of meeting at all levels from project teams to executives. This was an iterative process because some of these folks liked the idea of automation but had no idea what we were talking about. Next was the onboarding processes. With a focus just on NextGen teams, the first couple team were relatively easy. We still had to work around project schedules, but they were already closet building with Jenkins so it was just a matter of moving code and getting them up and running and tweaking processes. Others, obviously needed some work, Then we started to migrate legacy systems. These were just a lot of work to get there code in a state that’s CI friendly. “buildable” by Jenkins. So there was some refactoring that had to be done first. So if you’ve ever worked in this type of environment, telling a team they have to refactor their code is like calling their baby ugly. Refactoring was a bad word. We got some bumps and bruises until we got our way. We have since move past that and now it’s beginning to be part of the process, even though technical debt isn’t generally managed well yet. It’s funny, we didn’t think anything about the sensitivity. We refactor all the time, probably to the point where we don’t like we know what we’re doing to the untrained eye. both technology and processes. So we assumed all agile teams did.
- #22 We had spent time with mature teams and focused primarily on tools. It was time to get serious about the People and Processes Involved with this major culture shift Co-Hosted Migration Kicoff meetings (eng, ops, cicm) all negotiated with Process & Policy teams to ensure in-line with other parts of the organization
- #23 Initially we met with the Nextgen teams to understand the architecture and tech stacks. From here we began to develop the platform backlog of required plugins and other capabilities. From here we developed roughly objectives to get alignment with the stakeholders and to get alignment with other teams. This included both platform and project priorities. Then we do weekly planning where we task out the objectives. Additionally, we hold seimi-annual “summits” for the CIO and his direct reports. The goal of these meetings were to provide an update on progress and to elevate and organization blockers. This venue has actually been quite effective and engaging with the CIO. The next part of getting alignment is Communication. First obviously is our kanban wall in the team room. This display all projects in flight on a 4x6 card, what stages they are , as well as some meta data. This has been quite useful when engagement with new teams and management. There has also been an adoption of the kanban walls in peoples offices now as well. The other big communication mechanism is the bi-weekly news letter. This turned out to used as a status report and stick for the teams onboarding. In it, we show highlights, new platform capabilities. We also include the status of each project that has onboarded or is in the process of onboarding. Essentially a visual of our kanban wall. This turned out to be the bible on how executives evaluated to team were adopting the new ways of doing things. You’d think it real time data is provided in a dashboard automatically, people would strive to use it. It was actually very difficult to get management and teams to look at Sonar! Dev teams didn’t necessarily like it because it objectively showed the health of their application, especially unit test coverage. Ultimately, I think it’s just a little bit to far in the weeds for management outside of unit test coverage. That said it’s becoming a measure of performance for the teams and some teams do embrace it for the fast feedback value it provides. Good stuff. We have had a continual stream of projects onboarding for nearly 18 months, of all types of technologies. If you’re familiar with the open source tools used, it’s very maven centric. So getting applications to begin using the platform standards is very demanding and time consuming. However, we do now have better enforcement of technologies required by the teams. The benefit is now auditing is simplified because we know exactly how applications are build and deployed. We now have nearly all applications on the platform and can begin to retire some of the legacy ALM tools.
- #24 Even the rogue team using Jenkins really did not practice CI. The level of agile maturity was all over the place. So we needed to get a baseline of at least what were considered the most mature teams. So we did some agile assessments and provided reports and recommendations. We created a formal raining curriculum and put over 400 students through it. This also helped with the PR campaign at the developer and tech lead level. And gave them a change to ask questions. With that, we introduced XP practices, which CI is part of. It was a foreign concept for most to commit and build frequently. AND then to have the dev quality results in Sonar. Enforcement of standards. So obviously automation provides a high level of enforcement. However, giving teams the ability to manage and create their own Jenkins jobs has proved interesting. So quick story. I ran into a Portfolio Manager one day in the cafeteria and asked why all the unit test results in Sonar were grey? I don’t know I said I’ll check. Here to find out, the unit tests were and they didn’t want them to be exposed in Sonar a Red. So we enabled them and the team turned them off. A few times. So it’s a balance of keeping the teams moving forward and enforcing the best practices . ------------ Refactoring Initially a four-letter word Dependency management Externalizing environment variables for BODA Deployable units – release dependencies Accounting for technical debt No time for technical debt, just new features Testing Paradigm shift for traditional test organizations Must make scripts CI friendly If code changes it must be in SVN and built by jenkins. No direct uploading nexus for locally modified code
- #25 Initially we met with the Nextgen teams to understand the architecture and tech stacks. From here we began to develop the platform backlog of required plugins and other capabilities. From here we developed roughly objectives to get alignment with the stakeholders and to get alignment with other teams. This included both platform and project priorities. Then we do weekly planning where we task out the objectives. Additionally, we hold seimi-annual “summits” for the CIO and his direct reports. The goal of these meetings were to provide an update on progress and to elevate and organization blockers. This venue has actually been quite effective and engaging with the CIO. The next part of getting alignment is Communication. First obviously is our kanban wall in the team room. This display all projects in flight on a 4x6 card, what stages they are , as well as some meta data. This has been quite useful when engagement with new teams and management. There has also been an adoption of the kanban walls in peoples offices now as well. The other big communication mechanism is the bi-weekly news letter. This turned out to used as a status report and stick for the teams onboarding. In it, we show highlights, new platform capabilities. We also include the status of each project that has onboarded or is in the process of onboarding. Essentially a visual of our kanban wall. This turned out to be the bible on how executives evaluated to team were adopting the new ways of doing things. You’d think it real time data is provided in a dashboard automatically, people would strive to use it. It was actually very difficult to get management and teams to look at Sonar! Dev teams didn’t necessarily like it because it objectively showed the health of their application, especially unit test coverage. Ultimately, I think it’s just a little bit to far in the weeds for management outside of unit test coverage. That said it’s becoming a measure of performance for the teams and some teams do embrace it for the fast feedback value it provides. Good stuff. We have had a continual stream of projects onboarding for nearly 18 months, of all types of technologies. If you’re familiar with the open source tools used, it’s very maven centric. So getting applications to begin using the platform standards is very demanding and time consuming. However, we do now have better enforcement of technologies required by the teams. The benefit is now auditing is simplified because we know exactly how applications are build and deployed. We now have nearly all applications on the platform and can begin to retire some of the legacy ALM tools.
- #26 We had spent time with mature teams and focused primarily on tools. It was time to get serious about the People and Processes Involved with this major culture shift
- #27 There was a lot of good news, but I don’t really want to focus on that. We had: - Implemented a platform that gave everyone access (internal and external) - Moved 3 largest projects onto a common CM environment - Automated builds - Proven automated deployments were possible - Given POC around Selenium and other testing tools Brought a sense of the possible to many parts of the organization. Buisiness as usual for other parts Major issues at this point: Operations wouldn’t let us do deployments to post development environments Testing still not nearly at the level required for full automated releases
- #30 Platform Now that we are clearly out of pilot, customer is willing to spend a bit on license fees. Tools like cloudbees and NexusPro will allow us to
- #31 Platform Now that we are clearly out of pilot, customer is willing to spend a bit on license fees. Tools like cloudbees and NexusPro will allow us to
- #32 DevOps You can not buy it (services or Products) You are never done You’ll need a thick skin