DevOps or No DevOps? The Journey to be Fast, Accurate and Flexible!

This article is created to provide a high level overview of some of the challenges that do exist when implementing a DevOps structure. As always, there are many ways to “skin the cat”, and this document is not to be seen as the ultimate solution to all of the problems that may arise implementing a DevOps structure.

This article describes one possible approach to the problems, an approach that just might work for our company, but might not work for others. Some of you might even think to overcome the problems in a different way, but that is all fine. The purpose of this article is to provide some insights in the thought process to align strategy and implementation of the strategy within our company, and still be Agile, Accurate and Flexible, so the business can be - and will be - served in the best possible way.

What do we mean with these buzz-words?

Agile: Try fast, fail fast. Don’t be a simple bullet trying to hit the target in a straight line, but be a missile, that can be guided along the way to reach it’s final destination.

Accurate: Deliver fast, but deliver with Quality, without any exceptions.

Flexible: Don’t get yourself caught in an environment, where people can’t be missed. Rely on Technology and processes and build in your QA points where needed.

Although we are not at our destination yet, the process in this article might even become outdated soon. As mentioned, it’s the thought process that counts, and while thinking and implementing to come up with some solutions to problems that might arise….

1 DevOps… What does it mean?

Just like many other businesses, our part of the company wants to have the shortest development timelines possible, in order to serve our customers in the best possible, and in the fastest way. Many companies are therefore switching into a – so called – DevOps model. But what does that actually mean?

With DevOps, the end-to-end responsibility of specific functionality sits with a person: the DevOp. It’s not only a responsibility for development effort, but also to keep the functionality up-and-running as good as possible in production. In case issues arise, the DevOp will action this based on information he is able to gather him himself. Monitoring of the service in Production is therefore also one of the tasks of the DevOp.

1.1      Does this mean the DevOp needs access to Production?

Many people therefore ask the question, if a DevOp needs access to the production estate. This is never the case. Tools should be implemented to support the DevOp with necessary production information. Tools to perform the Monitoring of the Service, but also tools to analyze the Log files. Although there is a variety of tools available to do the monitoring and log file analyses, many of the companies use a tool like Splunk for Log file analyses, the monitoring often is dependent on the system to be monitored.

1.2      How does it then work?

Based on the tools provided, the DevOp is capable of understanding whether or not the service is running as it should, or if there is an issue with it. When there is an issue, the DevOp will revert to the Dev/Test/Sandbox environment to replicate, and fix the underlying cause of the issue. Once fixed, the DevOp is supported with a Continuous Delivery model, consisting of a Continuous Development (CD), Continuous Testing (CT) and Continuous Integration (CI) ‘street’, to get the development pushed to specific environments as quickly and accurately as possible.

2 How does the process look like?

In essence, the DevOp follows the following stages:

Each of the steps have their own set of tools available to support the DevOp to perform his tasks best.

Monitoring:

• Monitoring tools for the application;
• Incident management tool, for customers to raise issues

Analyse:

• Log file analyse tool, so no entry to production is needed

Adjust:

• A development environment, so the issue can be fixed
• Source code management system, to make sure the right source code is adjusted

Test:

• An automated testing environment, so all services are continuously tested when new work or fixes are being pushed
• Functional testing and approval by the customer, in order to have the functionality developed or fixed approved by the customer, before pushing to the production environment
• Integration testing done automatically, again the outcome should be approved (if applicable)

Deliver:

• The delivery phase consists of updating documentation and actual scheduling of the automated push to production

Many more tools and processes can be used during these phases, but this gives a good high level overview of the tasks and automation put in place, to let the DevOp perform his tasks with high quality and speed.

3 This all sounds good, but….

3.1      what are the drawbacks?

Well, this all does sound good, but there are indeed a few drawbacks in this approach:

1. The Functionality is bound to a Person. Knowledge transfer hardly ever happen, and the documentation is often not complete. The DevOp does have a good understanding of the functionality, and therefore the documentation is often lacking important pieces of information
2. Outsourcing the DevOp structure makes the company dependent on just 1 Service Integrator (SI). Switching SI is often very hard, not the least because of the documentation issue mentioned in bullet-point 1.
3.  Leave/sickness coverage is hard
4. Costs will increase, as the SI knows the company is dependent on them in order to fulfil the work needed.
5. Many many more….

3.2      Can we overcome the drawbacks?

From my perspective, aligning strategy with execution: yes. It’s a journey, not a simple implementation step, but the journey will keep the best of all worlds, and solve problems that we face (or will be facing).

The next chapter will give some insights in the steps taken, to overcome the drawbacks mentioned in the previous paragraph.

4 What steps to take to eliminate these drawbacks?

4.1      Simple process adjustment

The first step would be to adjust the process and build in an additional step around documentation: Document the problem.

This documentation will help understand the issues, and should be able to provide you – at a later stage – some insights in where the problem areas are, and how they can be improved moving forward.

4.2      How does this help?

Many companies, as does ours, want to use some kind of Agile methodology to do the actual development and bug fixing. Building in this Documentation step allows you to just follow that methodology and implement something like Scrum or any other Agile framework.

Based on the implementation of this step, we can now actually action the work in a slightly different way: the documented issue/task can now be put in a backlog and be prioritized appropriately. Some issues might need a higher priority than others, and using this approach, would reduce the interruption of some of the work for the DevOp while working on a specific task. It also provides good insights in the workload that does exists for a specific task and what specific problem areas might be of the service you are delivering.

4.3      What else does this show?

Just implementing this additional step to document any issue with the service, immediately shows there is a handoff moment possible between the Monitoring, Analysing and Documenting part of the DevOp work, and the actual building/fixing and delivery of the work. Often, these two areas are referred to as Run and Build.

Run:    Monitor the execution of the service and, in case of an issue, analyze the issue and document it properly.

Build:  Based on proper prioritization, work on the tasks to be delivered.

We now have reached the Agility point, with proper prioritisation and owner ship. But we have reached only partially the accuracy side of the DevOps structure….

4.4      So, how about Accuracy?

As outlined earlier, accuracy is achieved via proper documentation, development, but most of all, building in the right steps for testing. Testing, not only to be done by the developer, but also preferably by the end-user. Following the correct steps of the Software Development Cycle (SDC), testing needs to be done in different stages:

• Unit testing; Testing performed by the developer when coding is completed. It is suppose to test the units the developer has adjusted.
• Regression testing; Testing performed to find defects. This testing phase should cover the whole application, as there might have been gaps and it might be undoable for the Developer to test everything, especially with commonly used modules/procedures.
• Acceptance testing; Testing performed by the Owner (of the application, in conjunction with the owner of the issue and the reporter of the issue)
• Integration testing (when applicable); Testing to see if the whole service is still functioning, covering often more than just one application.

Although there are many more ways of testing, at least the above should be done to guarantee some level of quality for the end user, and to get the end users approval to deploy to production.

4.5      Doesn’t this affect “the speed” in solving the issue?

The speed to deliver is definitely affected by the testing phase. Often, testing can be done quickly, but more often, especially in big and major systems, the testing will impact the delivery timelines as the amount of tests to be executed can be quite large.

The DevOp structure can be supported in these via automation. The majority of tasks can be automated to make sure the speed of delivery is less impacted and the same level of Quality is achieved.

Automation should therefore be applied in the following areas:

1. Source code management; SCM is a definite need. Not only to make sure the right sources are being used while fixing an issue, it mostly also allows branching and merging fixes into the right branch.
2. Automated deployments; Deployment Automation will support the DevOp in not being focused on tasks he should not be focusing on, like merging into the right branch, and delivering the code to the specific environment.
3. Automated Unit Tests; Frequently, the Unit tests already exist. Ideally, the Unit tests are performed automatically before delivering the code to the desired environment. In case the Unit tests are failing, the DevOp can fix the issues, and just check-in the code with the SCM used. The chain is then triggered from start again.
4. Automated Regression Tests; Once a piece of code is delivered to the testing environment, automated Regression test should be executed. More often, this is a continuous process that runs a few times each day, and the results of these tests are evaluated immediately, in case of an issue. Fixes should – in case of an issue – be applied immediately, to keep the test environment running as smoothly as possible.

Once these steps have completed successfully, the Automated deployment functionality will be able to merge and deploy the changes into a UAT environment, which is pretty much in line with production. In case of frequent changes, automated regression testing can also be triggered on this environment. In all cases, the product owner, the Issue Owner and preferably the Customer will now test the functionality. In case Scrum or any other framework has been followed, these testers are already aware of the functionality changes and possibly screen changes (via interaction and review sessions), and should not be surprised by any of these changes (if applicable). This testing can be completed quickly.

During this phase, the DevOp can now adjust the documentation (if not part of any Definition of Done) now, and create proper documentation to be handed over to Run. If not immediate, scheduled deployment of the fix can now be scheduled for automatic deployment, or triggered in case of an emergency fix.

Based on the previous steps, we now have documentation in the right places, and quality is assured. This completes the Agility and Accuracy part of the work.

5 All of this is nice, but we wanted to be flexible as well!

Indeed. Flexibility was also one of the main objectives. Although some flexibility was already achieved with the prioritisation of the work, there is another level of flexibility we want to achieve: Become less dependent on the DevOp himself.

As shown in the previous chapter, Build and Run can be seen as something separate in the work of the DevOp. What if we would separate them completely? What if Build and Run where handled by different people?

This change perfectly ties in the Operating Model we want to achieve: Having a Service manager to be responsible for the service delivery, and have Centre Of Enablements (COEs) in specific areas (or just 1 COE if possible) being responsible for the Build part.

Separating these areas of responsibility will make sure there are hand-over moments. These hand-over moments will make sure all necessary documentation is properly evaluated, which makes sure there is less dependency on the Developer.

Another benefit in this approach is, that all the work can easily be outsourced to 2 independent SIs. Keeping the SIs independent will make sure the Quality Assurance (QA) step - on hand-over moments - is kept in place. Having 2 SIs also brings the benefit that you can swap them more easy, if not fulfilling your needs or making the agreed Service Level Agreements (SLAs).

Next to this, it’s easier to find an SI to provide 24/7 coverage for the RUN part, than to have an SI that does 24/7 support for the whole DevOps chain. This is another advantage that this model brings.

6 This does overlay perfectly with our Operating Model?

Yes. Our model – simplified – can be seen as:

Service management is fully responsible for the SLA to the business and following up with the different application teams to meet the SLAs for each of the defined services. A service, obviously, can touch more than just one application. The Service layer itself is not represented in this picture.

The Centre of Excellence(s) (COE) are seated within each application team, and is responsible for the development of the application. According to Scrum, and the fact that we as a company are service oriented, each service will have a service owner. These can be seen as the ‘Product Owner’ for the service.

Based on this model, Run and Build are separated. This ties in perfectly with the structure described in this document, and supports the following split:

7 Conclusion

What we now have seen in this structure is that the used operating model within our company and the usages of different SIs works perfectly well. But what are the exact benefits in this approach?

1. Clear split between Run and Build
2. Agility achieved, clear defined backlog of tasks
3. Automation in place to support a “DevOps” structure on Build and provide accuracy where needed
4. Clear responsibility and tracking of SLAs, to serve our customers
5. Flexibility achieved

And now, yes, we do meet all our initial requirements. The structure matches our organisation, and the DevOps setup is achieved through Automation, rather than end-to-end responsibility for the DevOp.

So, is this a DevOp structure? In essence: No. The Dev doesn’t do the OP stuff, but the DevOps way of working is adopted. It’s more of ‘the best of all worlds’ that might just work very well for your company as well…..

Note: this article is far from complete in many areas. It's intention is to give an overview of how some elements of DevOps can be used in a None DevOps Operating Model. Is this the single truth? No. It's not, and should not be seen as such...