Skip to main content

Immaturity is no excuse: How to effectively measure your DevOps

(Image credit: Image Credit: Profit_Image / Shutterstock)

DevOps is becoming more and more commonplace as businesses are discovering the need to improve their ability to deliver high quality software updates regularly to keep customers happy. The difficulty is that most businesses fall into one of two traps: either a) DevOps practices feel like just another fad with unclear outcomes (and therefore are a questionable investment), or b) businesses believe they are too immature to adopt DevOps practices. However, it is a well-known fact that adopting DevOps practices improves not only software delivery velocity (and associated value delivery), but the quality too.

It can be challenging to really deliver software faster while improving quality, but one of the best and most valuable ways to know if you’re getting it right is to measure everything. Metrics can tell you exactly how your teams and tools are performing, and this provides visibility into whether or not the delivery lifecycle is functioning to a high enough standard.

The problem is that not enough businesses are measuring the effectiveness of their DevOps transformation properly, or even at all. They might argue that they’re “too immature” to really dive into their DevOps inventory. But in reality, for a business to improve its DevOps processes, it needs to start measuring how they’re performing currently, to then identify where improvements can be made. Peter Drucker is quoted as saying “you can’t improve what you don’t measure.” That isn’t strictly true. You can improve it, but how would anyone know it was improved without measurement?

Measuring gives businesses the benefit of knowing where their processes are holding them back, or in other words, what their constraints are. The ‘Theory of Constraints’ (first published in The Goal by Eliyahu M. Goldratt and Jeff Cox, 1984) states that every organisation must have at least one, and it encourages organisations to focus on improving the constraint(s). The theory is that a business is only as strong as its weakest link, and so by focusing attention on the weak links, a business can improve its profits with minimal effort and resources. To establish what constraints a business has, there must be metrics to evaluate the success of every aspect.

Which metrics matter?

If we apply this theory to the world of software delivery, to improve the delivery pipeline, businesses must measure every process to determine which are creating bottlenecks and slowing down software delivery. By understanding where improvements can be made, and fixing them, businesses can push themselves to become elite performers, as outlined by the 2019 State of DevOps Report. In this report, pipeline performance is divided into four levels of businesses: low, medium, high, and elite. The research found that elite performers deploy 208 times more frequently than low performers; their lead times are 106 times faster; their change failure rate is seven times lower; and their mean time to repair (MTTR) is a staggering 2,604 times faster. But while it’s clear these numbers are impressive, what do these four key metrics really mean?

  • Deployment rate – How often a team deploys code to production
  • Change lead time – The length of time from code being committed to code running successfully
  • Change failure rate – How often a deployment goes wrong, in comparison to how many are right over time
  • MTTR – The time it takes from a failure to the recovery from that failure

These four metrics cover two of the most important aspects of the pipeline: the first two address speed, while the second two focus on quality.

According to the State of DevOps Report, elite performers are those which deploy on demand several times a day – contrast this with low performers, who are more likely to deploy anywhere from once a month to once every six months. The MTTR for the elite is less than one hour, and in comparison up to one month for the lowest performers. Clearly, these metrics can tell you very quickly how well a business is conducting its DevOps processes, and therefore these can be used to identify the constraints that each business should focus on to improve their delivery pipelines.

Using metrics to your advantage

The most crucial part of the measurement process is that you have to acknowledge that there’s a problem, before you can set out to resolve it. Teams should commit to using metrics to drive changes in their behaviour and measure the success of these changes. To effectively integrate these metrics into existing processes, DevOps teams should carry out value stream management (VSM) to ensure all parties understand the metrics in detail – what do they mean, what do they show, and what can be altered using this information?

The next step is to pick a pipeline and start measuring it. You have to look at each pipeline individually, so this can seem daunting to start with, but once the measurement processes are in place it becomes easier. For each application, start tracking how many times it goes into production – this gives you the deployment rate. Track lead times by recording when a deployment begins and when it is marked as complete. When something goes wrong, make a note of it, and also how long it takes before it is fixed. If there are a small number of releases, this may be simple enough for businesses to track manually. However, there are multiple tools available that can help by automating this and monitoring if there are more releases to manage.

The MTTR is arguably the most important to improve if it is not very low already. If it takes a business four to six months to restore service, it can’t deploy anything else during that time. With no new releases going out, the business is potentially missing out on profits. However, if the MTTR is low, new deployments can be delivered much more regularly, because any issues are resolved much quicker.; overall, the risk for every release is lower.

Once these measurements have been taken from one pipeline, move on to the next one, and repeat the process. By doing this for every pipeline, teams can build up an accurate picture of which areas they are struggling with, and this shows them where to focus their attention. As the theory of constraints predicts, businesses will see their delivery times and release quality improve quickly once small changes start to be made to the areas that these measurements suggest need attention.

The four key metrics are nothing new – they’ve been talked about before and they will continue to be the topic of conversation for as long as there are businesses that fail to improve their DevOps processes. But it is important to understand that no business is too immature to start measuring, and once you start, your maturity will gather momentum and DevOps will make a huge impact to your business value.

Jeff Keyes, VP of Product, Plutora