The seven deadly sins of agile metrics

(Image credit: Image source: Shutterstock/ESB Professional)

The “Cobra Effect” is named after an incident that took place during Britain’s rule in colonial India. Concerned about the abundance of poisonous cobras in Delhi, the British Government offered a bounty for every dead snake. After a time, resourceful locals started breeding cobras, leading to a dramatic increase in the venomous snake population.

The Cobra Effect teaches us the importance of implementing reward and measurement systems that drive the right kind of behaviours and avoid unintended consequences. As the old adage goes: the road to Hell is paved with good intentions.

Agile development teams should use metrics to improve delivery and avoid the road to a snake-ridden Hell. On this journey, however, there are temptations to use metrics incorrectly. To stay on the heavenly path, you must avoid the “Seven Deadly Sins of Agile Metrics”:

Pride

The impulse to tell positive stories that we can feel proud of is hard-wired into our human nature. This is what causes people to manipulate data sets to present things in a flattering way. However if Agile teams continually present nothing but positive information, eventually the ‘snakes’ multiply.

Agile team leaders must create the right culture and encourage members to raise issues that can be improved. There is also a strong case for using Agile metrics tools that generate consistent like-for-like data, rather than asking team members to manually generate ad hoc metrics and reports.

Agile is about continuous improvement, so why bother if you don’t want to find out what individuals and teams need to do to improve? Also, when the story is too consistently positive, what is there ever to be genuinely proud of?

Greed

Greed in Agile metrics is about aggregation - cooking up a lot of data so it’s both palatable and easy to digest. As an executive leader it’s tempting to want to gain insights by viewing simple dashboards with ‘high-level’ aggregated data. This is a bit like a fruit smoothie. If you blend apples, pears, bananas and kale together with milk, you get a surprisingly tasty, calorific drink that’s fast and easy to digest. To consume each of these items separately would take a lot more time, but it would also fill you up for much longer.

Let’s say you lead a small development team running three projects and you want to understand how much time is spent on developing features. Your aggregated number says 69.5 per cent. Looking at the individual projects, however, let’s say that for Alpha, teams are spending 92.5 per cent on features, while for Gamma and Foxtrot this figure is in the 60s. That 69.5 per cent doesn't tell you anything because the Alpha team is working on a new app project, developing all new code. In that case it’s normal to be at 92.5 per cent. But Gamma and Foxtrot’s teams are working on established technologies, so they need to do more bug fixing. For them it's normal for their percentages to be lower. So you have to be really careful not to be greedy for easy to consume aggregates and average because the real insights gets lost very quickly. By all means look for Agile metric dashboards that show aggregates, but they should also let you easily drill down into the details.

Lust

Lust in the context of Agile is all about the ‘one metric obsession’. Again, this highlights the risks posed by allowing team members to create their own dashboards. It's all too easy to focus and obsess on one metric and block out all others. For example “lines of code produced” is becoming the poster child of how not to use metrics in engineering. It doesn't say anything in and of itself. Four lines of beautiful code can be far more valuable than 100 lines of garbage.

To avoid falling into this trap, look at multiple metrics and understand the context around them. Ideally you use certain ‘leading indicator’ metrics to test a hypothesis, and then use ‘lagging indicator’ metrics to back it up. In a product team working to a release date, a leading indicator, could be for example, percentage of team availability. A lagging indicator follows an event and confirms that a pattern is occurring. For example, was an item delivered or not.

Envy

Envy is all about making comparisons. You don't want to create situations where teams or individuals become envious of others because you compare in the wrong context. To some degree it can be valuable to compare teams and people, but you have to understand the context.

Leaders in particular must be sensitive of the need to make comparisons in a context-aware situation. Going back to our previous example, the fact that Gamma’s project team is spending less time creating new features than Alpha’s doesn’t mean its performance is relatively worse than Alpha’s.

Gluttony

Gluttony is about metric overload. A team leader’s day job is not number crunching, even some might find it fun. It’s vital to stay focused on strategic goals and where to make  improvements to reach them. Ask yourself what are the key questions you need to answer to define success. Then aim to hit goals that will help to drive actionable insights.

To avoid gluttony and drive actionable insights, steer clear of tracking metrics with BI tools that are not context-aware. They're just tools to serve up a dashboard and provide metrics for metrics sake.

Wrath

Wrath occurs when metrics are used as a stick to turn people and teams against each other. Let’s say you have a developer, Tom, who is being branded by other team members as being ineffective and slow to deliver. The reality could be that Tom happens a key data scientist working on a really complex physics problem, which is naturally producing a lot of bugs. This engineer should really be given scope to fail fast and early.

Again, it’s vital not to use wrong, out-of-context numbers to punish people. Leverage metrics, dashboards as a team self-health check and an improvement tool, but don't use it as a management control tool. If you do, you are likely to get push-back from teams, and they might even gamify the process to tell you what you want to hear.

Sloth

Sloth is about lazy organisation of one or a few metrics. A lot of people talk to us about wanting to reduce cycle time. The risk here is that you can easily start to over-optimise in a small area. We’ve all seen those people at the gym that focus too much on one body part; they might have huge Popeye arms and really skinny legs that barely seem to support their weight. When you ask your team to focus on one thing, whether it be cycle times, velocity or whatever, you’re not going to have a well-rounded approach and team.

Undoubtedly establishing the right kind of culture and leadership are crucial for avoiding the Seven Deadly Sins. To further support your efforts and continuously improve, find a measurement tool specifically designed for Agile metrics. This will help you choose the right metrics and view them in the right context.

Davy Nys, CRO, Plandek