Software development metrics: to rely on your projections with confidence

Have you ever missed a delivery deadline? Unfortunately, the answer is probably yes! Software development can be like a black box: it's hard to know how a technology project is going. In this blog post, find out why it's essential to measure certain aspects of your development team and stabilize its workflow, plus some insights to do it yourself.

When we look at a development project at a high level, we may think that its progress is like a smooth river. For example, we might look at the type of items delivered and see that there are very few bugs, which is a good thing. We might also hear from team members that "everything is fine".

However, the reality may be quite different. For example, the team may say that "everything is fine" because we asked the question at the right time. Or maybe our cycle time is gradually increasing, but we cannot identify the cause. In short, it's challenging to have continuous visibility into a technology project. So how do we address the situation?

Measure, stabilize and forecast

Using a tracking tool can give you continuous visibility into different elements of a software project. In Axify's case, it's your software engineering processes, development activities, collaboration on items and team morale. Why these elements? Various frameworks form the basis of the platform, including the SPACE framework, which names these elements among those that drive better software delivery performance and, therefore, better organizational performance.

Measuring the elements that impact your workflow will allow you to analyze your delivery strategy to work on your stability. For example, a stable team prioritizes work in progress (WIP) to reduce the number of competing tasks. As a result, active collaboration on these tasks will increase and improve, reducing the wait time at each step. In addition, observing the right metrics allows you to quickly identify bottlenecks to deliver more value to your customers. The result? A smooth and continuous delivery flow.

"This consistency eliminates the randomness of software delivery: your team is predictable! If you can deliver an item in five days or less 95% of the time, you can be confident in your ability to repeat the same experience."

Measuring up

Performance indicators presented by Axify are framework agnostic (e.g. Scrum, Kanban, XP, Waterfall, etc.). This approach has the advantage of reflecting the results of your work methodology. In the words of Peter Drucker, who is described as the founder of modern management, "what can be measured can be managed." As of March 17, 2023 (because we're always working on improving our product!), Axify analyzes various key performance indicators, including the following, to give more visibility into the quality of a team's workflow.

• Cycle time
• Throughput
• Work in progress (WIP)
• DORA metrics (deployment frequency and lead time for changes)
  
  
  

Why these metrics?

These metrics are easy for everyone to understand, offer the ability to interpret queue theory, and allow concrete actions to improve workflow. They are also among the metrics that our VP of Engineering recommends tracking to improve your team's performance.

That being said, it is essential to understand that these metrics are interdependent. For example, let's say you want to deliver faster (reduce your cycle time) and increase the number of items delivered (throughput) for a given period. Invariably, changing these two variables will affect your work in progress (WIP), which can hurt the quality of the work done. Having more tasks in progress can cause the team to work in silos, increasing the risk of errors (and introducing bugs), which would mean correction time and a decrease in the value delivered. This is why it is so important to take the time to inspect your process to make the decision that best suits your reality.

"Indeed, most teams seek to improve only specific dimensions of performance and may forget that any change will impact the other dimensions. Every decision made today will impact the team's predictability tomorrow. This is why stability is critical!"

Stabilizing

In reality, teams seek stability, i.e., maintaining a steady delivery rhythm without significant variations, for a long time. The more stable a team is, the better it performs: it delivers more items faster, with higher quality, and does so continuously. And the more predictive a team is, the more accurately and confidently it can answer the question, "When will we deliver?"

Work in progress, cycle time and throughput are interdependent and will always affect delivery. At the process level, the goal is to limit WIP to have visibility into these key performance indicators and make them accessible to the entire team and organization. Quality communication and transparency are crucial to increasing performance.

"Creating and maintaining a visual display of productivity and quality metrics and current bug status, making it available to the development team and leaders, and aligning these metrics with business goals can help increase software delivery velocity."

In addition, using application and infrastructure performance data to make business decisions daily helps increase delivery quality and speed. Indeed, throughput and cycle time (how many items you deliver in how long) determine the Service Level Expectation (SLE), which facilitates client expectation management by presenting data in a common language.

Planning ahead

Is your development team stable? Great! You can now predict your future deliveries with more accuracy and confidence. But be careful not to be too quick to claim victory. Software development is not deterministic, i.e. it is impossible to repeat the same experiment and always obtain the same result: we must therefore be probabilistic.

Can we simply make a linear projection to know when we will deliver?

"If your team is perfectly stable (i.e. work in progress, cycle time and throughput are consistently steady) and you want to estimate the potential delivery date quickly and easily for your own benefit, maybe. However (and we stress this again), the world of software development is not deterministic! The future is always uncertain, and when there is uncertainty, it is best to use a probability approach, especially when creating expectations for customers or stakeholders. A proven algorithm makes this kind of prediction easier while introducing the concepts of risk and confidence levels: Monte Carlo simulation."

Imagine working on a project that started some time ago, and you want to know when you will deliver a feature. You could rely on your intuition to estimate the delivery date or extrapolate with a linear projection, but these solutions do not consider variables that could affect delivery. Now, instead, imagine that an army of 1,000 analysts is producing different scenarios for you:

• What happens if we work slower? Or faster?
• What if we discover that there are more items to work on along the way? Or fewer items to deliver?

These 1,000 analysts study the chances of different scenarios occurring and then choose the most likely scenario. Now imagine that you don't have to pay those 1,000 analysts, and you can get the answer right away. That's what Axify's prediction tool allows you to do. This technique is better than your intuition and simple regression models.

A few examples

1. What is the probability of completing 20 items by May 3, 2022? → Possible answer: You have a 42% chance of delivering by that date.
2. When will we deliver 10 items? → Possible answer: You have an 85% chance of delivering 10 items between April 15 and April 30, 2022.
   
   

A forecast example in Axify to deliver 10 items with a start date of March 11. Axify predicts a delivery date of March 25 with 85% confidence.

I already use a tool like Jira, Azure DevOps or PowerBi to get data. So how does Axify add value?

Indeed, several Agile project management tools provide various reports. In addition, some platforms even offer a high level of customization when creating custom reports. That being said, many of these reports are not suitable for supporting teams in stabilizing their processes. Here are some examples:

• Burndown/Burnup Charts: The problem with these charts is that you come to rely on trends and averages. Past performance does not guarantee future performance (again, the software development world is full of surprises!). These graphs are also used to make projections, often linear (pessimistic and optimistic scenarios), that do not meet the definition of an estimate and are less relevant than a simulation with a proven algorithm.
• Control Chart: This graph shows trends in cycle time over a period. The drawback of a Control Chart is that it is often based on a mean or standard deviation (based on variance). Although relevant, outliers have a significant impact on the result of these formulas. Axify, on the other hand, uses percentile-based graphs instead. The definition of an outlier is that they only occur once in a while. So, we don't necessarily need to remove them from our analyses. But, on the other hand, since they are infrequent, they must not have a significant impact on representing reality properly. This is why graphs based on averages are affected by outliers, which does not help the team to stabilize.
• Cumulative Flow Diagram: This graph is one of the most advanced for analyzing a team's Lean flow. It is generally used to detect trends, but it is not necessarily effective in predicting the future and would be limited to a linear projection. Moreover, it is not representative when we do not have enough data.