How to Measure the Quality of a Software Development Process?

Are we improving? Are we better than we were in the last sprint? Should we change our approach? I can’t recount the number of times I’ve been asked those questions as a Scrum Master. And I was never satisfied with the answers I would give to directors, managers, or even VPs. But, of course, I can feelthat we are progressing and getting better, that our process is more efficient and robust. But communicating with gut feelings should not be our standard for reporting progress.

How can I, as an Agilest, make sure that the higher-ups see the value of adopting Agile as we strive to be the best in our field? Lean manufacturing and the DORA research program inspired me to go back to basics. And it all starts with measuring your process! Below is my journey to get some basic metrics on the quality of our process and my humble advice on improving those key performance indicators.

On the Importance of Cycle Time

Here’s my rule of thumb when trying a new habit: start with the easiest thing. If you want to start measuring your development process, start with data you already have on hand. How long does it take for a story, a task, or Post-it to go from in-progress to done or deployed? As Agilest, we should try to reduce cycle time in our process. It’s an indicator of how things are going on the job. Here’s a small experiment that could help you get started.

First, start simple:

If you use a project management tool like Jira or Azure DevOps, you can directly access data on your cycle time in your dashboards. If you’re not there yet, here’s a simple way to start measuring cycle time.

• Grab a whiteboard, Post-it notes and markers.
• Divide your whiteboard into four columns (To-do, In progress, In review, Done).
• Write each task on a separate Post-it note.
• Add a dot on your Post-it for every day you spend on a task.
• Change the color you use whenever your Post-it changes columns.

You can then crunch the data by counting the dots for each colour on each Post-it note. The result will be your basis to calculate the average cycle time for a given period (a sprint, a week, or whatever makes sense to you).

Ready to add some challenge?

• Repeat the exercise and input your new data in a graph to determine if the performance indicator increases or decreases from the previous cycle.
• Do it again over a more extended period (I recommend three months) to keep track of any variation or improvement.

Why Is Cycle Time Important?

Cycle time can help you better communicate the progress you’re making with the team and stakeholders. It’s an excellent high-level indicator of your average dev team velocity and process, all other factors being equal. In general, you want a stable or decreasing trend for cycle time. So if you observe your cycle time increases, it can act as a trigger for action.

My Cycle Time Average Increased, Now What?

If you see an increase in the average cycle time during your sprint, dig deeper! Here are some questions to help you understand what’s going on:

• Is there someone on vacation or new hire on the team? Your team might have a knowledge problem. You might want to emphasize knowledge transfer activities. Pairing up is an excellent way to do it.
• Is the team working on something more significant? Is the work item’s size similar to the previous ones? If your longer cycle time is a symptom of a more complex work item, you might need to review your story slicing during the refinement phase.
• Which column caused the cycle time increase? If it’s the review, you might want to address it during a retrospective.
• Is there an unresolved conflict within teammates? First, make sure that it doesn’t get worse. Then, organize activities (1-on-1, group discussion, silent brainstorm) so people can voice their point of view.
• Were you affected by a production issue? Try to analyze your team’s response by planning a post-mortem activity.

TLDR:

• Start small.
• Count the number of days you consider a given task to be in progress.
• Measure your average cycle time for the current sprint.
• Take action if your cycle time increases sprint over sprint.

But cycle time alone isn’t enough to assess the quality of your process. This brings us to throughput!

Team Tempo: It’s a Marathon, Not a Sprint

Now that we measured cycle time let’s zoom out and look at team throughput. This performance indicator is dependent on your team’s context. Throughput is the number of work items processed over a given period. It’s simple enough to gather data for this one. Let’s say your time reference is a week: count the number of tasks that went from to-do (backlog) to done. You only have to look at the last column of your process.

Throughput is also a key performance indicator that you can observe in Jira or Azure DevOps. If you don’t use these tools,here are some key points to consider to get started:

• Define the time period (using sprint length is common).
• Count the number of items in the done column at the end of that period.
• Keep track of your data with a spreadsheet and a linear graph.
• Give yourself at least four iterations to identify trends.

Why Is Throughput Important?

The team’s throughput by itself is not the most helpful information you can gather. Instead, it should serve as an indicator of the team’s productivity to determine if you’re delivering more or less value to your customers over a fixed period. Many variables can influence throughput. After a while, you can even use this metric to predict your team’s capacity to deliver a project. Pretty cool, right?

My Throughput Decreased, Now What?

Here, we are looking for a stable or increasing trend. If your throughput for a given period is lower than your average, you might want to investigate. Here are some places to start:

• Was there a new impediment that challenged the team during the sprint? You might want to look at why it wasn’t caught before, during the planning or the refinement.
• Was there an unplanned item that popped up during the sprint? If this is a common trend for your team, start keeping track of those unexpected items. Make them visible to the team so they can act or change their habits.
• Were people not working together to tackle the sprint goal? Having multiple focuses with people working in silos can make the team quite dysfunctional. You might want to introduce a WIP (work in progress) limit,so people have no choice but to work together collaboratively.
• Was one work item bigger than initially expected? We typically slice stories and tasksduring the refinement process. If your team neglects that step, try to influence team members to have a newer definition of “ready” for your stories. It will help identifymore significanttasks beforehand.

TLDR:

• Start small.
• Count the number of completed items.
• Use a linear graph to watch out for throughput trends over time.
• Take action if your throughput tends to decrease steadily.

Final Thoughts

Cycle time and throughput aren’t the only metrics indicating the overall quality of your development process. But you’d be surprised at the level of information they can give you about your team. Confirming your gut feelings with these key performance indicators will make you more confident when you speak with directors, managers or VPs. You’ll also see the impact that your actions as a Scrum Master can have on your team!

Counting these metrics manually takes quite a bit of effort on your part. And let’s be honest, your added value is not being a data scientist for your team: it’s helping them. That’s precisely why we created Axify: a web platform that increases visibility in the software development process. It allows you to easily keep track of the key performance indicators mentioned above along with many more (we talked about a few of these metrics here).