Explore the realm of Change Failure Rate, a crucial metric in software development that gauges the success or failure of implemented changes. In our upcoming blog post, delve into why understanding this rate is vital for ensuring the stability and reliability of your software releases.
In today’s dynamic environment - technology companies struggle to evaluate the performance of their development teams. Different studies have been conducted aiming to define objective criteria that will make engineering teams more agile and efficient in the software development process.
In the past 10 years – more than 30000 IT professionals worldwide have participated in the global DevOps acceleration survey. As a result – in 2015 Google established the program called DORA (DevOps Research and Assessment) group. In 2018 they published their study Accelerate introducing the DORA Metrics – a concept that defines 4 key metrics that can distinguish the high-performance development teams from the average ones. The metrics are listed below and in this article we explain one of them - Change Failure Rate (CFR) as a number that measures the percentage of changes to production that result in degraded service (or incidents). We will clarify the role of CFR in tech companies and the need to identify and manage the change failure rate in a proper way.
The development team creates changes to production or are released to users. Some of these changes will cause incidents (i.e. service outage or hotfix), so the Change Failure Rate is the percentage of changes that result in service degradation.
Along with the CFR - DORA group defines three more metrics (called DORA metrics):
DORA metrics give the team leaders truthful insights, as they can analyze the metrics to assess the team performance. Then they can improve their development processes. By looking at CFR and MTTR, you can ensure that your code is robust and stable, while reducing the failures.
You need to work on speed AND stability at the same time. You can't only improve stability, and you can't only improve speed. For example, you could go faster but it will likely increase incidents. And you could take more time to code to increase quality, but you'll sacrifice speed and you'll not have a fast enough pace.
Change failure rate measures the percentage of deployments that cause a failure in production and need to be fixed or rollback after they are deployed. Change failure rate determines how many changes that we deployed to production resulted in an incident. For example, a high change failure rate indicates that your team didn’t pay enough attention to testing, or that their deployment process is not efficient enough. Therefore, CFR measures the stability and quality of your team's deployments.
Different organizations have different criteria of what failure means in their operations, since there is no universal definition of a failure. Thus, a need emerges to define the term failure on a high level at each organization, so all the teams will respect and intercept the failures when they occur. Here we present some cases that are treated as failures by the engineering teams:
Examples of failures include bugs introduced into production, system downtime, or any event that requires a 'hotfix' or rollback.
Not all issues that occur post-change are classified as failures. For instance, minor bugs that do not affect the system's overall performance or that do not impact the user (ex: fixing a label) are not considered failures.
Change failure rate is a great way to determine how often releases lead to problems. A high value indicates that your development or testing process has room for improvement. For example, if 30% of your deployments require rollbacks, it may be time to rethink your testing environments to catch issues earlier.
Frequent failures mean more disruptions for users, which can damage customer trust and cost you money. You can ensure smoother releases and more reliable services by tracking and taking steps to reduce change failure rate.
Change failure rate provides concrete data on the quality of deployments. This can motivate teams to write more robust code, improve testing, and communicate better across development and operations.
Measuring change failure rate is critical to DevOps practices like continuous improvement. It gives teams a clear metric to work on reducing, which helps them get better with each release. For example, an e-commerce site that gradually reduces its CFR may see fewer cart abandonment issues caused by release-related bugs.
Monitoring the metric over time can help your team decide when it’s safe to release more often or when to hold back and address issues. For example, a sudden spike in change failure rate to 15% after an architectural revamp may indicate that the new architecture is introducing unforeseen risks or incompatibilities. In this case, it would be prudent to pause deployments, investigate the root causes of the failures, and make necessary adjustments before resuming releases.
If you measure the correct metrics – it will be easy to compute the change failure rate. You just divide the number of deployments that cause failures by the total number of deployments. As example - if your team has made 6 deployments today, and 2 of them caused issues that require fixes, then the team’s CFR rate is 33%. In CFR calculation - you only consider the changes/ failures deployed in production, not the failures that appeared in the testing phase, as they were never deployed.
According to DORA group recommendations – you should also observe the three other metrics: deployment frequency, lead time for changes, and time to recover. To achieve better performance and optimize the right things - along with measuring change failure rate, you should establish a procedure that defines what failure is.
When the team is making a deployment to production, the work has been integrated to trunk, all tests and CI passed and should work as expected. In reality, many unplanned actions can occur, and some of the changes will result in failure or deprived service. A certain rate of failures is expected, but you need to know what is an indication of a low team performance.
According to the DevOps acceleration Report (2022), there are 3 classifications of CFR: high performers are between 0-15% , medium performers between 16-30% and low performers between 46-60%. Still, this classification changes every year. In companies - 11% of the teams are high performers, 69% are medium performers, 19% are low performers. Measuring CFR gives you an insight about your engineering teams’ development quality and helps you see where changes are needed, before this becomes a serious issue.
*Classification of Change Failure Rate, According to the DevOps acceleration Report (2022)
Even though the change failure rate formula is straightforward, there are some caveats to consider to avoid inaccurate or misleading values. Below, we will discuss these caveats, along with some advice on how to calculate accurate values each time:
As discussed above, different teams and organizations can define failure differently. Be clear on what counts as a failure (e.g., bugs, downtime, rollbacks) and ensure everyone on your team is aligned. Use automated tools like Axify to track and log these failures consistently so they don’t slip through the cracks.
Not all deployment failures are related to the code changes being released. For example, a deployment job may fail due to infrastructure, network, or configuration errors. These should be tracked separately and not counted as part of your Change Failure Rate.
If a failure is caused by something outside your control—such as a third-party service outage or network issue—it does not reflect the quality of your code changes. Exclude these from the CFR, too, to keep the metric focused on your team’s performance.
As your system evolves, so might your definition of what constitutes a failure. Establish a policy to periodically revisit your criteria and keep the change failure rate metric relevant and accurate.
Failed deployment recovery time is defined as the time the system takes to restore the service when an incident occurs. Together with the change failure rate, MTTR is a measure of the quality and stability of your team’s delivery process. If the Change Failure Rate is high, MTTR usually tends to be high as well. This means a longer-than-desirable time is required to restore services to their normal functioning state, which hampers the overall software reliability and business productivity.
By improving the engineering culture, DevOps practices will lead you to smaller, but more frequent changes. With smaller changes, incidents are easier to spot and manage, hence reduce the time it takes to fix them. It will be safer to deploy more changes. It is less risky for a business to have a team that can deploy more changes with fewer failures and can fix a mistake quickly (eg. in less than 30 minutes). Also, high CFR and high MTTR means more time spent on fixing bugs and service degradation, which is a dollar spent to not create more business value.
In order to understand where there are flaws in the deployment process, CTO-s usually keep track of the team's change failure rate. A large CFR may cause multiple consequences in terms of both financial, operational and product costs, so it is crucial to identify this rate and manage it. More bugs result in losing customers, due to the trust and reputation impact. Here is a list of the main issues that can arise from having a high CFR:
In financial terms, the cost of change failure rate can vary a lot depending on the severity of the issues, the type of the failures, and the size and complexity of the system. It also depends on the size of your organization, the number of customers affected, and the specific industry or sector. It is difficult to define an acceptable monetary cost for change failure rate as this varies a lot.
In the ROI of DevOps transformation there are costs of downtime based on different factors: e.g. for Fortune 1000 companies, the average hourly cost of an infrastructure failure is $100,000, and average cost of a critical application failure per hour is around $500,000.
In order to reduce the change failure rate and their associated financial costs, it is good practice to conduct regular assessments of the CFR costs and compare them with similar organizations within your industry. By doing this, you will understand the relative costs and can set realistic goals and objectives.
To reduce CFR - teams must embrace best practices such as rigorous automated testing, continuous integration and continuous deployment (CI/CD), performing diligent code reviews, and deploying efficient system monitoring solutions. Reducing the change failure rate is crucial to improve your team performance, and to minimize it - the best approach is to implement a Continuous Delivery, including the following practices:
Other best practices to minimize CFR include:
Along with CFR, you need to track other associated details like - duration of the outage or service degradation due to the failure, and the steps needed to restore the service. Tracking the outage duration helps the team to prioritize its efforts and improve the processes. Conversely, tracking the restoration steps helps the team to understand the main cause of the failures.
Manual calculations can provide you useful insights, but an automated tool like Axify.io can accurately model system behaviour and track change failure rate effortlessly. Axify is a single platform to observe all the key performance indicators that will help you improve your development and delivery processes. It is equipped with superior dashboards and provides constant tracking of DORA metrics in real-time, hence simplifying the whole process and empowering teams to concentrate on making improvements. There is no manual calculation, so less possibility for human error. With Axify - each team calculates their CFR and it is not normalized between them.
Some of the key Axify features regarding the DORA metrics include:
Keeping your finger on the pulse of the Change Failure Rate can yield revealing insights about the overall system health. By making diligent efforts to lower this rate, stability is dramatically improved, fostering a streamlined operation and a satisfied team, resulting in a win-win situation for everyone involved.
To find out more, read our article | Software development metrics: to rely on your projections with confidence
