Top 5 QA Metrics For QA Managers

QA metrics are values used to measure the effectiveness of a testing process, team efficiency, and the outcomes of test results throughout the development lifecycle. Different teams prioritise different goals for monitoring and improving processes.

#Metric vs KPI

#How good is your software?

Instead of answering that the software is good enough and customers are happy or tests are passing, there should be quantifiable data to validate the quality of software.

The number of production down scenarios for customers, mean time for failure, soap test results provide data about reliability and add more weightage to your answer.

KPIs (Key Performance Indicators) are specific metrics driven by business objectives and are crucial for measuring zsoftware performance in key areas and lead to important insights. This could be the number of deals booked for the product

OKRs or Objectives and Key Results are transparent goals that can be reviewed periodically. Unlike KPIs, which are often specific to business processes, OKRs are broader targets and not limited to individual roles, such as a tester. For example, an OKR might be a target of a million in revenue.

#QA Benchmarking

A QA benchmark is a standard for assessing the quality of a software product or a testing activity. This involves comparing the metrics and results of current projects or organizational efforts with set benchmarks or industry best practices. Through this comparison, organizations can set realistic and achievable quality targets that align with industry standards.

#Why QA Metrics?

• **Track the bug status\ to understand how many bugs were found, fixed, reopened, closed, or deferred and identify which ones were critical

• **Diagnose issues \ diagnose, verify and localize issues effectively

• **Identify areas of improvement \ to enhance processes, and refine testing strategies to optimize performance

• **Predict Risks
  **Anticipate potential risks to address them proactively

• **Make informed decisions \ Utilize metrics to make data-driven decisions

• **Plan effectively \ Improve planning by understanding trends and outcomes from metrics

#Absolute Metrics and Derivative Metrics

#Base / Absolute Metrics

As the name suggests, they provide absolute numbers from actual data (number of tests passed, failed, total number of requirements, tests executed, time to run etc) obtained during the test case development and execution throughout the testing process.

#Derivate Metrics

Absolute metrics are used in conjunction with Derivative metrics for more detailed insights into the testing process, such as identifying trends, correlations, or areas for improvement. They allow testing teams to dive deeper into issues that may be hindering the accuracy and speed of the testing pipeline.

Choosing the right combination of metrics is important to ensure that the metrics used reflects the quality of the testing process.

The strategy should be aligned on measuring future changes and how testing strategies and processes are expected to evolve over time.

Metrics help measure:

• the test duration and estimate efforts required

• bug count, its severity

• effectiveness of the test cases

• progress of your test execution

• stability of testing environments

• overall quality

• time to ship, expectation on release

• cost involved

#How to pick the right metrics? How do we decide what metric is useful?

This depends on the project demands, goals (internal such as test case or test execution related, and external such as customer satisfaction, a business outcome).

We can classify these metrics into three categories:

• Lagging indicators

  Lagging indicators evaluate past performance based on results like revenue, customer satisfaction scores, and churn rate.

• Leading indicators

  Leading indicators guide managers with a focus on predicting future performance of key outcomes. Leading metrics guide products forward successfully. Eg, open bugs, pass percent, build failures, MTTR, code coverage etc.

• Vanity Metrics

  Vanity metrics or feel good metrics such as the number of bugs logged, number of test cases etc.

#Arriving at a metric

The popular GQM (Goal Question Metric) framework is a good place to start.

These metrics start with a Goal in mind, Questions to reach the Goal and the Metric to arrive at the outcome.

Some questions could be:

• Are there particular types of defects that are consistently missing?

• What are the key functionalities and features that need to be tested?

• Are there any high-risk areas or critical features that require more focus?

Now, let's look at some important metrics that help QA teams measure the quality of testing.

#Test Monitoring and Efficiency metrics

Measures how effectively the team find and address bugs. It provides insights into areas needing improvement, helps prioritize where to enhance test coverage.

Percentage of Passed test cases = Tests Passed / Total Tests executed*100%

Similarly, % of failed test cases, % of blocked test case percentage.

These are used in conjunction with defect KPIs such as average resolution time, time taken by devs to fix defects, % of critical defects, % of deferred defects, % of defects rejected, % accepted, % fixed defects etc.

#Test Effectiveness

This metric provides information on how much value the tests provide to the quality.

Defect Detection Effectiveness (DDE) or percentage is the overall effectiveness of regression testing which is calculated by comparing the number of defects found before and after software release to customers. Though the goal is to have a high test effectiveness percentage, best 100% which implies that all defects are found before going live which is hypothetical. A good way to ensure this is to keep track of metrics like the defect discovery rate and time-to-fix the defects.

Bugs found in Testing stage / Bugs found in Testing + Bugs found after Release to customers * 100

#Defect Discovery

A good way to ensure a high test effectivness is to keep track of metrics like the defect discovery rate and time-to-fix the defects. The Mean Time To Detect Defects (MTTD) or Mean Time To Resolution (MTTR) help QA managers plan future projects and measure progress.

MTTD = Number of defects detected / Total time to detect all defects​

Mean Time to Repair (MTTR) measures the average time to resolve a defect once it has been identified. This explains the efficiency of the defect resolution process.

MTTR = Number of Repairs / Total Downtime

#Team Metrics

QA managers could use team metrics to check if any team member requires help with a testing process or project knowledge. However this metric is only for information and not to point fingers.

QA team metrics could include:

• Defects reported, accepted, rejected per team member

• Distribution of open defects for retest per test team member

• Test cases allocated per test team member

• Test cases executed per test team member

#Defect Distribution

Defects can be categorized based on:

• Defect root cause like coding errors, design flaws or system configuration issues

• By feature/module

• Severity, Priority

• By type - functionality issues, usability issues or performance issues

• By tester-whether issue was discovered by development testers, QA testers, UAT testers or end users

• Test type / testing activity that identified the issue like code review, walkthrough, test execution, exploratory testing

• By platform/environment (code errors, architecture, design etc)

#Test Coverage

Test Coverage is a critical indicator of quality that shows the thoroughness of a test plan. It measures the extent to which the software application has been tested. Examples of test coverage metrics are:

• Test cases per requirement

• Defects per requirement

• Defects per set of requirements

Let's look at the test coverage metrics in detail:

Total Execution Percentage = Number of tests executed / Total number of tests to be executed * 100

This gives us an idea of the total tests executed compared to the test runs that are outstanding.

#Requirements Coverage

Requirements Coverage = (Number of Requirements covered / Total number of Requirements) X 100

#Functional Coverage

Functional coverage checks if the key functions of the application were tested. It ensures that each function was invoked by the running tests.

Functional Coverage = (Number of Functional Requirements invoked by Test Plan / Total Functional Requirements) * 100

#Product Coverage

This metric shows how many different products the app has been tested on.

For example, a web app being tested across a range of devices and operating systems to make sure it works well everywhere.

#Risk Coverage

It identifies the potential risks that a software application might impact the user experience. When tests are crafted to ensure the software remains functional and effective in such scenarios. One such scenario would be when a third party API that handles important transactions becomes unresponsive.

#Cost

It is a measure of the actual cost of testing compared to allocated cost. This would include people, infrastructure and technologies.

Cost of testing = (Total defect resolution time x Hourly cost of a dev) + (total replay time x Hourly cost of a tester)

Multiple factors contribute to the cost - in terms of automation, it could be script development time, maintenance and analysis.

In addition to these metrics we discussed, Defect Density and Defect Age are also indicators for QA managers on their team, testing efforts.

#Defect Density

Defect density = number of defects found / by the software size (in terms of lines of code, function points, or modules) for a given time

If a software product has 10 defects and 1,000 lines of code, the defect density is 0.01 defects per line of code which is 1 defect per 100 lines of code.

#Defect Age

Defect Age is calculated as the difference between the time a defect is fixed and the time it was discovered.

A lower defect age indicates that bugs are being resolved more quickly within each test cycle.