You are here: » Software Testing » The Problem with Software Testing Metrics

The Problem with Software Testing Metrics

If you have been a software tester for any length of time, or if you have gone through the ISEB or ISTQB certification process, you probably understand some of the issues regarding software testing metrics.  There always seems to be some ambiguity around questions like how to measure the effectiveness of software testing, how to determine which items must be quantified, and more.  The reason is that the process of identifying and defining software testing metrics is subjective, and will differ depending on the specific company or industry.

Many companies focus on the “low hanging fruit” of software testing metrics, such as measuring testers by how many bugs they find.  But although this is an easy thing to measure, there is not a true correlation between the number of defects found and the quality of the tester’s performance.  Additionally, some testers who are evaluated based on the number of bugs found may try and increase this number by reporting very minor, inconsequential defects that have no bearing on the final outcome.  Therefore, simply focusing on the number of bugs found is not a good indicator of performance or quality.

So now that we know that software testing metrics are subjective and will differ by company and industry, how can we define the most relevant ones?  To start, companies must consider things such as the scope and overall goal of the project, the number of team resources involved, the testing methodologies used, the degree to which similar testing projects with potentially similar metrics have been executed, the types of data points the company typically measures, and similar things.

These points of consideration will allow you to create a master list of potential metrics.  Then for each one, you will want to determine the degree to which it measures something that is important to the overall business, the degree to which it focuses on results, and the degree to which the metric could be manipulated by the testers. This process will allow you to remove the irrelevant and/or questionable metrics, thereby allowing you to focus only on the most important ones.

Although it is somewhat tedious, going through this process to identify the key metrics for each testing project is critical to generating the desired results.  The bottom line is that we cannot improve what we cannot measure, so it is critical to identify metrics that can be accurately measured in the context of overall business objectives.


Related posts:

  1. The Necessity of Organizational Bug Tracking
  2. The Everyday Trials and Tribulations of Software Testers

Leave a Reply