What to Do When Bug Counts Don’t Speak for Themselves?
Bug counts on a project speak volumes
about the quality of testing for a particular product and how vigorous
the test team is working to “assure quality.” Bug counts are invariably
a primary area of test metrics that are reported to management. What is
the rationale behind drawing so much attention to the number of bugs
being found through the course of a project?
I have heard it said that QE’s job is to find bugs. If this is the
assumption of management, bug counts will be an important indicator to
them that QE is doing its job. They expect to see bug counts rise
dramatically in the early stages of testing, and they expect to see the
find rate decrease as the project comes to an end. These are
management’s statistical expectations when they believe bug counts are a
metric to assess quality of testing.
If high bug counts, then, are an indicator that quality is going up, low
bug counts can be seen as an indicator that something just isn’t right
with the testing process. Management might imagine different problems
that are preventing bugs from being found:
Management might see red flags when
bug counts are low, but a number of causes may contribute to low bug
counts. On the second or third iteration of a product, the bulk of the
defects may have been found on an earlier cycle. Or especially good
development practices may have been implemented: strong unit testing,
code reviews, good documentation, and not working developers to death.
These are supposed to result in lower bug counts.
Ultimately, however, QE will justify low bug counts when it can justify
its test coverage. If the product under test is being tested with
thorough coverage, the bug count should be treated only as a supporting
statistic, not the primary one. After all, we all know that a quality
product hasn’t been reached when a certain bug count is reached. Quality
is achieved when test coverage is maximized and bug finds decrease to a
minimum.
There are several things you can do when bug counts are low and
management is questioning the quality of testing:
One thing to bear in mind: while you
can use the above methods during testing cycles to understand and cope
with a low bug count, the ideas are still applicable before testing even
begins, while test cases are being written for a project, and while
development is still in full swing. Good test coverage is something to
be planned ahead of time, and having gone through the effort of mapping
coverage and functional test cases early in the project, you will
prevent yourself from spending valuable testing cycles repeating
tasks.
While low bug counts can cause people in both development and management
to question the effectiveness of the testing, do not be defensive about
it. Use it as a trigger to prove what you should already know—your
testing efforts are appropriate, effective, and your coverage is
maximized. Don’t let your bug counts do the talking—your test coverage
should say it all.