Classic Testing Mistakes is a paper by Brian Marick, 1997.
From the Introduction:
Classic mistakes cluster usefully into five groups, which I've called "themes":
The Role of Testing: who does the testing team serve, and how does it do that?
Planning the Testing Effort: how should the whole team's work be organized?
Personnel Issues: who should test?
The Tester at Work: designing, writing, and maintaining individual tests.
Technology Rampant: quick technological fixes for hard problems.
In the rest of the paper, Brian describes several "classic mistakes" in each category.
The role of testing
- Thinking the testing team is responsible for assuring quality.
- Thinking that the purpose of testing is to find bugs.
- Not finding the important bugs.
- Not reporting usability problems.
- No focus on an estimate of quality (and on the quality of that estimate).
- Reporting bug data without putting it into context.
- Starting testing too late (bug detection, not bug reduction)
Planning the complete testing effort
- A testing effort biased toward functional testing.
- Underemphasizing configuration testing.
- Putting stress and load testing off to the last minute.
- Not testing the documentation
- Not testing installation procedures.
- An overreliance on beta testing.
- Finishing one testing task before moving on to the next.
- Failing to correctly identify risky areas.
- Sticking stubbornly to the test plan.
Personnel issues
- Using testing as a transitional job for new programmers.
- Recruiting testers from the ranks of failed programmers.
- Testers are not domain experts.
- Not seeking candidates from the customer service staff or technical writing staff.
- Insisting that testers be able to program.
- A testing team that lacks diversity.
- A physical separation between developers and testers.
- Believing that programmers can’t test their own code.
- Programmers are neither trained nor motivated to test.
The tester at work
- Paying more attention to running tests than to designing them.
- Unreviewed test designs.
- Being too specific about test inputs and procedures.
- Not noticing and exploring "irrelevant" oddities.
- Checking that the product does what it’s supposed to do, but not that it doesn’t do what it isn’t supposed to do.
- Test suites that are understandable only by their owners.
- Testing only through the user-visible interface.
- Poor bug reporting.
- Adding only regression tests when bugs are found.
- Failing to take notes for the next testing effort.
Test automation
- Attempting to automate all tests.
- Expecting to rerun manual tests.
- Using GUI capture/replay tools to reduce test creation cost.
- Expecting regression tests to find a high proportion of new bugs.
Code coverage
- Embracing code coverage with the devotion that only simple numbers can inspire.
- Removing tests from a regression test suite just because they don’t add coverage.
- Using coverage as a performance goal for testers.
- Abandoning coverage entirely.