缺陷发现即成功:ISPE GAMP《Testing GxP Systems》第三版谈测试成功标准
Finding Defects Is Success, Not Failure
ISPE GAMP《Testing GxP Systems》第三版指出,测试成功不应以未发现缺陷或文档数量衡量,而应看是否在发布前充分挑战软件、找出重要缺陷。发现缺陷说明测试按预期发挥作用,真正失败是缺陷逃逸到运行使用中。文章建议关注缺陷发现早晚、同类缺陷是否复发及是否逃逸等指标。

Walk into almost any project meeting and watch the reaction when a defect is discovered during testing. The mood changes. The release schedule comes under pressure. Project manager begin reassessing timelines. Developers prepare themselves for difficult conversations. The immediate assumption is often that something has gone wrong.
But perhaps that perspective is flawed.
Finding a defect during testing is not a failure of the testing process. It is evidence that the testing process is working exactly as intended.
The fundamental purpose of testing is to identify defects before software is released into operational use. Every defect identified throughout the system lifecycle is one less defect that can affect patient safety, product quality, data integrity, or business operations. The real failure is not finding a defect during testing. The real failure is allowing that defect to escape into operational use.
Why Do We See Defects as Bad News?
For many years, success in computerized system validation was often measured by the completion of documentation rather than by the quality of the software itself. Teams celebrated error free test reports, successful protocol execution, and large volumes of completed paperwork. In this environment, finding defects during testing frequently became associated with schedule delays, rework, additional cost, project overruns and concerns about how the issue might be perceived during an audit.
That mindset is understandable. Defects cost time and money to correct. Interestingly, this is hardly a new concept. As far back as 1979, Glenford J. Myers, in The Art of Software Testing, described testing as executing a program with the intent of finding errors and argued that a successful test is one that exposes them. The same principle has been reflected in GAMP guidance for decades. Yet, almost 50 years later, finding defects during testing can still be perceived as bad news rather than evidence that testing is doing exactly what it was intended to do. However, from a quality perspective, discovering a defect while the software is still under development is exactly what testing is supposed to achieve. Every defect identified before release represents a risk removed before it can impact a regulated process.
Perhaps we should be asking a different question.
Instead of asking:
"Why did we find another defect?"
we should be asking:
"How fortunate are we that we found it before the system went live?"
The Purpose of Testing Has Never Changed
The technologies and methodologies have changed dramatically over the years:
Cloud computing, agile delivery, DevOps, continuous integration and continuous delivery, automated regression testing, software as a service, and computer software assurance have fundamentally changed how computerized systems are developed and maintained.
The purpose of testing; however, remains exactly the same.
Testing provides confidence that a computerized system is fit for its intended use.
Everything else, the documentation, protocols, reports, traceability, exists to support that objective, not replace it.
When testing becomes primarily a documentation exercise, we risk losing sight of why we are testing in the first place.
Shift the Success Criteria
One of the key themes throughout the third edition of the ISPE GAMP Good Practice Guide: Testing GxP Systems is that testing should focus on improving quality rather than generating documentation.
That requires a different way of measuring success.
A successful testing phase is not one in which no defects were found.
A successful testing phase is one that challenged the software sufficiently to provide confidence that the important defects have been identified before release.
If a test team completes hundreds of perfectly executed test scripts without identifying a single meaningful defect, we should perhaps question whether the testing challenged the software enough.
Conversely, if testers uncover important issues early, investigate them, correct them, and demonstrate that the corrected system is fit for intended use, then the testing has added significant value to the project.
Finding defects should increase confidence in the testing process, not reduce it.
Defects Become Information
Another important mindset shift concerns how we view test incidents. Not every issue identified during testing requires the same level of investigation or documentation.
Issues discovered during software development are often resolved within normal development practices. As testing progresses through later lifecycle stages, more formal test incident management becomes appropriate.
Importantly, those incidents provide valuable information.
Patterns begin to emerge.
Are defects consistently occurring in particular functional areas?
Are similar issues recurring release after release?
Are defects being detected earlier in the lifecycle than previous projects?
These questions tell us far more about the effectiveness of our testing strategy than simply counting the number of test scripts executed.
Measure What Matters
Software teams naturally like metrics.
The problem is that we often choose metrics because they are easy to collect rather than because they are useful.
Counting executed test cases or pages of evidence generated may tell us something about project activity, but very little about software quality.
More meaningful measures include:
- How early defects are being found
- Whether similar defect types continue to recur
- Whether automated regression testing is covering the critical paths
- Whether defects are escaping into operational use
- Whether testing continues to improve release after release
These metrics encourage continual improvement rather than simply reporting progress.
Used well, they become a feedback loop that improves future testing, future development, and ultimately future systems.
Better Testing for Better Systems
Perhaps the biggest change introduced by modern testing approaches is not new technology, automation, or artificial intelligence.
Perhaps it is simply a change in mindset,
Testing should not be judged by the amount of documentation produced
It should not be judged by how many test scripts were executed
It should not even be judged by how many defects were found
It should be judged by one simple question:
Did the testing provide confidence that the computerized system is fit for its intended use?
When we view testing through that lens, defects become opportunities to improve quality rather than problems to hide.
Finding defects during testing is not failure.
Finding them after release is.
来源:ISPE iSpeak 专家博客 · ispe.org