ISPE 博客谈 GxP 中 AI 项目首小时评估如何按风险后果界定验证范围

It's easy to read that as a story about human review making a system safe. The more useful lesson sits one step further in. The analyst sees the flags. The analyst doesn't see the chromatograms the model passed over. A false positive costs a few minutes, because a qualified person looks at it and sets it aside. A false negative is a peak the model should have flagged and didn't, and if nothing else looks at that result, the error can travel toward a batch decision and, in the end, a patient. Review of the flags protects against one kind of error and can be blind to the other.

So the question that decides whether eight weeks is enough is what happens to the results the model doesn't flag. If they still go through the lab's normal second-person review, a miss falls back on a control that already exists. If the model's silence means no one looks again, the model is choosing which results get scrutiny, and the risk to product quality is a different size. At the second firm, every unflagged chromatogram still went through the lab's normal second-person review, so that is where a miss would be caught. The time the second firm saved came from asking questions like that and putting the testing effort where a miss could do harm: scripted tests on the miss rate against acceptance criteria agreed up front, and unscripted testing around the edge cases. ISPE GAMP 5^® Guide (Second Edition) calls this critical thinking (Appendix M12), and the ISPE GAMP Guide: Artificial Intelligence applies it to AI in Appendix M4.

Both approaches could be defended. One of them spent four extra months and a good part of its validation budget on effort the risk didn't call for, and that's budget and expert time taken from systems where a patient could be hurt. I've watched the first hour of an AI project shape what follows more often than any protocol or vendor claim. When the conversation goes well, the later decisions line up behind it: how deep the testing goes, how closely the system is monitored, how changes are controlled, and who signs. When it goes badly, a team builds a six-month file on the wrong foundation, or worse, an eight-week file on a system that needed the six months.

Start from What Happens to the Patient When the Model Is Wrong

When a team meets its first AI system, the validation effort usually gets scoped from the technology. The model is complex, so the file must be heavy. It sounds prudent and it's backwards, and everyone raised on GAMP already knows why: assurance effort follows risk to patient safety, product quality and data integrity, judged in the system's actual context of use. ICH Q9(R1) says the same in its own words, linking the evaluation of risk to the protection of the patient and matching the level of effort to the level of risk. The complexity of an algorithm tells a team how hard the system is to build. It says very little about how much assurance the system needs.

Take a pair of examples. A neural network that summarizes deviation trends for a monthly leadership review is technically sophisticated and almost harmless in operation: if it's wrong, a chart misleads a meeting until someone checks the numbers. A plain decision rule that routes out-of-specification (OOS) investigations is technically trivial and serious in operation: if it's wrong, investigations go down the wrong path, the records they produce inherit the error, and a batch that should have been held can move on. Scope by complexity and the first system gets heavy validation while the second is waved through. Scope by consequence and both get what they need.

Who Needs to Be in the Room for That First Hour

If the first hour shapes so much, it deserves real design, and the most common failure is letting one function decide alone. A data scientist working alone reads the model and under-weights the workflow. A quality assurance (QA) reviewer working alone reads the worst case everywhere. A business owner working alone reads the benefit case. The initial assessment needs four people in one room for one structured hour: the process owner who knows what the system does in the workflow, the data scientist who knows what the model can and can't do, the QA reviewer who knows what an error would mean for product and patient, and the validation lead who carries the lifecycle consequences of what the room decides.

They need an agreed process scope and objective before they sit down, so they're all assessing the same thing. And they need to reach a shared view of the worst case, because that view informs a long list of decisions later in the project and in operation.

A Question That Gets the Conversation Going

When the room stalls, and it will, one question helps: what happens to patients and product if this system is wrong on 10 percent of records for six months, without anyone noticing? The numbers are deliberately arbitrary. Ten percent is big enough to matter and small enough that no single-record alarm will catch it, and six months is long enough for an error to spread into decisions. For some uses, a team should change them, because a model that screens for rare events can do real harm at a far lower error rate. The question is there to start a discussion, and what comes out of it is a list of what the team needs to do.

Ask it twice, once for each kind of error. Too many false alarms and reviewers start to tire, which matters because a tired reviewer is the control the system relies on. Too many misses and the question becomes who, if anyone, looks at what the model didn't flag. Then follow the answers to the controls.

A misleading trend line in a monthly meeting points to a periodic plausibility check on a sample of outputs. A stream of misclassified deviations points to the reviewer: if the reviewer caught them, the override log should show it, and if the reviewer approved them by reflex, the human oversight was never there. Months of misrouted investigations quietly corrupting corrective and preventive action records across sites point to acceptance criteria broken down by site and product family, and to monitoring that starts on the day of go-live. A wrong decision that could release product to market raises a harder question, which is whether AI belongs in that decision at all, and with what human oversight. Draft EU GMP Annex 22 is cautious on exactly this point for critical applications.

Keep the Harm and the Controls Apart on the Record

It's tempting to let the controls answer the risk question. “A qualified person reviews every recommendation” describes the control design. The harm a wrong output could do to a patient, to product quality or to data integrity stays the same whatever review sits around it. What the controls change is how likely that harm is to get through, and how quickly someone would notice.

Writing the two down separately matters because controls weaken in operation in ways the harm doesn't. When the override rate drifts toward zero, look closer before celebrating, because it can mean the reviewers have stopped reviewing. A simple test of any human review claim is to ask for the last time a reviewer disagreed with the system, and to check the records can show it. The exception path deserves the same test. If overriding the system's routing takes four screens and a supervisor password while accepting it takes one click, the exception path exists on paper and the oversight is weaker than the file says.

The GAMP AI Guide Already Gives the Room a Common Language

Section 21.2 of the ISPE GAMP Guide: Artificial Intelligence sets out a classification of AI sub-system autonomy, and it's the right shared language for how far a system acts on its own in the workflow. The first hour builds on it with what only the people in the room know: the specific harm in this process, how it could reach a patient, and which controls the team is relying on to stop it.

The same Guide asks a second question alongside autonomy, which is whose data established the model. A model trained entirely on a firm's own data sits under the firm's own change control. A vendor's pre-trained model, which the firm can't retrain or inspect, changes when the vendor decides. A system with modest consequences built on a vendor model still carries real change-control and supplier-oversight obligations, because the thing that changes its behavior lives outside the firm's quality system. That's the half of the assessment leaders miss most often, and it's the half the supplier audit depends on.

Write the Decision into the Validation Plan

A decision that lives only in meeting minutes is hard for anyone to follow later. The validation plan is the right home for it, with the reasoning alongside. A short section is enough: the system and its intended use stated in workflow terms, the worst case in each direction and how it could reach a patient, the controls the team is relying on, the names of the four people, and the triggers that would reopen the discussion. When someone later asks how the AI was validated, the answer starts there, and it's a strong start because it shows the thinking as well as the conclusion.

Figure 1. The initial assessment hour: four roles, one question asked for each kind of error, and the decision recorded in the validation plan.

Where the First Hour Usually Goes Wrong

The assessment looks simple on a slide. In practice a handful of patterns cause most of the trouble, and each one has a working countermeasure.

What to Do with This on Monday

For a quality leader, three moves stand out. Commission an inventory of the AI already inside the operation, and make sure it includes the assistants, dashboards and vendor features that were never labelled AI, because the real inventory is usually bigger than the one on paper. Put the four-role initial assessment into a procedure, so the next AI system meets a process and a shared vocabulary. And when a validation plan comes up for approval, read the worst-case reasoning first, because that reasoning is what protects the patient, and a conclusion without it is only an opinion.

For the people who run validation, the moves are just as concrete. Run the initial assessment on current systems, including the ones already live, and write the result into each validation plan. Check every system where a person reviews AI output for override capture, and pull the actual override data; if the rate has sat at zero for a quarter, treat it as a signal worth following up. Find out what happens to the results the model doesn't flag. Walk the exception paths and time them. Gate the pilots. And revisit the assessment at every periodic review against the triggers the team named, because the workflow it describes won't stand still.

All of it runs on one disciplined hour per system and the honesty to scope effort from consequence. That's the whole method, and it costs less than the first day of the remediation it prevents.

Objects in the Mirror are Closer Than They Appear

Every passenger-side mirror carries that etched warning, and I've started reading it as a regulatory forecast. A couple of years ago, AI guidance in GxP felt like something on the horizon. Look at the mirror now. US Food and Drug Administration’s (US FDA) Computer Software Assurance guidance is final, and it anchors assurance effort to risk. The draft EU GMP Annex 22, the first EU GMP text written specifically for AI, has been through public consultation, and its draft wording points to human oversight proportionate to risk and to performance at least as good as the process the AI replaces. And on 2 April 2026, US FDA issued the first warning letter citing AI use in drug manufacturing (320-26-58). The firm had let an AI agent draft its specifications, procedures and master production records and took the agent's word for what the regulations required. US FDA's answer was that AI output must be reviewed and cleared by an authorized person in the quality unit. The direction is consistent, and the distance is shorter than it looks.

The encouraging part is that everything the regulators are converging on is thinking GAMP has carried for more than 30 years, applied to a new kind of system. There's no need to wait for a final text. Inventory the AI and put 4 people in a room for an hour per system. Ask what happens to the patient when it's wrong, in both directions, and write the reasoning into the validation plan. The regulations in the mirror are closer than they appear, and that hour is still the cheapest distance a team can buy.