跳到正文
ISPE iSpeak 专家博客·· 18 天前AI 评分54

超越人在回路:机器在回路能否解决GxP验证中的覆盖失败?

Beyond Human-in-the-Loop: Can Machine-in-the-Loop Address Coverage Failure in GxP Validation?

AI 导读

GxP验证中的人在回路可能因覆盖失败而未能发现缺失的风险,ISPE专家博客文章据此讨论机器在回路的挑战架构。文章区分评估失败与覆盖失败,指出审查者容易核对可见内容,却难以识别AI分析中完全未出现的风险或测试条件。文章引用生物医学数据集开发中的机器在回路应用作为概念先例,但明确其能否改善验证策略仍需实证评估。

正文 · 原文

The authors make a compelling argument that it may not. Automation bias, cognitive ease, review fatigue, and inappropriate trust calibration can transform human review from an effective detection control into little more than formal approval step. A highly structured and plausible AI-generated output may be difficult to challenge precisely because it looks complete, coherent and convincing.

There may, however, be an additional failure mode that deserves consideration. The challenge is not only whether a human reviewer detects an error contained within an AI-generated output. Equally important is the possibility that the AI may fails to identify or include a relevant issue in the output at all.

For the purpose of this discussion, this phenomenon will be referred to as coverage failure.


From Evaluation Failure to Coverage Failure

Consider AI-supported development of a validation strategy. The AI identifies a set of risks, proposes corresponding test conditions, and presents a well-structured validation approach. A qualified expert subsequently reviews the strategy.

Two fundamentally different failures may occur.

In an evaluation failure, an incorrect or inadequately supported statement is present in the AI output, but the reviewer fails to recognize it. This is the situation to which automation bias, cognitive ease, and review fatigue directly apply.

In a coverage failure, however, a relevant risk, assumption, failure mode, or necessary test condition is absent from the AI-generated analysis altogether.

The distinction matters.

A reviewer can challenge an incorrect risk assessment that is visible. Detecting a missing risk is fundamentally more difficult because there is no explicit object to review. The reviewer must independently reconstruct the relevant problem space and recognize that something is absent.

This creates a potential limitation of conventional HITL architectures. If the AI generates the initial analytical frame and the human subsequently reviews it, the human may unintentionally inherit the boundaries of the problem space established by the AI.

The question therefore becomes not only:

“Is what the AI has produced correct and sufficiently justified?”

but also:

“Have the relevant risks, assumptions, and necessary tests been sufficiently identified and plausibilized according to the applicable risk definition—including those that are absent from the AI output?”

These are different cognitive tasks.

Plausibility Is Not Coverage

Structured AI outputs can be highly valuable. Tables, categorized findings, traceability matrices, discrepancy reports, and structured risk assessments may reduce cognitive load and make anomalies easier to detect.

Structure itself is therefore not the problem.

The problem arises when the quality of representation is interpreted as evidence of completeness.

A validation strategy may contain five well-reasoned risks, each linked to appropriate test conditions and evidence. All five assessments may be correct. Yet a sixth relevant risk may be missing entirely.

Reviewing the quality of the five identified risks does not establish that the relevant risk space has been adequately covered.

This is particularly important for generative AI systems. Such systems can produce highly coherent answers to a given task. Coherence, however, should not be confused with exhaustive exploration of the underlying problem space. An AI system may converge rapidly on a plausible interpretation and continue reasoning within that frame.

A human reviewer presented with the resulting coherent analysis then faces two tasks simultaneously: evaluating the proposed strategy and independently recognizing what the strategy failed to consider.

This suggests that some GxP applications may benefit from separating generation and challenge more explicitly.

Reversing the Direction: Machine-in-the-Loop

For certain tasks, an alternative architecture may be considered: machine-in-the-loop (MITL).

In this model, the qualified expert remains the primary generator of the validation strategy. The machine is introduced as a challenge function rather than as the primary generator whose work is subsequently approved by a human.

A possible workflow consists of three functionally distinct stages.

1. Expert-Led Strategy Generation

A qualified expert develops the initial validation strategy, including relevant risks, assumptions, test hypotheses, test conditions, controls, and expected evidence.

For complex problems, this stage does not necessarily have to be human-only. Critical dialogue between an expert and an AI system may support exploration of the problem. However, responsibility for defining the strategy and its rationale remains with the expert.

The objective is not to prohibit AI assistance, but to avoid automatically treating an AI-generated problem frame as the starting point that the human merely verifies.

2. Independent Machine Challenge

A separately configured AI system performs a challenge of the resulting strategy.

Its task is explicitly not to confirm that the strategy appears reasonable. Instead, it should search for omitted risks or failure modes, unsupported assumptions, insufficiently justified exclusions, inadequate test coverage, missing evidence, alternative explanations, boundary conditions, and counterexamples.

The design of this challenge function matters. Prompting and task design should discourage confirmatory behavior and sycophancy and explicitly require the system to search for omissions and counterexamples.

However, sycophancy is not the only concern. The independence of an AI challenge function should not be assumed merely because a separate agent or even a different model is used. Multi-agent systems may also be affected by peer identity bias, correlated reasoning, and failures of assigned role fidelity. Empirical work has shown that exposure to peer-model identity can influence inter-model evaluation and that assigned adversarial roles may not always be maintained reliably.

Consequently, instructing an AI agent to act as an “independent critical reviewer” does not demonstrate that it actually behaves as one.

Where practical, independence should therefore be considered an explicit design objective. Model diversity, independent contextual framing, controlled separation of intermediate results, different evidence-retrieval strategies, anonymization where appropriate, and deliberate adversarial task definitions may help reduce correlated failure modes.

Their effectiveness, however, should be demonstrated rather than assumed.

The value of an additional AI control layer therefore depends not primarily on the number of agents involved, but on the degree to which it contributes genuinely different information, challenges assumptions, and exposes disagreement.

3. Independent Resolution and Review

The challenges generated in the second stage are then evaluated by another qualified reviewer or an additional independently configured AI system.

Importantly, the purpose of this stage should not simply be another complete review of the original document.

Instead, disagreement becomes an explicit information object.

If the challenge system identifies a risk that was absent from the original strategy, the question is not immediately whether the human or the AI is “correct.” The divergence should trigger examination of the underlying assumptions and evidence.

A useful auditable structure could therefore be:

Challenge → Response → Resolution → Evidence

A challenged risk may ultimately be accepted, rejected, or addressed through an additional control or test. What matters is that the disagreement does not disappear silently into another approval step.

MITL Is Not Merely a Theoretical Concept

A MITL approach has previously been applied in biomedical dataset development.

Dietrich and Kazzer used a MITL approach during the development of a pharmaceutical and biomedical named-entity recognition corpus. During multi-stage annotation cycles, T5 models were trained on the current dataset, and false-positive and false-negative results from fivefold cross-validation were manually reviewed to identify incompleteness and inconsistencies in previous human annotations.

The machine did not replace the human annotators and was not assumed to provide ground truth. Instead, disagreement between human annotation and model output generated a signal for additional examination.

This provides a useful conceptual precedent for the present proposal:

Disagreement can be treated as information rather than merely as error.

Transferring this principle from dataset quality control to GxP validation-strategy development is, however, a hypothesis. The previous application does not demonstrate that MITL will improve validation strategies, nor that it is generally superior to HITL.

That question requires empirical evaluation.

MITL Does Not Solve the Completeness Problem

MITL should not create a new illusion of control.

Neither a human expert, an AI system, nor a combination of multiple humans and AI systems can demonstrate from their own analysis that every relevant risk has been identified.

Even a detailed human–AI dialogue may itself produce false confidence:

“We discussed the problem extensively, therefore we must have considered everything.”

The conclusion does not follow from the premise.

The objective of MITL should therefore not be to claim completeness. It is to increase the detectability of omissions, unsupported assumptions, and inadequate test conditions by introducing a challenge process that is at least partially independent of the original generation process.

Lifecycle and post-deployment monitoring remain necessary because no pre-deployment architecture can eliminate unknown or previously unrecognized failure modes.

From More Review to Different Functions

A central implication is that simply increasing the number of review layers may be less important than differentiating their functions.

Three sequential reviewers performing essentially the same task may reproduce similar assumptions and biases.

A more useful architecture may deliberately separate:

Generation → Challenge → Resolution

The first function constructs a validation strategy.

The second attempts to identify what the first function missed or inadequately justified.

The third evaluates the resulting disagreements and determines what additional evidence, controls, or tests are required.

Such an architecture can also be substantially automated. Automation, however, should not result in a pipeline in which several AI agents automatically generate, challenge, and approve one another until a “green” status is produced. That would risk recreating the original illusion of control at a higher level of technical sophistication.

The effectiveness of the architecture itself therefore needs to be demonstrated.

A Testable Question

The comparison between HITL and MITL can ultimately be formulated as an empirical question rather than a conceptual preference.

For validation-strategy development, controlled scenarios could contain known but non-obvious risks, unsupported assumptions, or inadequate test conditions. Different control architectures could then be compared, for example:

AI generation → human review (HITL)

versus

expert generation → independent AI challenge → independent resolution/review (MITL)

Relevant outcomes could include detection of intentionally seeded omissions, identification of additional valid risks, false-positive challenge rates, resulting test coverage, reviewer workload, and the persistence of unresolved disagreements.

Such experiments could also test whether the presumed independence of multiple AI agents actually exists—for example, by varying model identity exposure, model diversity, context separation, and assigned reviewer roles.

This would allow a more meaningful assessment than simply asking whether a human—or another AI agent—was “in the loop.”

Conclusion

The discussion initiated by Henrichmann and Herrmann is important because it challenges the assumption that human presence automatically creates effective control over AI-supported GxP processes.

An additional question should now be considered.

Human review can fail not only because reviewers accept an incorrect AI-generated statement, but also because relevant risks or assumptions never enter the AI-generated problem space they are asked to review.

For tasks such as validation-strategy development, it may therefore be useful to reverse the conventional control direction. Instead of asking only how humans can more effectively supervise machine-generated analyses, we should also ask when machines can serve as challenge functions within human-led analytical processes.

The relevant question may not always be how to keep the human effectively in the machine’s loop, but when the machine should instead be placed in the human’s loop.

Neither architecture guarantees completeness. But explicitly separating generation, challenge, and resolution—and validating the independence and effectiveness of those functions—may provide a practical and testable way to improve the detectability of what the original analysis failed to see.1, 2, 3, 4

来源:ISPE iSpeak 专家博客 · ispe.org