Human-in-the-loop: the person in the system

Third and final post in the series on testing AI systems. In the first we built the yardstick, in the second we handed it to a model. The person remains.

The blurred parcel has already gone past. This quality check is well designed: there is a person, they have the authority to stop the line, they know what to look for. The question this post asks is what happens to that person in the third month, when the line has never once stopped.

There is a sentence you hear often in the meetings where a team decides whether to ship an LLM-based feature: "for now let us put a human in the middle". Put like that it sounds like a solution. In reality it is the start of a design problem, because "putting a human in the middle" can mean five different things, with completely different costs, guarantees and failure modes.

This post tries to sort that out: what human-in-the-loop actually means, which variants exist, when it makes sense, when it is theatre, and above all how you can tell whether yours is working.

The definition, and why it is not enough

Human-in-the-loop (HITL) denotes any architecture in which a person is a mandatory node in an automatic system's decision cycle. Without their intervention, the flow does not proceed.

The key word is "mandatory". If the system can carry on by itself, you are not in-the-loop. You are doing something else — which may be perfectly fine, but should be called by its own name.

The three variants, plus one

Human-in-the-loop. The AI produces an output or proposes an action. A person approves, edits or rejects it. Only afterwards does the action take effect. It is a blocking gate: the cost is latency and human labour, the guarantee is that no output reaches the world without passing under a second pair of eyes.

Human-on-the-loop. The system acts autonomously, a person supervises and can intervene, correct or stop everything. There is no prior authorisation, there is monitoring. The cost is far lower, but the guarantee is probabilistic: some errors reach their destination and get corrected afterwards.

Human-out-of-the-loop. Full autonomy. Control exists only after the fact: logs, metrics, alerts, periodic audits, and a channel for reports. It is not the absence of control, it is control moved onto the statistical plane instead of the individual case.

It is worth adding a fourth, less often cited but heavily used in practice:

Human-in-the-loop-by-exception. The system is autonomous by default, but certain conditions trip the gate: low confidence, an amount above a threshold, a customer in a given category, an irreversible action, an out-of-distribution input. It is the compromise that holds up best in production, and it is also the hardest to tune well.

These are not four equivalent labels to pick between by taste: they are four points on a scale running from total control to statistical control, and each costs and guarantees different things.

Another distinction that often gets muddled: where in the lifecycle

"Human-in-the-loop" is used to mean three different things depending on the point in the lifecycle:

They are different activities, with different people and different goals. When someone in a company says "we have HITL", the first useful question is: in which of the three phases? It often turns out there is one in training, none at runtime, and one in evaluation in name only.

In a series about testing, the phase we care about most is the third — but the second is the one that determines how much being wrong costs you, so we treat them together.

The concrete patterns

In practice HITL materialises in a handful of recurring patterns. Recognising them helps you avoid reinventing the wheel badly.

Approval gate

Every action with side effects requires explicit confirmation. It is the typical agent pattern: sending email, running commands, writing to a database, payments, deploys.

The decisive design variable is granularity. Confirming every single step of an agent that takes forty of them is the perfect recipe for automatic clicking. Confirming the plan once and then letting execution run is more sustainable, but exposes you if the agent deviates from the plan halfway through. The middle ground that works: plan approval plus a mandatory gate on irreversible actions only.

Confidence routing

Below a certain confidence threshold the case queues for a human, above it passes automatically. It is the most widespread pattern because it balances cost and risk with a single parameter.

The whole trick lies in the fact that the confidence an LLM declares is not calibrated. A model saying "I am 90% sure" is not right nine times out of ten. If you use a threshold, it has to be built on a calibrated signal: the probability of a classifier trained for the purpose, variance across several samplings, disagreement between different models, presence or absence of citations in the context. Not on the phrase "confidence: high" you asked the model to generate.

Sampled review (audit)

Everything passes automatically, but a percentage of outputs is inspected after the fact to estimate real quality.

Pure random sampling is almost always the worst waste: if the error rate is 2%, in a hundred cases you find two. Better a stratified sample: a random share (to get an unbiased estimate), plus a share targeted at the cases the system itself flags as doubtful, plus a share on rare or high-impact cases. The three shares answer different questions and should be kept separate in the reports too.

Suggestion instead of execution

The AI does not decide: it pre-fills. The person remains the formal author of the act. It is the dominant pattern in medical, legal and HR settings, and not only out of caution: it moves responsibility in a legally clean way.

Watch out, though, because it is also the pattern that generates the most automation bias (coming up). A pre-filled field is an extremely powerful cognitive anchor: the person is no longer deciding, they are looking for reasons not to change what is already written in front of them.

Escalation / arbitration

The system handles the normal case, the human handles the exception. It works well when "exception" can be defined by explicit rules, far less well when the definition is "when the AI got it wrong" — because that is exactly the information you do not have.

Feedback loop

Every human correction is recorded and reused: as a few-shot example, as a case in the golden dataset, as fine-tuning data, as an additional rule in the prompt.

This is the part almost everyone forgets to implement. A HITL that corrects outputs but does not collect them is a sticking plaster; a HITL that collects them is an improvement engine. The difference in value, over 12 months, is enormous.

When it is genuinely needed

The guiding criterion is the cost of the error, not the difficulty of the task. This is a frequent mistake: review gets put on the complicated tasks, when it should be put on the tasks whose consequences are expensive to undo.

The conditions under which a human gate is justified:

• Irreversibility. Sending, deleting, paying, publishing, deploying to production. If undoing costs more than confirming, put the gate in.
• Consequences for third parties. A refused credit application, a discarded candidacy, a diagnosis, a report. Whoever bears the decision is not whoever bought the software.
• Regulatory constraints. The AI Act (Reg. EU 2024/1689) requires under art. 14 that high-risk systems be designed to allow effective human oversight, with the capacity to interpret, monitor and override the output. The GDPR under art. 22 restricts decisions based solely on automated processing that produce legal or similarly significant effects.
• Identifiable accountability. In a dispute you need someone who signed. This is not bureaucracy: it is the condition under which the organisation can defend the decision.
• Young system. In the first weeks of production you have no idea of the real error rate. The gate is there to buy the data that will let you remove it.
• Out-of-distribution cases. Inputs never seen, unexpected languages, anomalous formats, ambiguous requests.

And the conditions under which it is counterproductive:

• High volume, low unit value. Categorising fifty thousand tickets. The cost of review exceeds the damage of the errors.
• Tight latency. Real-time fraud detection, live chat moderation, trading. The gate does not fit.
• The human is objectively worse. This happens more often than people admit, especially on boring, repetitive tasks where attention collapses after twenty minutes.
• No real power to intervene. If the reviewer cannot say no — because they lack the information, or because the process penalises them for slowing down — the gate is theatre. And it is expensive theatre, because it produces the documentation of a check that never happened.

A practical rule I use to decide: if the reviewer cannot, under any realistic circumstance, block the flow, then they are not a reviewer. They are a bureaucratic step, and it should be removed or redesigned.

The advantages (the real ones)

It catches the worst category of error. Generative systems do not fail noisily: they fail by producing plausible, well-written, wrong output. There is no exception, no stack trace, no alert. A competent reader is, as of today, the most reliable detector for this class of defect.

It produces accountability. There is a person who looked and said yes. It changes the conversation entirely when something goes wrong.

It generates the best data you will ever have. A human correction is a labelled data point in real context, on your domain, on a case the system actually got wrong. It is worth ten times an annotation done cold on a synthetic dataset. It is raw material for the golden dataset — the subject of the first post in this series.

It lets you start sooner. With a gate you can go to production with a system you would not trust to run free, and use the first months to gather the evidence needed to loosen it.

It covers compliance requirements in a verifiable way, provided it is documented.

The disadvantages (the ones nobody puts in the slides)

It does not scale

Cost grows linearly with volume. If your business plan projects 10x traffic, full-gate HITL projects 10x reviewers. That does not work, and sooner or later somebody "solves" it by raising the throughput required per reviewer — which is the elegant way of saying that review becomes fake.

Automation bias

This is problem number one, and it is psychological before it is technical. People tend to accept what the system proposes, especially if the system is right most of the time. The phenomenon has been studied for decades in aviation and medicine, with two distinct modes: errors of omission (I fail to notice what the system did not flag) and errors of commission (I uncritically adopt a wrong output). Time pressure amplifies both.

The bitter twist: the better the model, the less effective review becomes. A system that is right 99% of the time trains reviewers to approve without looking, and it is precisely in that 1% that attention was needed. The AI Act is aware of this to the point of explicitly citing the risk of over-reliance among the things the supervisor must be made aware of.

A reviewer who approves everything is worse than no reviewer: they cost money, and they produce the documented illusion of control.

An example you probably have in front of you every day. Agentic coding assistants such as Claude Code are, in effect, pure human-in-the-loop: before running a command, editing a file or installing a dependency, they ask for confirmation. On paper this is the correct design. In practice, anyone who has used them for a few days recognises the sequence: the first twenty requests are read carefully, the next two hundred become y, y, y, Enter, Enter. Not out of carelessness — for three structural reasons that stack up.

First, volume: a non-trivial task generates dozens of requests, and nobody maintains critical vigilance across dozens of consecutive decisions. Second, the signal-to-noise ratio: the overwhelming majority of requests are harmless (read a file, run the tests), so the brain learns statistically that the right answer is always yes. Third, and most interesting for anyone designing systems: the competence asymmetry. If the proposed command is sed -i 's/.../.../' $(grep -rl ... src/), the average user does not have the tools to assess it in two seconds. They can read it, they cannot judge it. And that is exactly the case the gate was supposed to be for.

The result is a gate that is formally present and substantially switched off — which is why these tools also offer granular permissions and per-category allowlists: not for convenience, but because removing the harmless confirmations is the only way to bring attention back to the ones that matter. It is the same principle as rule 5 further down: fewer gates, taken more seriously.

Three requirements for any HITL fall out of this example: the gate has to be rare, it has to be comprehensible to whoever receives it, and it has to show what changes, not just what will be executed.

Review fatigue

Related but distinct. This is not cognitive bias, it is tiredness. Review quality collapses with frequency and with repetitiveness. If an operator makes four hundred approvals a day, the last hundred are not reviews.

Moral crumple zone

The expression is Madeleine Elish's, and it is worth keeping in mind: the human in the loop risks becoming the crumple zone that absorbs the impact of the model's systemic errors. Formally responsible, materially without the time, the information or the authority to genuinely decide. When something goes wrong, the blame lands on them — and the structural defect of the system stays where it is.

If you are designing a HITL, this is the ethical question to ask: am I giving this person the tools to decide, or am I giving them the responsibility without the tools?

Inconsistency between reviewers

Two competent people, on the same case, decide differently. This is not a pathology, it is the norm: in the annotation literature an inter-annotator agreement (Cohen's kappa) around 0.6-0.7 is considered acceptable, which means a not-small share of disagreement is physiological even among experts. If you do not measure agreement among your reviewers, you do not know how much noise you are injecting into the system — and into the golden dataset that comes out of it.

Friction and latency

The gate shows up in the product. A user waiting for a human approval has a different experience, and that has to be designed too.

False security

The meta risk: the presence of HITL gets used as an argument for not investing in automated tests, monitoring and systematic evaluation. "There is review anyway." It is the justification under which systems stay unmeasured for years.

How to tell whether your HITL is working

This is the part missing from almost every implementation I have seen. A human gate is a system component like any other: it has to be measured.

Override rate. The percentage of cases in which the reviewer edits or rejects the proposal. It is the most informative metric and the most misread.

• Very low override (< 2%): either the model is excellent, or the reviewers are rubber-stamping. The two situations are indistinguishable from logs alone, and separating them takes something else (see below).
• Very high override (> 30%): the model is not ready for that task, or the prompt is wrong, or the reviewers and the model are using different criteria — and in that last case the problem is the rubric, not the model.

Blind audits. The way to distinguish the two cases above: periodically inject cases with known errors into the review flow and measure how many get caught. It is a test of the reviewer, not of the model. It has to be announced in advance to the people involved, otherwise it becomes surveillance; but it has to be done, because it is the only honest data on real vigilance.

Inter-reviewer agreement. Have two people review the same set of cases and compute the kappa. If it is low, you do not have a model problem: you have a definition problem. The rubric is ambiguous, and as long as it stays ambiguous any metric built on those judgements is noise.

Median time per review. If it falls over time it may mean the reviewers have become expert, or that they have stopped reading. Cross it with the override rate: time down and override down together is an alarm signal, not a sign of efficiency.

Post-gate error rate. How many errors got through anyway, measured by complaints, downstream corrections, incidents. It is the measure of the gate's actual value. If it equals the pre-gate rate, the gate is doing nothing.

Cost per review and cost per error avoided. The second is the number you need in order to decide whether to loosen the gate. If avoiding an error costs more than the damage that error causes, the gate should go — or move to only the cases where the ratio flips.

Designing it well: seven practical rules

1. Make review cheap. Show the diff, not the whole text. Highlight what changes, which sources were used, what happens if I approve. Every second of reading saved is attention left available for the hard case.
2. Show uncertainty where it is. A uniformly confident output invites uniform approval. If the system knows it is on slippery ground in one specific paragraph, it has to say so there, not in a generic note at the bottom of the page.
3. The default must not be "approve". No preselected button, no confirmation by pressing Enter. It saves time and destroys the value of the gate.
4. Ask for an action, not for consent. Having the reviewer select the reason for rejection, or tick the two or three things they verified, produces a qualitatively different attention from a single click on "OK". It costs a few seconds more and changes the nature of the task.
5. Reduce the gates, do not add them. Three gates taken seriously beat thirty rubber-stamped. Every gate you add lowers the value of all the others.
6. Close the loop. Every correction goes somewhere: dataset, prompt, rules, backlog. If corrections die in the database, you have a recurring cost with no return.
7. Treat autonomy as a parameter, not as a binary choice. Per category of action, not per system. "Drafts pass automatically, sends require confirmation, deletions require double confirmation" is a sensible policy. "The system is supervised" is not.

On that last point, a fragment that gets the idea across. The level of autonomy is a property of the action, not of the system:

from enum import IntEnum


class AutonomyLevel(IntEnum):
    AUTONOMOUS = 0       # executes immediately, logged for sampling
    NOTIFY_AFTER = 1     # executes, then notifies a human (human-on-the-loop)
    REQUIRE_APPROVAL = 2 # blocks until a human approves (human-in-the-loop)
    DUAL_CONTROL = 3     # requires two distinct approvers


def resolve_level(action, context, approval_threshold=0.85):
    # Irreversible actions never run unattended, regardless of confidence
    if action.is_irreversible and action.impact >= Impact.HIGH:
        return AutonomyLevel.DUAL_CONTROL

    if action.is_irreversible:
        return AutonomyLevel.REQUIRE_APPROVAL

    # Calibrated confidence, not the model's self-reported one
    if context.calibrated_confidence < approval_threshold:
        return AutonomyLevel.REQUIRE_APPROVAL

    if context.is_out_of_distribution:
        return AutonomyLevel.REQUIRE_APPROVAL

    if action.impact >= Impact.MEDIUM:
        return AutonomyLevel.NOTIFY_AFTER

    return AutonomyLevel.AUTONOMOUS

The code is not the point. The point is that this function has to exist, be readable by people who do not write code, and change when the data changes. In most projects this logic is scattered across six different places and nobody can say, at a glance, what the system is allowed to do on its own.

Three stories from the field

The review that approved everything. A flow generating answers to support tickets, with mandatory approval before sending. In the first two weeks the override rate was around 25%: operators fixed the tone, added details, occasionally rewrote from scratch. After two months it had dropped below 3%. The optimistic reading was "the model improved thanks to the feedback". The correct reading emerged when we injected twenty deliberately wrong answers into the flow: six were caught. The model had not improved much; the operators had learned that it was usually fine. The fix was not technical: the gate was narrowed to tickets involving contractual commitments or refunds, and on those attention went back up because the volume was manageable.

The disagreement that was not the model's. A document classification project in which the model looked unpresentable: around 40% of judgements "wrong" according to the reviewers. Before touching the prompt, we had the same sample of two hundred documents reviewed by two independent reviewers. They agreed with each other on 68% of the cases. The model did not have an accuracy problem: there was no shared definition of what the right answer was. Two days spent writing an explicit rubric with edge examples did more than the previous week of prompt engineering — and finally made it sensible to build a golden dataset on top.

The gate in the wrong place. An agent running a sequence of operations on company data asked for confirmation at every step. Twelve, fifteen confirmations for a single task. After a week users were clicking "approve" in rapid fire without reading — the behaviour was so mechanical that somebody was holding down Enter. The gate was everywhere and therefore nowhere. Redesigned: a single approval on the plan proposed at the start, free execution for read-only operations, a mandatory non-bypassable gate only on writes and deletions. From fifteen confirmations to two, and those two were actually read.

The common thread across all three: the problem was never the model. It was the design of the point at which the person was called on to decide.

The regulatory picture, briefly

Two references worth knowing even if you do not do compliance.

AI Act (Reg. EU 2024/1689), art. 14. It requires high-risk systems to be designed so that natural persons can effectively oversee them, being able to understand how they work, interpret the output, and override it or stop the system. The measures have to be proportionate to the risk and to the system's autonomy. Relevant for us: the rule explicitly requires whoever supervises to be put in a position to remain aware of the risk of over-reliance on outputs — that is, automation bias is a design requirement, not a UX detail. For remote biometric identification a double-verification principle applies to critical decisions, with delimited exceptions.

GDPR (Reg. EU 2016/679), art. 22. It restricts decisions based solely on automated processing that produce legal or similarly significant effects on a person, with a right to obtain human intervention. Mind the word "solely": purely nominal supervision, in which the operator ratifies with no real margin, does not take the processing outside the scope of the rule.

The shared message of the two rules is the same one that holds from an engineering point of view: facade supervision does not count. Which is convenient, because it means designing HITL well and designing it compliantly are largely the same job.

How to survive while you build it

• HITL is not one thing: distinguish in-the-loop, on-the-loop, out-of-the-loop and by-exception, and always say in which phase (training, runtime, evaluation).
• The question is not "how difficult is the task" but "how much does the error cost and how reversible is it".
• The typical failure is not the reviewer who gets it wrong: it is the reviewer who approves without looking. Design against that.
• Measure the gate: override rate, blind audits, inter-reviewer agreement, errors that got through anyway, cost per error avoided.
• Fewer gates, taken more seriously.
• Every human correction is raw material. If you do not collect it, you are paying twice.
• Autonomy is raised gradually and per category of action, on the basis of data the gate itself produced.

And above all: HITL is not a substitute for systematic evaluation. It is how you survive while you build it.

Where the series closes

If a human reviewing everything does not scale, the obvious temptation is to replace them with another model: that is what we saw in the second article, with the documented biases and the calibration against human judgement needed before it can be used as a metric. But a judge, automatic or human, still needs something to be measured against — and that is the golden dataset from the first article.

The three are not alternatives to each other. They are the three pieces of the same measuring apparatus, and anyone who skips one finds out late.

Sources

• Regulation (EU) 2024/1689 (AI Act), art. 14 — Human oversight: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=OJ%3AL_202401689
• AI Act Service Desk, summary of art. 14: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-14
• M. C. Elish, Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (2019)
• Landis & Koch (1977) and McHugh (2012) on interpreting Cohen's kappa
• Rosbach et al., Automation Bias in AI-Assisted Medical Decision-Making under Time Pressure: https://arxiv.org/abs/2411.00998
• Wu et al., Style Over Substance: Evaluation Biases for Large Language Models: https://arxiv.org/abs/2307.03025