When not to automate
Every AI project has a moment where someone asks why the system still needs a human. It is usually the most important question in the room, and it is almost always answered too quickly.
A while ago we built a support platform for a utility company. The shape was ordinary enough: internal and customer-facing assistants over a retrieval layer, connected straight to the support mailbox, drafting a reply to every incoming request. The interesting part was where we stopped.
The operator stays the last step. They read the draft, rate it with a thumbs up or down and a line of free text, edit it, and send it. The ratings feed a dashboard. And continuous learning — the obvious next move, the one every vendor deck puts on slide four — was deliberately left out.
The constraint was not technical
The customer did not want the knowledge base reshaped implicitly by whoever happened to be on support that week. They wanted to own it: to know what was in it, who put it there, and when. That is not caution about AI. It is an entirely reasonable position about who is accountable for what the company tells its customers.
We could have argued. There are good counterarguments — human review gates, staged promotion, versioned knowledge, rollback. I have made all of them before. But the honest read of that room was that the argument would have cost more trust than the feature was worth, and that a system shipped without it would be used every day while a system shipped with it would sit behind a review process nobody had time to run.
An autonomous system nobody switches on is worth nothing. An assistive one in daily use is worth everything.
What you give up, and what you get
Without continuous learning, the system improves in batches instead of continuously. Somebody has to look at the dashboard, read the low-rated drafts, and decide what to change in the knowledge base. That is real work, and it is slower.
What you get is a system the organisation actually trusts, a knowledge base that stays legible to the people responsible for it, and a feedback signal that is honest — because operators rate freely when their ratings are not silently rewriting the source of truth behind their backs.
The pattern, generalised
The interesting question was never what the model can do. It is what an organisation can absorb. Absorption is a function of trust, regulation, accountability and habit, and none of those move at the speed of the technology.
So the useful move, when you hit a constraint like this, is to ask whether it is a misunderstanding you can correct or a position you should design around. Misunderstandings are worth an hour of explanation. Positions are worth an architecture.
In practice I now assume assistive before autonomous, and I keep the human at the last step until there is evidence — measured, not asserted — that removing them is safe. Not because I am cautious about AI. Because that is how it ends up being used.