Notes from inside the work.
Written when I have something specific to say, which is not often. Everything here comes from systems that are running, not from a conference slide.
This is not a blog. These are notes from actual engagements, written when a problem has recurred often enough to have a recognisable shape.
Three themes keep returning. The first is the boundary of autonomy — how much of it an organisation can absorb, and why the most autonomous system is rarely the right one. The second is measurement: how to test an LLM system without building an evaluation harness nobody runs twice. The third is the distance between a demo that works and a system running in production inside a regulated industry, which is almost always longer than it looks and almost never for technical reasons.
Client names are absent on purpose. What matters is the pattern, and the pattern generalises.
Human-in-the-loop: the person in the system
"Let us put a human in the middle" can mean five different things, with opposite costs and guarantees. The patterns, and how to tell whether review works.
Read →LLM-as-judge: AI evaluating AI
Using one model to evaluate another works, but only once you have calibrated it against human judgement. The variants, the documented biases, the real cost.
Read →Golden dataset and test set
Without a set of examples carrying the expected answer, "it works better" stays an opinion. How to build the yardstick that says whether you are improving.
Read →When not to automate
The most autonomous system is rarely the right one. Notes on designing AI around the constraint instead of arguing against it.
Read →How to test an LLM system
Test the arguments a model produces, not its prose. A cheap, deterministic regression suite for retrieval and tool calls — and why it also cuts the bill.
Read →Six months to hours
How a research body went from a six-month manual process for updating its guidelines to an agentic pipeline that runs in hours.
Read →