Notes from inside the work.
Written when I have something specific to say, which is not often. Everything here comes from systems that are running, not from a conference slide.
This is not a blog. These are notes from actual engagements, written when a problem has recurred often enough to have a recognisable shape.
Three themes keep returning. The first is the boundary of autonomy — how much of it an organisation can absorb, and why the most autonomous system is rarely the right one. The second is measurement: how to test an LLM system without building an evaluation harness nobody runs twice. The third is the distance between a demo that works and a system running in production inside a regulated industry, which is almost always longer than it looks and almost never for technical reasons.
Client names are absent on purpose. What matters is the pattern, and the pattern generalises.
Encoder, experts and memory
What is actually under the open models your client names in a call, and the four questions that decide the hardware bill.
Read →Context is a budget
As context grows, quality drops, and nothing tells you. Why compaction is not enough, what a handoff changes, and why the number belongs on screen.
Read →From traces to test cases
A trace says what happened. A test case says what should have happened. What has to be in the record for the first to become the second, six months later.
Read →Human-in-the-loop: the person in the system
"Let us put a human in the middle" can mean five different things, with opposite costs and guarantees. The patterns, and how to tell whether review works.
Read →LLM-as-judge: AI evaluating AI
Using one model to evaluate another works, but only once you have calibrated it against human judgement. The variants, the documented biases, the real cost.
Read →Golden dataset and test set
Without a set of examples carrying the expected answer, "it works better" stays an opinion. How to build the yardstick that says whether you are improving.
Read →When not to automate
The most autonomous system is rarely the right one. Notes on designing AI around the constraint instead of arguing against it.
Read →How to test an LLM system
Test the arguments a model produces, not its prose. A cheap, deterministic regression suite for retrieval and tool calls — and why it also cuts the bill.
Read →Six months to hours
How a research body went from a six-month manual process for updating its guidelines to an agentic pipeline that runs in hours.
Read →