From traces to test cases

Fourth article in the series on testing AI systems. In the first we built the yardstick, in the second we handed it to a model, in the third the person stayed in the loop. One question is still missing, the one everything starts from: where do the cases come from?

The case nobody could reproduce

A user reports that the assistant answered badly on a question about an internal policy. The ticket arrives, somebody confirms the problem, the prompt gets fixed. So far, an ordinary day.

Three weeks later, during the monthly review, somebody remembers the rule we had set ourselves: every incident becomes a permanent case. They open the ticket to turn it into a row in the golden set, and find they cannot.

The document the context had been retrieved from has been updated in the meantime — that paragraph no longer says the same thing. The prompt template has changed twice. Nobody knows which model version was running that day, because the provider has since moved the alias. What is left is the user's question and a screenshot of the wrong answer.

Which is to say: what is left is an anecdote. And an anecdote is not a test, because there is no way to run it again and see whether it goes better today.

The case had not been lost. It had never been recorded in a form that allowed it to be lived again.

A trace is not a test case

The two objects look alike enough to suggest that one is the compressed format of the other. They are not, and the difference is what makes the crossing hard.

The row that matters is the first. A trace records a behaviour; a test case declares an expectation. Between the two sits a judgement, and that judgement is exactly what the system does not possess — if it did, you would not need to test it.

This is also why "generate your evals automatically from traces" is a promise to read carefully. Generating the candidates from traces is doable, and useful. But it is the easy step, and it gets sold under the name of the hard one.

Instrumenting to replay is not instrumenting to debug

Almost every production system already has instrumentation. The problem is that it was written to answer different questions.

Whoever instruments for debugging wants to know where it broke and what it cost: latency per step, tokens, errors, throughput. Useful material, and none of the fields you need in order to replay a case ends up in there by accident.

Here is the list that does, with the reason for each.

The resolved prompt and the template, kept separate. The resolved prompt alone tells you what the model saw but not what changed when the behaviour changed. The template alone is the opposite: you know what you wrote, not what arrived. You need both, with the template version alongside. It is the difference between "the answer got worse" and "the answer got worse once we added that sentence to v7 of the template".

The tool call arguments. This is where the two ends of the series touch. The assertions from the article the series grew out of — checking that the model produced the right retrieval filters, without ever running the query — are exactly what a well-instrumented trace already contains. If you record the arguments, that test writes itself from production rather than by hand out of a laboratory imagination. It is the single highest-return field on this list.

The identifiers and version of the retrieved chunks, not the text. Saving the text looks like the prudent choice and is the worst of the three available: it bloats the spans, it drags in personal data you do not need, and above all it is a copy. With a copy you never notice that the document underneath has changed: the case keeps running against a frozen context that no longer exists anywhere. With identifier plus version you see it immediately.

The model version and the sampling parameters. Temperature, top-p, and the seed if you use one. Without these the comparison against the baseline is between two different things, and you do not know it. Watch out for provider aliases: gpt-something-latest is a field that changes meaning underneath you without a single one of your files being modified.

The feedback signal, attached to the trace. Not in a separate ticket, not in a side table with a timestamp you then hope to trace back from. It is the difference between "we collect feedback" and "the feedback is usable": in the second case a thumbs-down takes you in one click to the input, the context and the arguments that produced it.

There is an asymmetry worth keeping in mind, and it is the same one as with logs: what you do not record at the time, you do not recover later. Nobody has ever reconstructed a missing trace. That makes choosing the fields a heavier decision than it looks, because the cost of getting it wrong is paid months later, when you need a case and it is not there.

What has to be versioned

A case is reproducible only if everything it depends on is identified. In LLM systems the things that move underneath a test are five: the document corpus, the index, the tool schema, the prompt template, the model.

If any one of them moves without your knowing, the test keeps running and keeps producing a number. It does not fail: it measures something else. It is the worst kind of silent failure, because it does not produce an error to look at, it produces a wrong metric with a straight face.

The first article had a rule: a change in the numbers must be attributable either to the system or to the dataset, never to both at once. Here the rule needs extending: never to three things at once. If the score drops and in the same week you updated the knowledge base, the conversation about what happened lasts days and almost always ends in a hypothesis.

The uncomfortable part has to be said too: full pinning is often not practicable. You do not freeze the production corpus in order to run tests, and in many cases you do not even control the model versions your provider serves you. The pragmatic answer is not to give up, it is to record the version even when you cannot freeze it. It does not give you reproducibility, it gives you attribution — which is 90% of the value and costs one field.

Which traces become cases

There is a decision to take here, and the default choice is the wrong one.

Random sampling is almost always the worst waste, for the same reason already seen in the third article about sampled review: if the error rate is 2%, out of a hundred cases drawn at random you find two, and you have spent human attention on ninety-eight cases that were fine. What you need is stratified selection, built on different signals that answer different questions.

Explicit failure. Thumbs-down, tickets, escalations, human corrections. It is the best source and the scarcest: users report a small fraction of what goes wrong, and they report it when it is serious. Every item in this category deserves to be looked at.

Implicit failure. The user rephrases the same question a different way, repeats, abandons halfway. It is a noisy signal — sometimes they simply changed their mind — but it is as abundant as the other is scarce, and almost nobody collects it. A rephrasing within seconds of the answer is, in most conversational systems, the cheapest way to know the answer was not useful.

Disagreement. The cases where the automated judge and the person give different verdicts on the same output. They count double, because they do not only put the system in question: they put the rubric in question. The third article described a project where two independent reviewers agreed with each other on 68% of the cases, and the conclusion was that the problem was not the model but the definition. These cases are how you find those situations before they turn into a month of pointless prompt engineering.

Novelty. Inputs far from anything the golden set already contains. It is the signal that attacks the gap the first article admitted it could not cover — two hundred examples do not describe the long tail of what users actually ask. You will not cover it anyway, but you can at least notice when a zone of traffic appears that you have no measurement for.

Cost and latency outliers. They are not quality errors and should be kept separate from the others, but they are real regressions. The article the series grew out of had already established that in these systems cost is part of the conversation about quality, not part of a different conversation.

The trap

If you select only on failures, you build a set that describes yesterday's bugs.

The mechanism is insidious because it produces improving numbers: the set contains the cases that were broken, you fix them, the score goes up. Meanwhile ordinary traffic — the kind that worked and that nobody ever reported — is not represented anywhere, and if it degrades nobody notices until the users say so.

You need a fixed quota of ordinary traffic that passes. It is the same logic as the first article's holdout, applied to the composition of the set rather than to its use: a structural defence against a drift nobody commits on purpose.

Promotion is a human act

There is a shortcut that occurs to everyone sooner or later: take the production output and promote it to expected output. It is the fastest way to fill a golden set, and it takes five minutes for a thousand cases.

It is also the way to freeze current behaviour and call it truth. The first article already has the vocabulary for it: what you get is a silver dataset wearing a golden label, and the risk is confusing "the model is consistent with itself" with "the model is right". From that moment every regression against today's behaviour reads as an error, corrections included.

The division of labour is this. The trace supplies the input, the context, the actual output and the metadata: all the bulky, boring part, for free. A person adds three things that are nowhere in the trace — the acceptance criteria, the category, and the judgement that this case is worth keeping forever.

It is two or three minutes per case. That is exactly why the golden set stays small, and that is a good thing: a set of two hundred cases somebody thought about one by one is worth more than one of ten thousand produced by a script.

It is worth noting that this is one of the human-in-the-loop evaluation activities described in the third article, and it inherits their problems: if two people write different criteria for the same case, the disagreement enters the golden set and from there every measurement you build on top of it. The kappa among the people promoting cases is an unromantic and very informative metric.

What you can use today

The practical question is whether this flow is bought or written. The answer is that you buy the boring half and write the half that counts, and they are exactly the two halves this piece has been separating all along.

The step from trace to case is already a button. On Langfuse it is called + Add to dataset and it is available on any observation of a production trace; there is a bulk version too, selecting rows from the observations table and mapping the fields. The item that comes out keeps the source_trace_id, so the provenance the eighth rule asks for is not something you write by hand: it is a field of the product. On LangSmith the button is called Add to Dataset and starts from a filtered selection of runs. These are the same tools you will find further down among the OTLP exporters: the trace you write once serves both for looking and for promoting.

Human review has somewhere to happen too. Langfuse's annotation queues are the queue from the previous paragraph, turned into a product: you create one by choosing the scoring dimensions, traces enter it one at a time or in bulk, and the reviewer sees one task at a time with scores and comments, navigable from the keyboard. This is not a cosmetic detail. Two minutes per case multiplied by twenty cases a month is the difference between a practice that survives and one that collapses in the first busy week.

What none of these products decides is the judgement. The acceptance criteria, the category, and whether that case is worth keeping forever are in no field of the trace, and there is no button that produces them. The division of labour described above does not change because you bought a tool: what changes is only that the bulky part stops costing you time.

And there is a shortcut they now offer you themselves. LangSmith states that its engine can automatically generate ground truth examples from production traces. It is worth rereading the paragraph on promotion with that sentence in front of you: generating the candidates from traces is useful and is the easy step, promoting the production output to expected output is the way to freeze current behaviour and call it truth. The button does not distinguish the two. You have to, and the distinction is the whole value.

If you do not want a product, you do not need one. The five fields in this article can be recorded with any tracing library, stratified selection is a query on your store, and promotion is a text file in git with a review on top. It is the route I recommend to anyone with a single system: you will buy the platform when you have three systems and no appetite for maintaining three versions of this.

The payoff: measuring how far the set has drifted from reality

So far we have talked about how to feed the golden set. The part that makes this work something more than a convenience is a different one, and it concerns maintenance.

In the first article the maintenance problem was admitted and the solution was discipline: a fixed share of every monthly review dedicated to realigning the set with real traffic. In the same article, a few lines later, there was also the honest sentence: maintenance is the first thing to go when there is a deadline.

With the category distribution of the traffic on one side and that of the golden set on the other, that discipline becomes two numbers.

The coverage gap. Categories that weigh in the traffic and are under-represented in the set. It is the alarm that, without this measurement, arrives in the shape of unhappy users six months later — with an excellent score on the dashboard, because the set was measuring well something nobody was asking for any more.

The dead cases. The opposite: cases in the set that no longer correspond to anything anyone asks. The first article had a rule number seven, «Date it and prune it», which was an intuition: if a case has been in there two years and always passes, perhaps it is not measuring anything any more. With the traffic distribution it stops being an intuition and becomes a decision you can defend in a meeting.

The difference between the two versions is not technical, it is organisational. A metric survives a deadline; goodwill does not. It is the same reason why, in the first article, the CI gate survived only once it stopped being a block to endure and became something you could argue with.

Privacy, concretely

In the first article the GDPR was one line: examples must be anonymised before they end up in a repository the whole team can read. When the source becomes real traffic the question stops being one line.

The content of prompts and completions is personal data the moment the user writes something into it, and this is not a textbook hypothesis: it is the normal case. People paste emails, case numbers, client names, and in healthcare or legal contexts far more delicate things. The moment that content enters a span, it has entered the observability system, which typically has a far wider access surface than the production database.

Four practical consequences.

Content capture should be treated as opt-in, not as the default. It is also the choice OpenTelemetry made: by default the GenAI conventions do not capture prompt content or tool arguments, precisely because they can contain sensitive data, and an explicit flag is needed to turn them on.

Redaction belongs in the collector, not in the application. If you do it in the application, every service has to remember, and sooner or later one of them forgets — usually the one written in a hurry. In the collector it is a single point, verifiable, that applies to everything passing through.

Trace retention and golden set retention are two different policies. The first expire after days or weeks, the second is append-only and lives for years. The moment of crossing from one to the other is the point where anonymisation must already have happened, because after that there is no expiry left to save you.

The golden set stays private. In the first article the argument was benchmark contamination: a public set is a burnt set. When the cases come from real traffic that reason adds to the one about data protection, and the two together do not leave much room.

What OpenTelemetry gives you and what it does not

It is worth being precise, because this is a field where the marketing runs ahead of the substance.

What it gives. A trace and span model that fits agent graphs well: one span per node, the parent-child relationship that reconstructs the path, the context propagating across calls. A neutral export format, which means that where you look at your traces is a reversible decision. And the fact that the specialised tools in this space — Langfuse, LangSmith and the others — are OTLP exporters: you can use them as an implementation without adopting their conceptual model and vocabulary.

What it does not give. The decision about what is correct. No semantic convention will ever tell you whether that answer was any good. Everything downstream of the trace — the selection, the criteria, the judgement — remains your work, and it is the work that counts.

And the part to know before leaning on it. The GenAI semantic conventions are still in Development: as of mid-2026 no GenAI-specific span, metric or attribute is marked stable. In 2026 they were moved into a dedicated repository, which at the moment has neither releases nor tags — meaning there is no version to pin against. And there have been renames, not small ones: 1.37.0 replaced the per-message events with the attributes gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions, and renamed gen_ai.system to gen_ai.provider.name; 1.27.0 had already changed the token naming. The practical result is that frameworks emitting different generations of the convention coexist in the same system, and queries have to be written accounting for that.

This is not an argument against using them: it is an argument against building the architecture of your own thinking on top of them. The question in this article — what has to be in the record for it to be reusable — stays valid after the next rename. The attribute names do not.

(State verified in September 2026. If you are reading much later, check: this is exactly the kind of information that ages silently.)

Eight practical rules

1. Record the template and the resolved prompt, separately and versioned. One of the two alone is worth nothing.
2. Record the context identifiers, not the text. The text is a copy, and a copy hides the changes underneath it.
3. Attach feedback to the trace, not to a ticket. Feedback detached from the execution that produced it is an opinion, not data.
4. Do the redaction in the collector. Because the application, sooner or later, will forget.
5. No production output becomes expected output without a person deciding it. The shortcut costs more than it saves.
6. Select on five signals, not only on failures. With a fixed quota of traffic that passes, or you are describing yesterday's bugs.
7. Measure the divergence between the set's distribution and the traffic's. It is the only form of maintenance that survives a deadline.
8. Every case carries its own provenance. Trace identifier, date, the signal that selected it. The first article was already asking for it; now there is a way to honour it.

Where the series closes

Up to here this series was a ladder. Move the assertions to where they cost little, build the yardstick, decide who holds it, keep a person at the point where the error costs. Four rungs, each slower and more expensive than the one below.

With this piece it becomes a loop. Production generates the traces, the traces feed the cases, the cases measure production. And the point where the loop closes is not a tool: it is a person looking at a real case and deciding what would have been correct — which is the third article, read differently at this point.

One thing about the order. This article should have been the first: without traces you have no real failures to start from, and "start from real failures" was the first rule of the first piece. It arrives last, which is exactly the order it happens in on real projects — observability goes in when it is needed, that is, when it is already late.

What remains is the practical advice, and it is less ambitious than the word observability suggests: you do not need a platform. You need five fields recorded well, a place where feedback attaches to the execution that produced it, and somebody who once a month looks at twenty cases and promotes three.

Sources

• OpenTelemetry, Inside the LLM Call: GenAI Observability with OpenTelemetry — https://opentelemetry.io/blog/2026/genai-observability/
• OpenTelemetry, dedicated repository for the GenAI semantic conventions — https://github.com/open-telemetry/semantic-conventions-genai
• John Hodge, The state of the OpenTelemetry GenAI semantic conventions (July 2026) — https://john-hodge.com/blog/opentelemetry-genai-semantic-conventions/
• A Practical Guide for Evaluating LLMs and LLM-Reliant Systems — https://arxiv.org/abs/2506.13023
• Hamel Husain, Your AI Product Needs Evals — https://hamelhusain.substack.com/p/evals
• Langfuse, Datasets and Annotation Queues — https://langfuse.com/docs/datasets/overview
• LangSmith, Manage datasets — https://docs.langchain.com/langsmith/manage-datasets