Golden dataset and test set

First article in the series on testing AI systems. Before deciding who evaluates — a model, as in the second article, or a person, as in the third — there is a question sitting underneath both answers: against what, exactly, are we measuring?

A scene I have watched too many times

An alignment meeting on a RAG-based assistant. Someone changed the system prompt to close a reported issue. The product owner asks: "fine, but does it work better or worse than before?".

Silence. Then the answer we all know: "it looks better to me".

The problem is not the good faith of whoever answers. The problem is that there is no way, in that room, to tell "I fixed one case and broke three" from "I fixed one case, full stop". Traditional software solved this thirty years ago with regression tests. In generative systems the concept is the same, but the ground truth is something you have to build by hand.

That hand-built ground truth is called a golden dataset.

What it actually means

A golden dataset (or gold standard, or golden set) is a curated collection of examples in which the expected output — or at least the acceptance criterion — is treated as reference truth.

The term comes from diagnostic medicine: the gold standard is the test that by convention is assumed correct, and against which every cheaper or faster test is measured. The substance is identical: it is the reference you decide to treat as true, in full awareness that this is your decision and not a fact of nature.

Golden dataset ≠ test set

The two terms get used as synonyms, but the operational difference matters:

In classic machine learning you do 70/15/15 and move on. The test set inherits the quality of the dataset: if 5% of the labels are wrong, your test set has 5% wrong labels and your measurable accuracy ceiling is 95%.

A golden dataset is a different object. It is written or selected with the explicit intent of answering specific questions: does this system handle ambiguous questions well? Does it respect the format? Does it refuse when it should refuse? It is small because every example cost human attention, and it is useful for exactly that reason.

What it looks like in practice

In the simplest case it is a list of input → expected output pairs. In LLM systems it almost always turns into something richer:

- id: REF-014
  category: refund_outside_window
  input: "bought it on january 3rd, can i still send it back?"
  expected_context: [returns_policy_v3, holiday_exceptions]
  criteria:
    - must cite the 30-day limit
    - must not promise a refund
    - must point to the support channel
  notes: "edge case, the date falls inside the extended Christmas window"
  origin: production_ticket_2025-02

Note there is no word-for-word expected answer. For generative tasks exact comparison makes no sense: what you fix is a set of acceptance criteria, later verified by an automatic judge, by a deterministic check, or by a person. The golden dataset defines what has to be true, not how it has to be written.

Five criteria for a dataset worth having

Recent literature carries a useful mnemonic, the 5 Ds (Practical Guide for Evaluating LLMs). I repeat it here because it is an honest checklist:

1. Defined scope. One dataset per component, on top of the end-to-end one. If the retriever and the generator each have their own set, when the number drops you know where to look. It is unit-test reasoning applied to a pipeline.

2. Demonstrative of production. Built on real traffic, not on what you imagined users would ask. It is the most violated criterion of them all. A dataset generated from the documentation is too clean, too well written, too grammatical. Real users type in lowercase without punctuation, change their mind mid-sentence, paste fragments of email.

3. Diverse. Topics, intents, difficulty, languages, adversarial cases. If every example is a happy path, the score will sit at 95% forever and tell you nothing.

4. Decontaminated. It must not overlap with training or fine-tuning data. I come back to this shortly, because it is the most treacherous point.

5. Dynamic. A dataset assembled in January describes January traffic. By July users are asking something else and the product has new features. Treat it as an append-only log, with a date on every item, so you can see at a glance how stale it is.

Where the examples come from

Three sources, and you need all three.

Production. The best source, by a distance. Real traces, support tickets, conversations that ended badly. Every user report is a candidate to become a permanent test. Mind the GDPR: examples have to be anonymised before they land in a repository the whole team can read.

Domain experts. Needed when the task requires knowledge the model does not have and you cannot find in the logs: sector regulation, internal procedures, edge cases that occur twice a year and cost dearly. This is the slow, expensive part, and it is the part that gives the dataset its value. Anyone doing annotation seriously also measures inter-annotator agreement (Cohen's kappa): if two experts disagree on a case, that case is not ready to enter the golden set — either it needs reformulating, or the acceptance criterion is ambiguous and the problem sits upstream.

Synthetic generation. Useful to fill gaps: you have three examples of a rare category, you generate twenty variations. Recent literature separates silver datasets (labels produced by an LLM) from golden ones (verified by humans), and even from super-golden ones (curated by separate expert teams). The distinction is healthy: keeping them apart, and always knowing which one you are looking at, stops you confusing "the model agrees with itself" with "the model is right".

The mix that works in practice: a core of real production cases, enriched with cases written by experts on the known sore spots, expanded synthetically only where real data is too thin.

How big it has to be

The question always comes, and the answer always disappoints: far smaller than you think.

The orders of magnitude found in recent practical guides agree with each other:

• ~50 examples — enough to catch large regressions. This is the fast gate to run on every pull request.
• ~200 examples — you start having statistical confidence on differences of 3-5%. This is the threshold where the number stops being anecdotal.
• beyond 500 — diminishing returns, unless the system has very different sub-tasks that deserve separate sets.

For deterministic tasks (classification, structured extraction) the numbers go up, because evaluation there costs almost nothing and you can afford thousands of cases.

The real point is that provenance matters more than size. A hundred cases taken from real failures are worth more than two thousand generated by an LLM from the user manual. The second dataset will give you prettier numbers and zero information.

How it actually gets used

Three levels, with different costs and cadences. The ladder is the one proposed by Hamel Husain and picked up almost everywhere since, and it has the merit of being realistic:

Level 1 — Deterministic unit tests. Assertions that run in milliseconds on every commit: is the JSON valid? Is the mandatory field there? Does the answer contain the required disclaimer? Did it go past 500 tokens? They are trivial, they cover more cases than people expect, and they cost nothing. Many teams skip them to run straight at the judge, then pay in latency and tokens for what a regular expression would have given them.

Level 2 — Golden dataset plus evaluation (judge or human). Runs on the pull request or on the nightly build. This is where most of the signal lives: change the prompt, change the model, change the chunking strategy → rerun the set → look at the delta against the baseline.

Level 3 — A/B testing in production. Real traffic, meaningful samples, useful only once the product is mature and the first two levels are stable.

Anyone starting at level 2 or 3 without level 1 loses the fast feedback loop that makes the slow levels bearable.

On the CI gate: common practice is to block the merge if a primary metric degrades beyond a threshold (5% is the figure you see most often). And an important note, one that sounds heretical to anyone coming from traditional software: the pass rate should not be 100%. It is a product decision. A system passing 100% of the golden set is probably telling you the golden set is too easy.

What it gives back

Reproducible signal at almost no cost. A human evaluation over 200 examples costs person-hours, and you will not redo it for every comma changed in the prompt. A golden set runs in two minutes and you can launch it twenty times a day.

Discussions become objective. People stop arguing by impression. "I lost 4 points on the refunds category" is a sentence you can work with; "it feels worse to me" is not.

You see the invisible regressions. The ones the eye misses because you were looking elsewhere. It is the benefit you only appreciate the first time it saves you.

Diagnosis, not just a grade. If the dataset is stratified by category, the result tells you where you are failing. An aggregate 90% can hide a 40% on the under-represented critical category — and that is almost always the one that will hurt.

Governance. In regulated contexts (the AI Act, supervised sectors) having documented, versioned evidence of how the system was evaluated before each release is not a flourish, it is audit material.

The drawbacks, which are the interesting part

Goodhart's law

When a measure becomes a target, it ceases to be a good measure.

If you iteratively optimise against the same 100 examples, after twenty iterations you have a system that excels on those 100 examples. Not necessarily on the rest of the world. It is overfitting done by hand, by the team, without gradient descent: you look at the failures, adjust the prompt to fix them, rerun, repeat. The number goes up and all of it is true — on the cases you are looking at.

The mitigation is disciplinary, not technical: keep part of the set as a blind holdout, never looked at during development and used only for final validation. If the development set climbs and the holdout does not, you have your answer.

Contamination

This is the problem that made public benchmarks nearly unusable as absolute measures. If the evaluation examples ended up in the model's training data, the numbers measure memorisation, not capability.

The cleanest experiment on this is Scale AI's GSM1k: they commissioned a thousand grade-school maths problems built to mirror the distribution, difficulty and answer statistics of GSM8k, the industry standard benchmark. Then they re-evaluated the models. Some families (Phi, Mistral, some Llamas) lost up to 13 percentage points; frontier models lost almost nothing. And the gap correlated with how likely a model was to reproduce GSM8k text verbatim. Translation: part of that score was recall.

A detail that says everything: they decided not to release GSM1k publicly, beyond around fifty samples, precisely so as not to burn it.

The lesson for anyone building products: your golden set has to stay private, and public benchmarks should be read as coarse hints, not as measurements. If your dataset ends up on GitHub in the clear, know that you have started a countdown.

The gold standard is not gold

This is the most uncomfortable point. The assumption "the reference label is correct" is, precisely, an assumption.

MMLU is probably the most cited benchmark in recent LLM history. A manual analysis of 5,700 questions carried out by fourteen experts estimated roughly 6.5% of questions to be wrong, with absurd peaks: in the virology subset over 57% of the examples analysed had some problem, and 30% had outright wrong ground truth. Re-evaluating the models on the corrected dataset, the ranking changes.

If that happens to an academic benchmark on which half the industry's narrative was built, imagine what is in your CSV file put together over two afternoons.

The practical corollary: when a golden set case fails, the first hypothesis is not always "the model got it wrong". Sometimes the model is right and the label is wrong. It is worth revisiting the cases that have been failing systematically for months: every so often you find a mistake of your own, fossilised, that the system has been calling out all along.

Limited coverage and false confidence

Two hundred examples do not cover the long tail of what real users ask. "The golden set passes at 94%" while in production people find ways of breaking the system nobody had imagined — because nobody can imagine them all, and that is exactly why sampled review and online monitoring exist.

Maintenance cost

It has to be updated when the domain, the product or the regulation changes. A golden set frozen two years ago measures a problem that no longer exists, with the calm confidence of someone who does not know they are out of date. And maintenance is not fun: it is the first job to go when there is a deadline.

The ambiguity of "correct"

For classification it is easy. For "summarise this document" or "write this email" there is no single correct output, and string-by-string comparison does not work. You move to criteria, rubrics, semantic similarity, LLM-as-judge — which brings its own biases and its own variance, and which in turn has to be validated against human judgement. This is where this article connects to the next one in the series: the golden set is also the judge's test bench, not only the system's.

Lessons from the field

The fake dataset that scored 97%. On a ticket classifier we had built the evaluation set starting from the official category taxonomy: for each category, a few examples written by us. Result: 97%, and straight to production. In production, perceived accuracy was around 60%. The reason was obvious in hindsight: our examples were written by people who already knew the taxonomy, so they contained the right keywords. Users wrote "nothing has worked since this morning". We rebuilt the set from 300 real tickets, the score collapsed to 61%, and that was the moment we started understanding something.

The case that had been failing for eight months. An inherited test case, red forever, that everybody stepped over because "that one is known". When we finally opened it, the expected output was wrong: someone had written it misreading an internal policy. The model had been right for eight months. Hence the rule: no test stays red for more than two sprints without an explicit decision — you either fix it or delete it.

The set that became the product. On a document assistant the golden set had grown to 400 cases, all built around one particular kind of question we had fixated on. Six months later the score was excellent and the users were unhappy: in the meantime traffic had shifted to another kind of question, represented in the set by four examples. The dataset described our past, not their present. Since then a fixed share of every monthly review goes to realigning the set with the real distribution of questions.

The gate nobody respected. We had set the CI block at 5% regression. In the first weeks it fired often, and every time the reaction was "yes but it is a false positive, unblock it". The gate survived only once we added two things: the list of the specific cases that had changed, not just the aggregate number, and the ability to explicitly approve a regression with a stated reason. A gate you can only suffer gets switched off; a gate you can argue with stays.

How to build one that does not die after three months

An operational checklist, in order of importance:

1. Start from real failures. Not from the documentation, not from the taxonomy, not from imagination. If you do not have production yet, start from the support calls or from the cases domain experts tell you about through gritted teeth.
2. Version it like code. In git, with a changelog. A change in the numbers has to be attributable either to the system or to the dataset, never to both at once.
3. Put metadata on everything. Category, difficulty, date of entry, origin. You need them to stratify results and to see how much it has aged.
4. Keep a holdout you do not look at. Even a small one. It is the only real defence against human overfitting.
5. Include the negative cases. Ambiguous inputs, out-of-domain requests, prompt injection, questions the system must refuse to answer. A system answering confidently where it should stop is failing, however well written the answer is.
6. Every incident becomes a case. This is the rule that keeps the dataset alive without requiring willpower: if something generated a ticket, it enters the set. Forever.
7. Date it and prune it. If a case has been in there two years and always passes, perhaps it is no longer measuring anything.
8. Do not use it alone. The golden set covers "we know what should happen". It belongs alongside sampled human review and production monitoring for everything else.

Necessary, and far from sufficient

The golden dataset is your regression suite, not your QA. It tells you whether you broke something you knew had to work. It does not tell you whether the product is good, it does not tell you what you failed to foresee, and it stops telling you anything the moment it becomes the target to maximise instead of the measure to consult.

It is necessary. It is a long way from sufficient. And it is the thing that, in an AI project, is worth building first — because without it every other technical decision comes down to "it looks better to me".

Sources

• Zhang et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k, Scale AI) — https://scale.com/research/llm-performance-grade-school-arithmetic
• Full version of the GSM1k paper (NeurIPS 2024, Datasets and Benchmarks Track) — https://proceedings.neurips.cc/paper_files/paper/2024/file/53384f2090c6a5cac952c598fd67992f-Paper-Datasets_and_Benchmarks_Track.pdf
• Leaderboard and dataset description for GSM1k (Scale Labs) — https://labs.scale.com/leaderboard/math
• Gema et al., Are We Done with MMLU? (MMLU-Redux, NAACL 2025) — https://arxiv.org/abs/2406.04127
• Version published on the ACL Anthology — https://aclanthology.org/2025.naacl-long.262.pdf
• A Practical Guide for Evaluating LLMs and LLM-Reliant Systems (the 5 D framework) — https://arxiv.org/abs/2506.13023
• Hamel Husain, Your AI Product Needs Evals (the three evaluation levels) — https://hamelhusain.substack.com/p/evals
• Langfuse, Golden dataset evaluation: build and maintain LLM test sets — https://langfuse.com/resources/engineering/golden-dataset-evaluation
• Getmaxim, Building a "Golden Dataset" for AI Evaluation: A Step-by-Step Guide — https://www.getmaxim.ai/articles/building-a-golden-dataset-for-ai-evaluation-a-step-by-step-guide/
• Statsig, Golden datasets: creating evaluation standards — https://www.statsig.com/perspectives/golden-datasets-evaluation-standards
• The Evolution of LLM Adoption in Industry Data Curation Practices (silver, golden and super-golden datasets) — https://arxiv.org/abs/2412.16089