What an AI Evaluation Actually Measures
A benchmark score measures performance on a particular test, under its scoring rules. It does not establish that a model will work on your documents or make the decisions your business needs. Before buying, define a correct outcome on your own cases, keep an unseen test set, and examine the mistakes as well as the
A vendor’s tallest bar can be accurate and still leave your buying question unanswered. A model that wins on mathematical questions may struggle with the abbreviations in your support tickets. A document extractor may perform well on clean invoices and fail on the scans your team
That gap is not marketing dishonesty. It is a measurement problem, and it has been documented carefully enough that you can check for
A benchmark is an instrument, and instruments need validation
In the sciences that measure things you cannot see directly, a measurement instrument has to earn trust. The property being measured has to be defined, and the test has to measure that property rather than something merely correlated with it. Psychometrics calls that relationship construct validity. Uncertainty estimates answer a separate question: how precise is
Language model benchmarks are instruments of this kind. “Reasoning” and “helpfulness” are not directly observable, so a benchmark stands in for them. In 2025, a team of 29 reviewers examined the strength of that relationship across 445 benchmarks drawn from leading natural language processing and machine learning conferences, in work published in the NeurIPS 2025 Datasets and
The numbers are worth
- 16.0% used statistical tests or uncertainty estimates to compare results. A reported difference without those checks may be less meaningful than it looks.
- 47.8% of the benchmarks that defined their target at all were measuring a contested phenomenon, meaning something the field does not agree on the definition of.
- 81.3% used exact matching for at least part of the scoring, and 40.7% used it exclusively. That can suit a precise label or number; for an open response, it can reject a valid answer phrased differently.
- 42.6% reused items from existing benchmarks. Reuse can carry forward a test’s limitations as well as its useful comparisons.
The benchmarks remain useful. Still, a benchmark score is a weaker claim than it appears, and the strength of the claim is rarely stated
The contamination problem makes the number softer still
A test of performance on unseen examples depends on those examples remaining unseen. With large, partly undisclosed training corpora, that is difficult
Researchers at Shanghai Jiao Tong University tested 31 models for evidence of benchmark leakage on GSM8K and MATH, two mathematical reasoning datasets. About half showed signs consistent with exposure to benchmark data. Their detection procedure sampled five starting positions per example and tested prediction of the following five-token spans. Qwen-1.8B predicted all sampled spans correctly in 223 GSM8K training examples, 67 MATH training examples and 25 MATH
These results indicate possible exposure while leaving the models’ full training history unresolved. Memorisation can inflate a score, making it harder to separate familiarity with the test from the ability to solve a new case. It does not establish that the model has no useful ability outside
What this looked like in research
My doctoral work was on getting pretrained models to do visual tasks they were never trained for, without labelled examples: finding semantic correspondences using a diffusion model at NeurIPS 2023, then keypoints from the same class of model at the Computer Vision and Pattern Recognition conference in 2024. Both were measured against standard labelled benchmarks, because that is how the field
Those numbers were real and the comparisons were fair. They also could not answer the question a company would ask, which is whether the method works on their images, at their resolution, with their lighting, under their definition of a correct answer. Benchmark performance and deployed performance are different measurements. Anyone who has moved a model from a paper to a product has watched
Build an evaluation around your decision
The fix starts with your own cases. Before committing to an AI system, assemble examples from your operation and label the correct outcome by hand. This gives the evaluation a task and a definition of success that belong to
Include the difficult cases your team actually receives. Pull real leads, real tickets, real invoices, real photographs, including the malformed ones. A set drawn only from clean examples measures a system you do
Choose the sample size for the decision. Start with cases your team can label carefully, then size the evaluation around the performance difference you need to detect and the risk of drawing the wrong conclusion. NIST’s guidance on testing proportions makes those choices explicit. A convenient number of examples is not, by itself, evidence that the test can answer
Write down what correct means before you look at any output. Two people on the same team frequently disagree about whether a lead was qualified or a document was filed properly. Resolving that disagreement on paper is worth doing regardless of whether you buy anything, and doing it afterwards means the definition bends to fit
Label the ambiguous cases as ambiguous. Forcing a binary answer onto a case your own team would escalate builds a test that punishes
Hold some of it back. Keep a portion unseen by anyone building or tuning the system, including your vendor. It gives you a cleaner estimate of performance on new cases. Its usefulness still depends on representative sampling, reliable labels and enough examples to support
Review disagreements between your own labellers. Agreement measures how consistently people label the same cases; accuracy compares answers with a reference. Resolve disputed labels or mark cases ambiguous before using them to judge the system. To compare people with the model, evaluate both against the same reviewed answers using the same metric. The labellers’ agreement rate alone cannot establish human-level performance or prove that a vendor has seen the
One case, with a usable definition of correct
The following hypothetical billing ticket shows how to evaluate an assistant that routes requests and
| Part of the evaluation | What the reviewer records |
|---|---|
| Customer message | “My bank shows two charges for one order. One is pending and one is completed.” |
| Expected route | Billing review. The message alone does not establish two settled charges. |
| Acceptable draft | Acknowledge the concern, distinguish the pending entry from the completed charge, and say billing will check the payment record. |
| Failure | Promise a refund or state that the customer was charged twice without checking the record. |
One labeller might initially mark “refund needed” while another marks “payment review”. Resolve that disagreement against the payment policy before scoring the model. If the available record cannot settle it, the correct outcome
A reply can sound excellent and still fail this case. Across the evaluation set, record unsupported refund promises separately from wrong routing: those errors have different consequences. Agree which errors prevent automatic use and which permit a draft that a person reviews. Then judge the candidate on the unseen cases against those same rules. The average score should not hide the mistake that makes
What to ask a vendor
The useful questions are about method rather
Ask what the reported number was measured on, and whether the system was tuned on that same data. Ask to see the failures, grouped by their consequences for your workflow. Supply cases the vendor has never seen. Ask for uncertainty estimates, and repeat runs where the system’s answers vary. Repeating the same unrepresentative test does not fix
A vendor who can answer these is doing the work. A vendor who responds only with a leaderboard position has told you where they rank on someone
How we apply this
The workflows in our AI systems catalogue need different tests: correctly extracting a document and correctly qualifying a lead are different outcomes. For either, the build scope should specify the real cases used for acceptance, who reviews the expected answers and what happens when the
Our AI Opportunity Audit is the planning engagement: a workflow inventory, ROI math, build-versus-buy calls, a data and access map, and a 90-day roadmap. That plan should make the evaluation work explicit before an estimate of hours saved becomes the reason
The evaluation is also what makes a written kill threshold meaningful. Agreeing in advance to stop at a defined level of performance is an empty promise without an agreed way to
Sources
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks, NeurIPS 2025 Datasets and Benchmarks track: arxiv.org/abs/2511.04703
- Benchmarking Benchmark Leakage in Large Language Models, Xu, Wang, Fan and Liu: arxiv.org/html/2404.18824v1