What an AI Evaluation Actually Measures

Every AI vendor deck contains a chart where their bar is taller than the other bars. The chart is usually accurate. It is also usually irrelevant to the question the buyer is actually asking, which is whether the thing will work on their own documents, their own tickets, their own images.

That gap is not marketing dishonesty. It is a measurement problem, and it has been documented carefully enough that you can check for it yourself.

A benchmark is an instrument, and instruments need validation

In the sciences that measure things you cannot see directly, a measurement instrument has to earn trust before its readings mean anything. The property being measured has to be defined. The test has to be shown to measure that property rather than something correlated with it. The result has to come with some estimate of how much of it is noise. Psychometrics calls this construct validity, and it is ordinary practice.

Language model benchmarks are instruments of exactly this kind. “Reasoning” and “helpfulness” are not directly observable, so a benchmark stands in for them. The question of whether the stand-in works is answerable, and in 2025 a team of 29 reviewers answered it across 445 benchmarks drawn from leading natural language processing and machine learning conferences, in work published in the NeurIPS 2025 Datasets and Benchmarks track.

The numbers are worth reading slowly:

• 16.0% conducted any statistical testing or uncertainty estimation. The other 84% report a number with no indication of how much it would move if you ran it again.
• 47.8% of the benchmarks that defined their target at all were measuring a contested phenomenon, meaning something the field does not agree on the definition of.
• 81.3% scored with exact string matching, and 40.7% used nothing else. A correct answer phrased differently counts as wrong.
• 42.6% reused items from existing benchmarks, which is how a flaw in one test propagates into the tests that were supposed to check it.

None of this means the benchmarks are worthless. It means a benchmark score is a weaker claim than it appears, and the strength of the claim is rarely stated alongside it.

The contamination problem makes the number softer still

A benchmark works only if the model has not already seen the answers. Modern training corpora are large enough, and scraped widely enough, that this is difficult to guarantee and easy to get wrong.

Researchers at Shanghai Jiao Tong University tested 31 models for evidence of benchmark leakage on GSM8K and MATH, two mathematical reasoning datasets, using perplexity and n-gram prediction accuracy to detect memorisation. About half showed signs of having trained on benchmark data. In the clearest case, Qwen-1.8B reproduced every five-word sequence in 223 examples from the GSM8K training set, 67 from the MATH training set, and 25 from the MATH test set, the split whose entire purpose is to be unseen.

A model that has memorised part of a test will score well on it and tell you nothing about performance on anything else. From the outside, that looks identical to a model that is simply good.

What this looked like in research

My doctoral work was on getting pretrained models to do visual tasks they were never trained for, without labelled examples: finding semantic correspondences using a diffusion model at NeurIPS 2023, then keypoints from the same class of model at the Computer Vision and Pattern Recognition conference in 2024. Both were measured against standard labelled benchmarks, because that is how the field compares methods.

Those numbers were real and the comparisons were fair. They also could not answer the question a company would ask, which is whether the method works on their images, at their resolution, with their lighting, under their definition of a correct answer. Benchmark performance and deployed performance are different measurements. Anyone who has moved a model from a paper to a product has watched the gap.

The eval you build yourself is the only one scoped to your decision

The fix starts with your own cases. Before committing to an AI system, assemble examples from your operation and label the correct outcome by hand. This gives the evaluation a task and a definition of success that belong to your business.

Sample from reality, not from the easy cases. Pull real leads, real tickets, real invoices, real photographs, including the malformed ones. A set drawn only from clean examples measures a system you do not have.

Choose the sample size for the decision. Start with cases your team can label carefully, then size the evaluation around the performance difference you need to detect and the risk of drawing the wrong conclusion. NIST’s guidance on testing proportions makes those choices explicit. A convenient number of examples is not, by itself, evidence that the test can answer your question.

Write down what correct means before you look at any output. Two people on the same team frequently disagree about whether a lead was qualified or a document was filed properly. Resolving that disagreement on paper is worth doing regardless of whether you buy anything, and doing it afterwards means the definition bends to fit the result.

Label the ambiguous cases as ambiguous. Forcing a binary answer onto a case your own team would escalate builds a test that punishes correct behaviour.

Hold some of it back. Keep a portion unseen by anyone building or tuning the system, including your vendor. That held-out portion is the only part whose score you can take at face value.

Review disagreements between your own labellers. Agreement measures how consistently people label the same cases; accuracy compares answers with a reference. Resolve disputed labels or mark cases ambiguous before using them to judge the system. To compare people with the model, evaluate both against the same reviewed answers using the same metric. The labellers’ agreement rate alone cannot establish human-level performance or prove that a vendor has seen the test answers.

What to ask a vendor

The useful questions are about method rather than score.

Ask what the reported number was measured on, and whether the system was tuned on that same data. Ask for performance on the cases it gets wrong, since error analysis was largely absent from the 445 benchmarks reviewed and is usually where the operational risk lives. Ask what the number would be on a set the vendor has never seen, and then supply one. Ask for a confidence interval, or for the result of running the evaluation more than once.

A vendor who can answer these is doing the work. A vendor who responds only with a leaderboard position has told you where they rank on someone else’s test.

How we apply this

Every system in our AI systems catalogue is built on a fixed sequence, and the second step is that a hand-labelled set of the client’s own real cases is agreed before anyone discusses hours saved. Nothing gets scaled on a benchmark number. Our AI Opportunity Audit is the planning engagement: a workflow inventory, ROI math, build-versus-buy calls, a data and access map, and a 90-day roadmap. The build scope should specify who labels the evaluation set and how it will be reviewed.

The evaluation is also what makes a written kill threshold meaningful. Agreeing in advance to stop at a defined level of performance is an empty promise without an agreed way to measure performance.

Sources

• Measuring what Matters: Construct Validity in Large Language Model Benchmarks, NeurIPS 2025 Datasets and Benchmarks track: arxiv.org/abs/2511.04703
• Benchmarking Benchmark Leakage in Large Language Models, Xu, Wang, Fan and Liu: arxiv.org/html/2404.18824v1