Canonical: https://www.strataigize.com/insights/what-an-ai-evaluation-actually-measures/
Description: Benchmark scores are measurement instruments, and most have never been validated as instruments. What to check before you trust one, and what to build instead.
Published: 2026-09-13T00:00:00.000Z
Modified: 2026-09-13T00:00:00.000Z

[← Blog](https://www.strataigize.com/insights/)

# What an AI Evaluation Actually Measures

By [**Eric Hedlin**](https://www.strataigize.com/about/team/eric/), Chief AI Scientist · Published September 13, 2026 · 6 min read

A benchmark score tells you how a model performed on one fixed set of questions, scored one particular way. It does not tell you whether the model will work on your data. A 2025 review of 445 language model benchmarks found that only 16% ran any statistical test on their results, and separate work shows roughly half of tested models had already seen benchmark content during training. Before buying an AI system, build a small labelled set from your own real cases and measure against that.

In this article

1.  [A benchmark is an instrument, and instruments need validation](https://www.strataigize.com/insights/what-an-ai-evaluation-actually-measures/#a-benchmark-is-an-instrument-and-instruments-need-validation)
2.  [The contamination problem makes the number softer still](https://www.strataigize.com/insights/what-an-ai-evaluation-actually-measures/#the-contamination-problem-makes-the-number-softer-still)
3.  [What this looked like in research](https://www.strataigize.com/insights/what-an-ai-evaluation-actually-measures/#what-this-looked-like-in-research)
4.  [The eval you build yourself is the only one scoped to your decision](https://www.strataigize.com/insights/what-an-ai-evaluation-actually-measures/#the-eval-you-build-yourself-is-the-only-one-scoped-to-your-decision)
5.  [What to ask a vendor](https://www.strataigize.com/insights/what-an-ai-evaluation-actually-measures/#what-to-ask-a-vendor)
6.  [How we apply this](https://www.strataigize.com/insights/what-an-ai-evaluation-actually-measures/#how-we-apply-this)
7.  [Sources](https://www.strataigize.com/insights/what-an-ai-evaluation-actually-measures/#sources)

[Add us as a preferred source on Google](https://www.google.com/preferences/source?q=strataigize.com)

A free Google setting. It puts our work higher in your own results, changes nothing for anyone else, and you can undo it any time.

Every AI vendor deck contains a chart where their bar is taller than the other bars. The chart is usually accurate. It is also usually irrelevant to the question the buyer is actually asking, which is whether the thing will work on their own documents, their own tickets, their own images.

That gap is not marketing dishonesty. It is a measurement problem, and it has been documented carefully enough that you can check for it yourself.

## A benchmark is an instrument, and instruments need validation

In the sciences that measure things you cannot see directly, a measurement instrument has to earn trust before its readings mean anything. The property being measured has to be defined. The test has to be shown to measure that property rather than something correlated with it. The result has to come with some estimate of how much of it is noise. Psychometrics calls this construct validity, and it is ordinary practice.

Language model benchmarks are instruments of exactly this kind. “Reasoning” and “helpfulness” are not directly observable, so a benchmark stands in for them. The question of whether the stand-in works is answerable, and in 2025 a team of 29 reviewers answered it across [445 benchmarks drawn from leading natural language processing and machine learning conferences](https://arxiv.org/abs/2511.04703), in work published in the NeurIPS 2025 Datasets and Benchmarks track.

The numbers are worth reading slowly:

-   **16.0%** conducted any statistical testing or uncertainty estimation. The other 84% report a number with no indication of how much it would move if you ran it again.
-   **47.8%** of the benchmarks that defined their target at all were measuring a contested phenomenon, meaning something the field does not agree on the definition of.
-   **81.3%** scored with exact string matching, and 40.7% used nothing else. A correct answer phrased differently counts as wrong.
-   **42.6%** reused items from existing benchmarks, which is how a flaw in one test propagates into the tests that were supposed to check it.

None of this means the benchmarks are worthless. It means a benchmark score is a weaker claim than it appears, and the strength of the claim is rarely stated alongside it.

## The contamination problem makes the number softer still

A benchmark works only if the model has not already seen the answers. Modern training corpora are large enough, and scraped widely enough, that this is difficult to guarantee and easy to get wrong.

Researchers at Shanghai Jiao Tong University [tested 31 models for evidence of benchmark leakage](https://arxiv.org/html/2404.18824v1) on GSM8K and MATH, two mathematical reasoning datasets, using perplexity and n-gram prediction accuracy to detect memorisation. About half showed signs of having trained on benchmark data. In the clearest case, Qwen-1.8B reproduced every five-word sequence in 223 examples from the GSM8K training set, 67 from the MATH training set, and 25 from the MATH **test** set, the split whose entire purpose is to be unseen.

A model that has memorised part of a test will score well on it and tell you nothing about performance on anything else. From the outside, that looks identical to a model that is simply good.

## What this looked like in research

My doctoral work was on getting pretrained models to do visual tasks they were never trained for, without labelled examples: finding [semantic correspondences using a diffusion model](https://proceedings.neurips.cc/paper_files/paper/2023/hash/1a074a28c3a6f2056562d00649ae6416-Abstract-Conference.html) at NeurIPS 2023, then [keypoints from the same class of model](https://openaccess.thecvf.com/content/CVPR2024/html/Hedlin_Unsupervised_Keypoints_from_Pretrained_Diffusion_Models_CVPR_2024_paper.html) at the Computer Vision and Pattern Recognition conference in 2024. Both were measured against standard labelled benchmarks, because that is how the field compares methods.

Those numbers were real and the comparisons were fair. They also could not answer the question a company would ask, which is whether the method works on their images, at their resolution, with their lighting, under their definition of a correct answer. Benchmark performance and deployed performance are different measurements. Anyone who has moved a model from a paper to a product has watched the gap.

## The eval you build yourself is the only one scoped to your decision

The fix is unglamorous and it works. Before committing to an AI system, assemble a set of real cases from your own operation and label the correct outcome by hand. It is the most useful half-day anyone spends on an AI project.

**Sample from reality, not from the easy cases.** Pull real leads, real tickets, real invoices, real photographs, including the malformed ones. A set drawn only from clean examples measures a system you do not have.

**Aim for 100 to 300 labelled items to start.** Enough to separate a system that is right most of the time from one that is right slightly more often than chance, and small enough that a person can actually do it in an afternoon.

**Write down what correct means before you look at any output.** Two people on the same team frequently disagree about whether a lead was qualified or a document was filed properly. Resolving that disagreement on paper is worth doing regardless of whether you buy anything, and doing it afterwards means the definition bends to fit the result.

**Label the ambiguous cases as ambiguous.** Forcing a binary answer onto a case your own team would escalate builds a test that punishes correct behaviour.

**Hold some of it back.** Keep a portion unseen by anyone building or tuning the system, including your vendor. That held-out portion is the only part whose score you can take at face value.

**Record the disagreement rate between your own labellers.** If two of your people agree 85% of the time, a system scoring 85% is performing at human level on your task, and a vendor promising 99% is either measuring something else or has seen your answers.

## What to ask a vendor

The useful questions are about method rather than score.

Ask what the reported number was measured on, and whether the system was tuned on that same data. Ask for performance on the cases it gets wrong, since error analysis was largely absent from the 445 benchmarks reviewed and is usually where the operational risk lives. Ask what the number would be on a set the vendor has never seen, and then supply one. Ask for a confidence interval, or for the result of running the evaluation more than once.

A vendor who can answer these is doing the work. A vendor who responds only with a leaderboard position has told you where they rank on someone else’s test.

## How we apply this

Every system in our [AI systems catalogue](https://www.strataigize.com/services/ai-systems/) is built on a fixed sequence, and the second step is that a hand-labelled set of the client’s own real cases is agreed before anyone discusses hours saved. Nothing gets scaled on a benchmark number. The [AI Opportunity Audit](https://www.strataigize.com/offers/ai-opportunity-audit/) that opens most engagements produces that labelled set as a deliverable, and it is yours whether or not we build anything afterwards.

The evaluation is also what makes a written kill threshold meaningful. Agreeing in advance to stop at a defined level of performance is an empty promise without an agreed way to measure performance.

## Sources

-   Measuring what Matters: Construct Validity in Large Language Model Benchmarks, NeurIPS 2025 Datasets and Benchmarks track: [arxiv.org/abs/2511.04703](https://arxiv.org/abs/2511.04703)
-   Benchmarking Benchmark Leakage in Large Language Models, Xu, Wang, Fan and Liu: [arxiv.org/html/2404.18824v1](https://arxiv.org/html/2404.18824v1)

Author

**Eric Hedlin**, Chief AI Scientist at Strataigize. PhD in computer science (UBC). Signs off AI systems so they hold in production, with a baseline.

Next step

Six questions, one bottleneck. The Growth Diagnostic benchmarks every stage of your funnel and hands you the fix that pays first.

[Run the diagnostic →](https://www.strataigize.com/tools/growth-diagnostic/)

Talk to us

### Talk to the team that would run it

Tell us where to look and we reply within 24 hours with where we would start. No pitch until you see the value.

[Add us as a preferred source on Google](https://www.google.com/preferences/source?q=strataigize.com)

A free Google setting. It puts our work higher in your own results, changes nothing for anyone else, and you can undo it any time.

## Related reading

[All AI & Automation articles →](https://www.strataigize.com/insights/topics/ai-automation/)

[How to Build an AI Lead-Gen System That Actually WorksMost 'AI lead gen' is spam. A system that works pairs real buying signals with hyper-personalized, human-reviewed outreach.](https://www.strataigize.com/insights/build-an-ai-lead-gen-system/)[Should your business be agent-callable, or is it a risk?A customer's AI assistant can change a record if you let it start that job. Write the no-list this week, before anyone connects anything.](https://www.strataigize.com/insights/should-your-business-be-agent-callable/)[When a Smaller AI Model Will DoMost agent work is one narrow task repeated. On that shape of work a small fine-tuned model often beats a frontier model, at a fraction of the cost.](https://www.strataigize.com/insights/when-a-smaller-ai-model-will-do/)

## Want AI doing this for your growth?

We build AI-driven acquisition, content, and automation systems for operators across North America. See your levers in 30 minutes.

[Book a growth audit →](https://www.strataigize.com/audit/) [Rated **5.0** on Clutch](https://clutch.co/profile/strataigize-marketing)

[Explore AI automation services →](https://www.strataigize.com/services/ai-automation-services/)