Evaluating LLMs

Capability, reliability, contamination, and the limits of benchmark scores

The problem

Part IV produced a family of systems: a base model trained on next tokens, checkpoints adapted with demonstrations and preferences, versions compressed for serving, and pipelines that construct context from an indexed corpus. Every stage presented itself as a set of choices. The remaining question is what any of these systems can be claimed to do. A benchmark score answers that question badly: the same model scores differently after a template change, a decoding change, an index update, a tool change, or a contaminated test item.

An evaluation produces evidence about a defined model or system under a defined input distribution, interface, scoring rule, and time. It does not produce a general scalar measure of intelligence, truthfulness, or fitness for every deployment. This chapter closes Part IV by treating evaluation as a design and measurement problem, connecting the earlier distinctions among token loss, instruction tuning, preference optimisation, inference, and retrieval. Part V then turns to reasoning models and the additional question of how to evaluate multi-step computation.

One frozen test makes the contract visible

The running test has labelled inputs, a fixed template, a checkpoint, a decoding policy, and an exact-match criterion. Changing any of these can change the reported score, which is the point: a score belongs to the whole arrangement, not to the model alone.

The first diagram lays that arrangement out as a contract. Follow the boxes left to right: the checkpoint and its interface, the test items and labels, the decoding and tool configuration, the scoring rule, and finally the reported metric with its uncertainty. Notice that the metric box comes last — everything upstream is part of what the number means.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle, FancyArrowPatch
plt.style.use("seaborn-v0_8-whitegrid")
blue,teal,orange,purple,grey="#24527a","#3a9188","#d97745","#7655a6","#5f6b76"
fig,ax=plt.subplots(figsize=(10,3.4)); ax.axis("off")
for x,label,c in [(.25,"checkpoint +\ninterface",blue),(2.25,"test items\n+ labels",teal),(4.25,"decoding +\ntools",orange),(6.25,"scoring\nrule",purple),(8.2,"metric +\nuncertainty",blue)]: ax.add_patch(Rectangle((x,1.3),1.4,.62,facecolor=c,edgecolor="white")); ax.text(x+.7,1.61,label,ha="center",va="center",color="white",fontsize=9)
for x in [1.68,3.68,5.68,7.68]: ax.add_patch(FancyArrowPatch((x,1.61),(x+.5,1.61),arrowstyle="-|>",color=grey,mutation_scale=11))
ax.text(5,.55,"A score is conditional on the complete evaluation contract.",ha="center",color=grey); ax.set_xlim(0,9.8); ax.set_ylim(.2,2.3); plt.show()

Figure 35.1: An LLM score is produced by a defined model and system interface, test set, decoding policy, and scoring procedure.

Read this way, a score is never a property of a model by itself. It is the output of a measurement procedure, and the rest of the chapter asks what each stage of that procedure must state for the number to mean something.

LLM evaluation. Measurement of a stated system interface on a defined task population and scoring rule. A benchmark score is conditional evidence, shaped by the model, prompt, decoding policy, data provenance, and development history.

Benchmark score and deployment claim. A benchmark score estimates performance on its items and interface. A deployment claim requires evidence about the intended users, inputs, consequence, and system configuration; it cannot be inferred from the benchmark alone.

Level 1 - Intuition

Scores answer bounded questions

Exact accuracy on multiple-choice questions measures success on those items under the supplied prompt and answer extraction rule. A factuality evaluation measures a different property. A safety red-team, long-context retrieval test, code execution task, and human preference study each answer other questions. A broad evaluation programme may combine them, but their scores should not be collapsed into one claim without explaining the aggregation and the uses it represents.

Benchmark choice defines which errors are visible. A test with short English questions can miss multilingual performance, tool failures, source use, refusal behaviour, latency, and long-document reliability. A test with a fixed answer key can be appropriate for a narrow capability while being unsuitable for an open-ended assistant response.

items=[True,True,False,True,False,True,True,False,True,True]
accuracy=np.mean(items)
print("accuracy:",accuracy,"n:",len(items))

accuracy: 0.7 n: 10

Reliability requires repeated and varied evidence

One successful answer does not estimate how often a system will succeed. Reliability concerns the distribution of outcomes across relevant inputs, including rare but consequential failures. For stochastic generation, the same input can yield different outputs. For tools and retrieval, external state can change. Evaluation should therefore record repetitions, seeds or sampling settings, versions, and the time at which an external collection was queried.

names=["overall","extraction","multilingual","long-context"]
scores=[.72,.86,.61,.49]
fig,ax=plt.subplots(figsize=(8,4)); ax.bar(names,scores,color=[grey,teal,orange,purple]); ax.set_ylim(0,1); ax.set_ylabel("score"); ax.set_title("Aggregate performance can hide task differences"); plt.tight_layout(); plt.show()

Figure 35.2: An aggregate score can conceal different performance across task or population slices, so evaluation should retain the component results.

The overall score of 0.72 averages three very different slices: extraction at 0.86, multilingual at 0.61, and long-context at 0.49. A deployment dominated by the weak slice inherits the weak slice’s performance, not the average.

Level 1 bounded what a score can claim. Level 2 makes the measurement mechanics explicit: units and denominators, sampling uncertainty, and the prompt and scorer choices that are part of the treatment.

Level 2 - Mechanics

The unit and denominator determine a metric

An accuracy denominator can be questions, examples, claims, generated tokens, tool calls, or complete episodes. These choices are not interchangeable. A system that answers one question with five claims may be correct on some claims and not others. A RAG system may retrieve a relevant document but fail the final answer. The report should state the unit, inclusion rule, missing-data rule, and whether an item can receive partial credit.

For binary accuracy over \(n\) independent labelled units, a simple standard-error approximation is \(\sqrt{\hat p(1-\hat p)/n}\). It is not sufficient for clustered, repeated, or adaptively sampled data, but it shows why a score from a small test set has substantial uncertainty.

p=.72; n=100
se=np.sqrt(p*(1-p)/n)
print("approximate 95% interval:",(round(p-1.96*se,3),round(p+1.96*se,3)))

approximate 95% interval: (np.float64(0.632), np.float64(0.808))

sizes=np.array([25,100,400]); estimates=np.repeat(.72,3); errors=1.96*np.sqrt(estimates*(1-estimates)/sizes)
fig,ax=plt.subplots(figsize=(7.5,4)); ax.errorbar(["25","100","400"],estimates,yerr=errors,fmt="o",color=teal,capsize=5); ax.set_ylim(.35,1); ax.set_ylabel("accuracy"); ax.set_xlabel("number of labelled items"); ax.set_title("Illustrative sampling uncertainty"); plt.tight_layout(); plt.show()

Figure 35.3: For a fixed observed accuracy, a larger evaluation set yields a narrower simple binomial uncertainty interval.

Prompt and scorer choices are part of the treatment

Few-shot examples, system messages, answer options, retrieval context, tool permissions, temperature, and output parsing can all change a score. The evaluation should freeze or systematically vary these choices rather than treating them as invisible plumbing. Automatic scorers should be validated on a labelled subset, particularly when judging open-ended answers, code, citations, or safety properties.

predictions=["A","B","C","A"]
labels=["A","B","B","A"]
exact=[p==y for p,y in zip(predictions,labels)]
print("exact-match score:",np.mean(exact),"errors:",[i for i,x in enumerate(exact) if not x])

exact-match score: 0.75 errors: [2]

The scorer is itself a component with error modes: this exact-match rule would reject a correct synonym, and a grader for open-ended answers needs its own validation.

Level 2 defined the metric, its uncertainty, and the treatment. Level 3 examines two boundaries that change what a score means: contamination of test material into training data, and the component structure of a complete system.

Level 3 - Mathematics & Code

Contamination changes the interpretation of benchmark performance

If test items, answers, or close variants entered pretraining, fine-tuning, preference data, or prompt examples, benchmark performance can reflect prior exposure as well as generalisation. Exact-string searches are useful but insufficient because paraphrases, translated items, templates, and solutions can differ visibly while preserving the test content. A contamination policy should state search methods, source coverage, match thresholds, exclusions, and what score is reported when exposure is suspected.

test_item="What is the eligibility rule?"
training_texts=["Eligibility requires residence in the district.","What is the eligibility rule?","Other text"]
matches=[text for text in training_texts if test_item.lower() in text.lower()]
print("exact matches:",matches)

exact matches: ['What is the eligibility rule?']

Contamination does not always make an evaluation useless, but it narrows the claim. A score may still measure retrieval from memorised material or performance under known exposure. It should not be presented as evidence of behaviour on unseen instances without a defensible boundary.

Evaluate complete systems at component boundaries

An LLM application can include a model, prompt template, decoder, retriever, index, tools, output validator, and user interface. A final failure may originate in any of these. The evaluation trace should retain input tokens, retrieved IDs, tool calls and outputs, generated IDs, validator decisions, and the final displayed answer. This supports component-level measurements and repair.

trace={"retrieval":"relevant chunk missed","tool":"not called","generation":"unsupported answer","validator":"passed"}
print(trace)

{'retrieval': 'relevant chunk missed', 'tool': 'not called', 'generation': 'unsupported answer', 'validator': 'passed'}

fig,ax=plt.subplots(figsize=(10,3.3)); ax.axis("off")
for i,(label,c) in enumerate([("input",blue),("retrieve / tools",teal),("context",orange),("generate",purple),("validate",teal),("display",blue)]):
    x=.15+i*1.62; ax.add_patch(Rectangle((x,1.35),1.2,.55,facecolor=c,edgecolor="white")); ax.text(x+.6,1.625,label,ha="center",va="center",color="white",fontsize=8)
ax.text(4.9,.55,"Record stage outputs to locate failures rather than treating every error as a model error.",ha="center",color=grey); ax.set_xlim(0,10); ax.set_ylim(.2,2.3); plt.show()

Figure 35.4: A complete-system evaluation traces failures across input construction, retrieval or tools, generation, validation, and final output.

A trace turns “the answer was wrong” into a located failure: in the running example, retrieval missed the relevant chunk and the validator passed an unsupported answer — neither is a generation error.

Level 3 located failure inside the pipeline. Level 4 turns to evaluation design: building tests around decisions, reporting disaggregated evidence, and recognising that benchmark use itself changes what scores mean.

Level 4 - Research & Systems

Build an evaluation around a decision

The intended decision should determine the test: whether to deploy a model version, select an adaptation, allow a tool action, use a retrieval corpus, or claim performance for a population. Define the target inputs, harm or cost of errors, baseline, required confidence, and stopping rule before inspecting the result. This prevents a convenient benchmark from becoming a proxy for a different deployment decision.

evaluation_plan={"decision":"deploy retrieval update","population":"current policy queries","baseline":"index-v11","primary":"supported-answer rate","guardrail":"no access-control failures"}
print(evaluation_plan)

{'decision': 'deploy retrieval update', 'population': 'current policy queries', 'baseline': 'index-v11', 'primary': 'supported-answer rate', 'guardrail': 'no access-control failures'}

Report disaggregated evidence and limits

An audit report should state model, tokenizer, prompt, decoding, tools and index versions; test construction and dates; item counts and strata; scoring and adjudication; uncertainty; contamination checks; excluded cases; failures; cost and latency; and known limits. Where human judgements are used, report judge instructions, information access, agreement, and aggregation. Where benchmarks are public, report whether they may have influenced development.

fig,ax=plt.subplots(figsize=(9,3.5)); ax.axis("off")
fields=["claim","population","interface","metric","uncertainty","contamination","failures"]
for i,field in enumerate(fields):
    x=.25+(i%4)*2.15; y=2.45-(i//4)*.85; ax.add_patch(Rectangle((x,y),1.8,.48,facecolor=[blue,teal,orange,purple][i%4],edgecolor="white")); ax.text(x+.9,y+.24,field,ha="center",va="center",color="white")
ax.text(4.5,.45,"The report should make the scope of every performance claim inspectable.",ha="center",color=grey); ax.set_xlim(0,9); ax.set_ylim(.15,3.2); plt.show()

Figure 35.5: A credible LLM evaluation report links each claim to its population, interface, metric, uncertainty, contamination status, and known limits.

Reliability includes calibration and failure concentration

An evaluation can ask whether a model’s expressed confidence or probability is related to correctness. Calibration is not solved by a single overall accuracy: a system may be reliable on routine examples and confidently wrong on one task slice. When a system does not expose calibrated probabilities, a proxy such as self-reported confidence, verifier score, or retrieval support should be treated as a separate predictor and evaluated against outcomes.

Failure concentration also matters. Ten errors distributed across low-stakes formatting tasks have different implications from ten errors concentrated in access-control, medical, or financial cases. An evaluation report should preserve error categories and severity rules, rather than allowing a broad average to conceal the decision consequences of a rare failure.

outcomes=[{"slice":"routine","correct":1,"confidence":.8},{"slice":"high-stakes","correct":0,"confidence":.95},{"slice":"routine","correct":1,"confidence":.7}]
for row in outcomes: print(row)

{'slice': 'routine', 'correct': 1, 'confidence': 0.8}
{'slice': 'high-stakes', 'correct': 0, 'confidence': 0.95}
{'slice': 'routine', 'correct': 1, 'confidence': 0.7}

Benchmark development is itself an intervention

Once a benchmark is public and used repeatedly for model selection, prompt design, data filtering, or reward design, its score can become a target. This can improve performance on the benchmark while narrowing the evidence it provides about unseen work. Held-out sets, refreshed item pools, task variants, and preregistered evaluation protocols can reduce this risk, but none removes the need to describe the development history.

development={"public_benchmark_used_for_tuning":True,"hidden_holdout":True,"prompt_fixed_before_holdout":True}
print(development)

{'public_benchmark_used_for_tuning': True, 'hidden_holdout': True, 'prompt_fixed_before_holdout': True}

Common misconceptions

• A high benchmark score establishes general intelligence. It measures performance under a specific evaluation contract.
• A single accuracy is enough for deployment. Slice performance, uncertainty, reliability, cost, and failure consequences may differ.
• Public benchmarks are necessarily uncontaminated. Development and training exposure require explicit assessment.
• A final wrong answer is always a model failure. Retrieval, tools, prompts, validators, and interfaces can fail first.
• Automatic judging removes evaluation subjectivity. It introduces a scorer model and its own validation requirements.
• More benchmarks automatically improve evidence. They can duplicate a narrow distribution or encourage benchmark-specific optimisation.

Exercises

1. Define the unit and denominator for a claim-level factuality metric.
2. Calculate accuracy and a simple interval for a supplied outcome vector.
3. Explain why clustered examples violate a simple independent-item interval.
4. List the interface fields to freeze for a reproducible benchmark run.
5. Design a slice analysis for multilingual assistance.
6. Identify an exact contamination check and one limitation of it.
7. Construct a trace for a RAG answer that retrieved the wrong chunk.
8. Compare a capability benchmark with a reliability test.
9. Specify an adjudication process for open-ended human scoring.
10. State why aggregate scores can conceal deployment risk.
11. Propose a guardrail metric for a tool-using system.
12. Design a prompt-sensitivity experiment.
13. List report fields needed for a public benchmark claim.
14. Explain why a validator can create false assurance.
15. Define a decision that an evaluation should support.
16. Identify the first chapter of Part V and its new evaluation problem.

Summary

LLM evaluation measures defined systems on defined tasks under defined interfaces. Capability scores, reliability studies, safety tests, retrieval evaluation, and service benchmarks provide different evidence. Strong reports retain the population, denominator, uncertainty, contamination policy, component trace, and known limits rather than reducing the system to one aggregate number.

Looking ahead

Chapter 35 begins Part V by distinguishing base models from reasoning-tuned models and examining what changes when systems are trained or prompted to use more computation on a problem.

Further reading

Holistic evaluation categories and scenarios are developed by Liang et al. (2022). The MMLU benchmark illustrates multi-task knowledge evaluation and its scope limits (Hendrycks et al. 2021).

Hendrycks, Dan, Collin Burns, Steven Basart, et al. 2021. “Measuring Massive Multitask Language Understanding.” International Conference on Learning Representations. https://arxiv.org/abs/2009.03300.

Liang, Percy, Rishi Bommasani, Tony Lee, et al. 2022. “Holistic Evaluation of Language Models.” arXiv Preprint arXiv:2211.09110. https://arxiv.org/abs/2211.09110.