When the Exam Is Wrong: What Do AI Scores Measure?

A rope-tension question in PHYBench produced a model answer that looked different from the reference. Ordinary denominator rationalization turns them into the same expression:

T = P / (3 * sqrt(6))
  = P * sqrt(6) / (3 * 6)
  = sqrt(6) * P / 18

Yet the Expression Edit Distance (EED) evaluator assigned zero credit and a failed binary judgment. This is not a counterexample I invented. Figure 3 of a physics benchmark audit submitted on September 11, 2026 documents it. The model supplied an equivalent answer; the measurement system failed to recognize it.

Researchers at Yale and other institutions conducted the study, How Good Are Frontier Models at Physics?, across six benchmarks: UGPhysics, PHYBench, PRISM-Physics, HLE-Physics, CMT-Benchmark, and CritPt. Its scope is text-only, closed-ended physics problems with verifiable final answers. Some scores rose sharply after repair: GPT-5.6-Sol's mean@4 increased from 61.00% to 87.24% on CMT and from 47.28% to 78.66% on HLE-Physics.

Sharing just those numbers invites a simple story: AI already knows physics; the exam was holding it back. I want to separate what was originally misgraded from what changed in the questions and what changed in the comparison. Those distinctions determine how much a high score can establish.

The denominator behind 95.2% is rejected answers

Experts classified cases as model errors, grader errors, or benchmark errors. In the first category, the problem and reference are sound and the model is wrong. In the second, a correct response is rejected. The third covers incorrect references, missing conditions, and ambiguous questions. When benchmark materials are defective, the paper gives that classification precedence rather than grading the response as though the task were valid.

Across four pooled audits, 250 rejected cases sent for review ended with 143 benchmark errors, 95 grader errors, and 12 model errors after adjudication. Thus 95.2% is the share of the first two categories within those rejections. It is neither the model's physics accuracy nor a claim that 95.2% of all benchmark questions are defective. CMT and CritPt used different audit coverage and are excluded from this pooled count.

That is not a disposable statistical footnote. An audit investigating failed judgments is naturally positioned to find false rejections. It shows why these failure records cannot be attributed directly to the model. It does not randomly inspect the entire question pool to estimate the complete measurement system's error rate.

Benchmark errors are not always careless arithmetic, either. In Figure 4, a CritPt question about the Kitaev honeycomb model asks for ground-state energy without explicitly specifying a Hamiltonian using Pauli matrices or spin operators. For spin 1/2, using S = σ/2 changes the two-body product by a factor of four. The repaired question supplies the Hamiltonian and convention. An individual researcher's customary convention cannot substitute for a condition absent from the question.

To me, this is more concerning than an unintelligent grader. If a problem permits multiple interpretations, a stronger LLM judge may still be unable to recover a unique answer. Establishing whether the question can be judged and determining whether the response is correct are separate jobs.

From 61 to 87 involves more than re-grading

CMT originally contained 50 questions. Experts found defects in 30, repaired 29, and excluded one, leaving 49. For HLE-Physics, 202 text-only questions remained after multimodal exclusions; removing 86 benchmark-error questions left 116. The figures below come from Table 1 and refer to GPT-5.6-Sol High's mean@4 under the corresponding tool conditions.

One detail deserves an explicit correction. CMT's native evaluator was not public, so this study used the same HLE-adapted evaluator before and after repair. Section 3.2.2 attributes the changes to benchmark materials. A new score does not automatically mean a new grading program, and the whole CMT increase cannot be counted as recovered credit for previously misjudged answers.

Keeping the original problem and response fixed while changing the grade tests grading error. Correcting a reference changes the judgment basis. Adding conditions and generating a new response changes the task. Excluding questions changes the denominator and difficulty composition. Each operation can be justified, but they are not the same measurement.

This is particularly important for HLE. Removing invalid questions is necessary, but the audit concentrates on rejected model responses, linking exclusion to prior failure records. The retained set need not preserve the original difficulty distribution. The point is not to keep defective questions. It is to report what was removed, why, and which task set the new score describes.

The revised result can be more meaningful without sharing the old score's scale. I would want separate experiments for re-scoring fixed outputs and re-evaluating repaired tasks, rather than one new percentage replacing the old record. Nor can interacting contributions be added arbitrarily.

One success in four is not one-shot success

On repaired CMT, GPT-5.6-Sol High achieves 87.24% mean@4 and 97.96% pass@4. The former averages correctness over four attempts per question. The latter asks only whether at least one of those four attempts succeeds.

The numbers are compatible, but pass@4 is closer to coverage assessed with hindsight. If one of four answers is correct and three are wrong, a user still has to identify the trustworthy one. The metric supplies no automatic selector. It cannot be translated into "ask once and get nearly 98% accuracy."

Sample size also limits interpretation. Each question in a 49-question pass@4 result represents about 2.04 percentage points. A ranking separated by one or two questions calls first for item-level differences and uncertainty, not a declaration of a large capability gap. Four responses to the same question cannot simply be counted as four independent questions to enlarge the statistical sample.

I therefore distinguish repeatability, validity for the target capability, and comparability across versions. Consistently rejecting an equivalent expression is a repeatable mistake, not valid measurement. Even excellent questions cannot guarantee comparable historical scores if tool access, reasoning budgets, or evaluator versions keep changing.

Bowman and Dahl's 2021 paper on benchmarking already discussed limitations in dataset design, annotation reliability, and scale. As models solve more valid questions, a few measurement defects can dominate the remaining failures. Near-saturation can mean the question set no longer distinguishes frontier models, not that those models have no weaknesses.

Experts disagree, and errors can favor the other side

Bringing in experts does not create an error-free judge. In the physics study, 196 cases had two reviewers: 140 initial agreements and 56 disagreements, later resolved by a third review. Another 54 cases received only one review. The 71.43% figure is initial agreement among double-reviewed cases, not final accuracy. Nor does it mean the other 28.57% of conclusions are all wrong.

Drawing a boundary between missing conditions and disciplinary convention requires judgment. Is an unstated assumption an unambiguous field default or an omission that changes the answer? Reviewers also saw model outputs and AI-generated preliminary analysis, which could influence interpretation. That is a methodological risk to control, not an accusation of deliberate favoritism.

I would try to assess question validity before showing the model response and preserve reasons for disagreements. Accepted responses also need sampling. Checking only failures more readily uncovers false negatives, where correct answers are rejected. False positives, where wrong answers are accepted, also distort scores, in the opposite direction.

In its August 2026 review of original HLE, Epoch AI randomly sampled six questions from each of eight categories, for 48 total. It confirmed substantial defects in 22: twelve were not clearly answerable as written; ten others could reject correct answers, five of which could also accept wrong ones. This is an equal-allocation stratified review, not a precise defect rate for all HLE, and not a finding about HLE-Rolling or HLE-Verified.

These observations support repairing evaluations, not assuming repairs must only increase scores. Widening acceptance for equivalent answers also requires checking whether false acceptance increases. Audit both directions to find out whether a fix exchanges one error for another.

The tests required a function name the task did not

Coding evaluation can suffer the same mismatch. OpenAI's SWE-bench Verified audit selected 138 tasks that o3 did not consistently solve across multiple runs. It found material test or description issues in 59.4%. This was a difficult subset of the 500-task benchmark, not a random sample of the whole set.

One Pylint task is especially concrete: tests directly imported a new function, get_annotation, although the problem statement did not require that name. A patch meeting the requirement through another internal implementation could still be rejected during test collection because the import failed. The tests had turned a detail of the reference patch into an unstated requirement.

Physics problems should require a correct result, not a particular denominator form. Software requirements specify behavior and should not silently require an internal helper with one name. Of course, testing a named interface is justified when it is part of a public contract. The question is whether the constraint appears in the task, not whether implementation details must never be tested.

There is an opposite risk as well. Public issues, patches, and discussions can enter training data. OpenAI reported that tested frontier models could reproduce human reference patches or problem-statement details on some tasks. Those observations support contamination concerns but do not quantify how much of each score comes from memory.

Correct grading establishes that a response satisfies the task's rules. It does not also establish that the model never saw the question. A long-public benchmark used for training feedback and model selection can increasingly measure adaptation to that test. Repairing false negatives and investigating contamination are separate responsibilities.

New questions make old scores harder to compare

LiveBench continually introduces new material and emphasizes objectively verifiable answers. LiveCodeBench collects contest problems over time and organizes evaluation by date ranges. These approaches reduce risks from prolonged exposure, but "new" does not automatically mean "equally difficult."

If model A scores 80 on one set this year and model B scores 85 on another next year, the five-point increase alone does not prove B is stronger. Subfield coverage, difficulty, and settings may have changed. Comparing new and old models should at least involve rerunning them on one frozen release under the same budget and tool conditions, rather than stitching annual leaderboard figures into a capability curve.

I would retain two sets for different purposes. Stable, validated questions help track regressions and configuration differences. New questions with controlled exposure help evaluate performance on new tasks. Saturated old questions can still detect degradation; they just become poor instruments for finely ranking the strongest models.

Cross-version calibration requires explaining methods and assumptions too. Common items that are memorized widely or solved nearly universally make weak anchors. Adding a few anchor questions does not create comparability by itself. Contamination, tool configuration, and task coverage still need separate checks.

Maintain the evaluator as software

In engineering terms, I want a benchmark to preserve the process producing a score, not just a dataset and leaderboard. Questions, references, graders, prompts, model versions, reasoning budgets, and tool permissions must be traceable. Even adding one condition should record whether the valid-answer set changed, rather than quietly overwriting the task.

Response generation and scoring should be separated. Preserve raw outputs and run settings so evaluator repairs can re-score the same responses. A changed question starts another experiment. That separation makes it possible to distinguish model progress from measurement changes and explain a ranking reversal.

Graders need their own regression cases. For formulas, include positive examples of algebraic equivalence and negative examples that look similar but differ, have wrong units, or omit conditions. Layer symbolic computation, numerical checks, and expert review. Do not just relax rules around previously observed false rejections.

Numerical substitution is easy to overuse. Equality at a few points does not establish an identity. Domains, singularities, and square-root branches can change equivalence. For example, sqrt(x*x) = x requires additional conditions; for general real numbers the result is abs(x). Symbolic simplification also needs assumptions. A plausible-looking simplification cannot substitute for physical constraints.

A report can retain parallel results: original tasks and grader, corrected scores for fixed outputs, and new evaluations of repaired tasks. Each should identify question counts, model and configuration, audit coverage, and uncertainty organized by question. It is less concise than one overall accuracy, but readers can identify which experiment they are comparing.

Evaluate problem solving and research separately

This audit provides evidence worth taking seriously: old procedures understated some frontier models on text-only physics problems with clear conditions and verifiable final answers. Those repaired results should not be dismissed to preserve an intuition that "AI does not understand physics."

But a problem set normally supplies the subject, conditions, and quantity to find. Open research also requires deciding what is worth asking, choosing representations, distinguishing explanations, and revising assumptions in response to evidence. Reading closed-ended scores as independent research ability skips that difference.

In his subsequent Is Physics Dead?, coauthor John Sous describes preliminary Agent attempts on open theoretical-physics problems. At the time, the system had not fully resolved any autonomously. He specifies that these results are not in the paper. They are not a rigorous upper-bound test and do not prove AI cannot do research. They more appropriately remind us to define and evaluate open research capabilities separately.

Harder tasks are not automatically more valid. An open task scored only by another model's vague claim of "insight" can also lose its basis for review. Derivations, validation steps, testable predictions, and human-review reasons need to be preserved to discuss research progress rather than merely change the scoring vocabulary.

After reading this paper, I would treat neither the old scores as final capability judgments nor the new ones as evidence that physics is solved. Read the task version, tool conditions, attempt budget, and audit coverage before the percentage. Scores remain useful when understood together with the experiments that produce them.

This article was edited using primary material reviewed through October 8, 2026. Physics audit figures refer to arXiv:2609.13009v1. Epoch and OpenAI reviewed different objects with different sampling methods; their findings cannot be pooled into an industry-wide defect rate. The regression-testing and reporting proposals here are the author's engineering suggestions, not a claimed reproduction of these studies.