Chapter 15 — Emergent Behavior & Scaling Laws

Overview

Purpose of the Chapter

This chapter provides a quantitative framework for understanding how capability, efficiency, and risk evolve as model, data, and compute scale. It distinguishes predictable scaling trends from threshold-like emergent behavior, enabling evidence-based decisions about training investment, evaluation design, and deployment readiness.

Role in Book Arc

This chapter explains how and why performance and behavior change nonlinearly as models, data, and compute scale. It extends the prior optimization-and-alignment chapters by showing that capability growth is often regime-dependent: smooth training curves can still yield threshold-like capability jumps with major implications for planning, safety, and governance.

Core Concept and Supporting Concepts

Main Concept: Scaling laws provide predictable average trends, while emergent behavior captures threshold-like capability changes that appear when systems cross representational or optimization regime boundaries.

Supporting Concepts:

Scaling is structured: power-law fits often predict diminishing returns in aggregate metrics.
Emergence can be abrupt: some capabilities appear after crossing critical scale thresholds.
Trend and threshold coexist: smooth loss improvements can coincide with sharp task-level jumps.
Exponents are strategic: scaling coefficients determine whether additional budget is worthwhile.
Regimes can shift: optimization dynamics and representation geometry can change with scale.
Noise can fake emergence: measurement design must separate real transitions from artifacts.
Domain returns differ: one domain may scale efficiently while another saturates.
Capability and risk can co-scale: safety-relevant behaviors may worsen as competence increases.
Saturation is actionable: weak scaling returns signal need for data/objective redesign.
Planning must be quantitative: release and investment decisions should use explicit scaling forecasts.

Learning Outcomes

By the end of this chapter, you will be able to:

Fit and interpret empirical scaling laws for model, data, and compute axes.
Estimate marginal gains from additional scale and compare against budget constraints.
Identify candidate emergence thresholds in capability benchmarks.
Distinguish true regime shifts from evaluation noise or metric artifacts.
Compare scaling exponents across domains to prioritize investment.
Diagnose saturation and determine when brute-force scaling is insufficient.
Relate representation changes to abrupt downstream performance jumps.
Evaluate capability-risk co-scaling against policy gates.
Design monitoring protocols near critical scales where behavior may change rapidly.
Use scaling evidence to support deployment and governance decisions.

Scope: What This Chapter Covers

This chapter covers six tightly connected themes.

Empirical scaling laws: power-law formulations, fitting practice, and forecast limits.
Emergent capabilities: threshold-style gains and phase-transition-like behavior.
Regime shifts: changing optimization and representation dynamics at larger scales.
Domain heterogeneity: unequal returns across tasks and data distributions.
Safety coupling: capability growth versus risk growth under deployment policy.
Operational planning: compute allocation, evaluation design, and release gating.

Connections to Other Chapters

This chapter connects directly to earlier foundations and later operational material.

Optimization chapters: links training dynamics to scale-dependent performance outcomes.
Constraint/alignment chapter: explains why compliance can change sharply across scales.
Distribution-shift chapter: complements drift analysis with model-capability evolution.
Evaluation chapters: motivates denser testing near predicted threshold regions.
Monitoring/governance chapters: supports adaptive control as capabilities change nonlinearly.
Systems chapters: informs compute-budget and release-timing strategy.

Questions This Chapter Answers

This chapter addresses the critical questions teams face when scaling frontier models.

What do scaling laws actually predict? And where do those predictions fail?
When does emergence occur? How can we detect credible capability thresholds?
How should compute be allocated? What gain should we expect from doubling scale?
Why do some domains scale poorly? Which bottlenecks are data vs architecture driven?
How can evaluation avoid false threshold claims? What controls are essential?
When should we stop scaling? How do we identify saturation early?
How does risk move with capability? Which safety metrics must co-evolve?
What governance policy fits nonlinear growth? Which release gates remain robust?
What signals indicate a regime shift? Which diagnostics should be monitored?
How does this alter long-term roadmap planning? What decisions become time-sensitive?

Concrete ML Examples

This purpose section grounds theory in practical forecasting, threshold detection, and release decisions.

Neural Scaling-Law Forecasting for Compute Planning
1. 1) Concept summary: scaling-law fits predict expected loss gains from additional model/data/compute before costly runs.
2. 2) Problem statement: estimate loss improvement from doubling compute under a fitted power-law curve.
3. 3) Problem setup: We use an empirical compute-scaling fit in a stable training regime to forecast returns. Planning compares projected gain with budget impact to decide whether to scale up. This avoids expensive blind trials at frontier training sizes.
4. 4) Explicit values: fitted law $L(C)=aC^{-\gamma}+L_\infty$ with $a=1.2$, $\gamma=0.20$, $L_\infty=1.50$, current compute $C_1=1$, candidate $C_2=2$.
5. 5) Formula with symbols defined: predicted losses $L_1=L(C_1)$, $L_2=L(C_2)$, gain $\Delta L=L_1-L_2$.
6. 6) Plug-in step: $L_1=1.2(1)^{-0.20}+1.50=2.70$; $2^{-0.20}\approx0.871$, so $L_2=1.2(0.871)+1.50\approx2.545$.
7. 7) Computed result: $\Delta L\approx2.700-2.545=0.155$.
8. 8) Decision / interpretation: doubling compute gives moderate improvement; proceed only if this gain justifies added training cost.
9. 9) Sensitivity check: if $\gamma$ were 0.10, gain drops substantially, indicating stronger diminishing returns and weaker scaling incentive.
Emergent In-Context Learning Thresholds
1. 1) Concept summary: some capabilities appear abruptly after crossing scale thresholds rather than growing linearly.
2. 2) Problem statement: detect a practical emergence threshold from capability scores across model scales.
3. 3) Problem setup: We evaluate a held-out reasoning benchmark on successive model sizes with identical prompting and contamination controls. Emergence is flagged when score jump exceeds a predefined operational threshold over one scale step. We then decide if incremental scaling is strategically worthwhile.
4. 4) Explicit values: scores by size: 1B $=0.09$, 3B $=0.11$, 7B $=0.14$, 13B $=0.41$; emergence trigger $\Delta\ge0.20$.
5. 5) Formula with symbols defined: step jump $\Delta_k=s_k-s_{k-1}$, where $s_k$ is capability score at checkpoint $k$.
6. 6) Plug-in step: jumps are $0.02$, $0.03$, and $0.27$ for transitions 1B->3B, 3B->7B, 7B->13B.
7. 7) Computed result: only $7\text{B}\rightarrow13\text{B}$ exceeds emergence trigger.
8. 8) Decision / interpretation: threshold-like behavior appears near 13B, supporting targeted scaling to unlock this capability tier.
9. 9) Sensitivity check: if evaluation noise is +/-0.05, re-run with larger test set to confirm jump is not a measurement artifact.
Power-Law Error Decomposition Across Domains
1. 1) Concept summary: domain-specific scaling exponents reveal where brute-force scaling is effective versus insufficient.
2. 2) Problem statement: compare predicted error reduction across two domains when scale doubles.
3. 3) Problem setup: We fit per-domain laws $L_k(S)=A_kS^{-\alpha_k}+B_k$ and simulate a scale increase. Different exponents imply unequal returns from the same investment. This guides domain-targeted data or objective improvements.
4. 4) Explicit values: Domain A: $A_A=1.0, \alpha_A=0.30, B_A=0.10$. Domain B: $A_B=1.2, \alpha_B=0.08, B_B=0.20$. Compare $S=1$ vs $S=2$.
5. 5) Formula with symbols defined: $L_k(1)=A_k+B_k$, $L_k(2)=A_k2^{-\alpha_k}+B_k$, gain $\Delta_k=L_k(1)-L_k(2)$.
6. 6) Plug-in step: A: $L_A(1)=1.10$, $2^{-0.30}\approx0.812$, $L_A(2)=0.912$, $\Delta_A=0.188$. B: $L_B(1)=1.40$, $2^{-0.08}\approx0.946$, $L_B(2)=1.335$, $\Delta_B=0.065$.
7. 7) Computed result: domain A gains about 2.9x more error reduction than domain B from the same scale increase.
8. 8) Decision / interpretation: scaling budget is better spent for domain A; domain B needs curation or architecture changes.
9. 9) Sensitivity check: if domain-B exponent improves after data cleaning (for example to 0.18), scaling becomes more competitive there.
Capability-Risk Co-Scaling in Frontier Models
1. 1) Concept summary: release decisions should jointly track capability growth and risk growth as scale increases.
2. 2) Problem statement: decide whether a new model meets dual-threshold release policy on capability and risk.
3. 3) Problem setup: We evaluate a candidate against product minimum capability and maximum allowable risk under stress tests. Passing only one axis is insufficient for launch. Governance uses explicit gates to prevent unsafe capability scaling.
4. 4) Explicit values: capability score $C=0.83$, minimum $C_{min}=0.80$; jailbreak success rate $R=3.2\%$, maximum $R_{max}=2.5\%$.
5. 5) Formula with symbols defined: release requires $C\ge C_{min}$ and $R\le R_{max}$.
6. 6) Plug-in step: capability check: $0.83\ge0.80$ pass; risk check: $3.2\%\le2.5\%$ fail.
7. 7) Computed result: model fails policy by $0.7$ percentage points on risk despite adequate capability.
8. 8) Decision / interpretation: block release and require additional alignment hardening before deployment.
9. 9) Sensitivity check: if mitigation lowers risk to 2.3% while preserving capability above 0.80, candidate becomes eligible.

Definitions

Scaling Law

Definition: A scaling law is a parametric relationship between a performance metric and a scale variable, typically of the form $M(s) = a s^{-\alpha} + b$ or $M(s) = a s^{-\alpha}$, where $s$ denotes scale (parameters, data, or compute), $a > 0$, $\alpha > 0$, and $b$ is an irreducible error floor.
Assumptions: The metric is measured under a controlled training protocol, the scale variable is monotonically increased, and the regime is sufficiently wide that a single parametric form fits across multiple orders of magnitude.
Notation: Use $s$ for scale, $M(s)$ for the metric, and $\alpha$ for the scaling exponent. Do not overload $\alpha$ as a learning rate in this section.
Usage: Scaling laws quantify diminishing returns: improvements shrink at rate $s^{-\alpha}$ as scale increases. The exponent $\alpha$ is the slope on a log-log plot and determines how costly additional gains are.
Valid Example: A language model with cross-entropy loss $L(P) = 2.4 P^{-0.095} + 1.7$ as a function of parameter count $P$ exhibits a scaling law with exponent $0.095$.
Failure Case: If optimization changes between scales (for example, different tokenization, curriculum, or optimizer), a single scaling law may not fit across scales, and the power-law form can fail.
Explicit ML Relevance: Scaling laws guide decisions on whether to invest in more parameters, data, or compute when training large models and help estimate the cost of performance targets.

Power-Law Relationship

Definition: A power-law relationship is a functional form $y = c x^{-\a\\alpha}$ or $y = c x^{\a\\alpha}$ with $c > 0$ and $\a\\alpha \in \mathbb{R}$, implying a linear relationship in log-log coordinates: $\log y = \log c - \a\\alpha \log x$.
Assumptions: The variables are positive, the log transformation is valid, and the relationship is stable across the range of interest.
Notation: Use $x$ for the independent variable and $y$ for the dependent variable. Keep $\a\\alpha$ reserved for the exponent only.
Usage: Power laws indicate scale-free behavior. In ML, they often describe how error decreases as model size or data size grows.
Valid Example: Perplexity of a transformer can follow $\t\\text{PPL}(N) = 10.5 N^{-0.12}$ where $N$ is the number of training tokens.
Failure Case: If $y$ saturates due to a noise floor or data mismatch, the power-law form fails and produces systematic prediction errors.
Explicit ML Relevance: Power laws underpin empirical scaling curves for language models, vision models, and speech recognition systems.

Parameter Count

Definition: Parameter count is the number of free scalar parameters $P$ in a model, typically the dimension of its parameter vector $\theta \in \mathbb{R}$.
Assumptions: Parameters are trainable and not tied or frozen; parameter sharing is counted once per shared parameter.
Notation: Use $P$ for parameter count and $\theta$ for parameters. Do not mix $P$ with data size $N$.
Usage: Parameter count measures representational capacity but does not alone determine performance because data quality and optimization matter.
Valid Example: A transformer with 32 layers, 4096 hidden size, and 32 attention heads may have $P \a\\approx 7 \times 10^9$ parameters.
Failure Case: Comparing models by parameter count alone can be misleading if one uses higher quality data or more compute, resulting in better performance despite fewer parameters.
Explicit ML Relevance: Parameter count is a common axis in scaling laws and a key driver of training cost and memory usage.

Data Scale

Definition: Data scale is the effective amount of training data used, often measured by the number of tokens, samples, or labeled examples $N$.
Assumptions: Samples are drawn from a stable distribution and are not heavily duplicated; the measure reflects distinct informational content.
Notation: Use $N$ for number of data points or tokens. Avoid reusing $N$ for batch size in this chapter.
Usage: Larger data scale improves generalization by reducing variance and exposing the model to more rare patterns.
Valid Example: A language model trained on $N = 2 \times 10^{12}$ tokens typically generalizes better than a similar model trained on $2 \times 10^{11}$ tokens, holding compute constant.
Failure Case: If the additional data is low quality or domain mismatched, performance may stagnate or degrade despite larger $N$.
Explicit ML Relevance: Data scale is a primary factor in compute-optimal training and is often the bottleneck in large-scale learning.

Compute Scale

Definition: Compute scale is the total floating point operations used in training, often denoted $C$, and a\approximated as $C \a\\approx k N P$ for transformer-style models with constant factor $k$.
Assumptions: The training algorithm is fixed, and compute is dominated by the forward and backward passes.
Notation: Use $C$ for compute, $N$ for data scale, and $P$ for parameter count. Do not conflate $C$ with wall-clock time.
Usage: Compute scale determines how far a model can be optimized and how fully it can fit the data distribution.
Valid Example: Two models with the same parameter count but different compute budgets can show different performance due to undertraining or overtraining.
Failure Case: If the optimizer or hardware limits training stability, additional compute may not translate to better loss.
Explicit ML Relevance: Compute scaling guides budget allocation for large training runs and informs whether a model is undertrained.

Emergent Behavior

Definition: Emergent behavior is a qualitative capability that becomes measurable only above a scale threshold, even if lower-scale models show negligible or zero performance on the same metric.
Assumptions: The capability metric is well defined, measured consistently across scales, and does not saturate at low performance levels.
Notation: Use $s$ for scale and $m(s)$ for the capability metric. A capability is emergent if $m(s) \a\\approx 0$ for $s < s_0$ and $m(s) > \epsilon$ for $s \geq s_0$.
Usage: Emergence indicates that a new representational regime has become accessible, not necessarily that the system is fundamentally different.
Valid Example: Few-shot chain-of-thought reasoning often appears only beyond a model size threshold in large language models.
Failure Case: Apparent emergence may be an artifact of evaluation, such as a metric with a hard threshold or insufficiently sensitive measurement at small scale.
Explicit ML Relevance: Emergent behaviors complicate risk assessment because new capabilities can appear suddenly and without gradual warning signs.

Phase-Transition-Style Behavior

Definition: Phase-transition-style behavior is a sharp change in an observable metric $m(s)$ over a small interval of a control variable $s$, with a steep gradient $|m'(s)|$ relative to adjacent regions.
Assumptions: The control variable is continuous or finely discretized, and the metric is measured at sufficient resolution to detect steep changes.
Notation: Use $s$ for the control variable and $m(s)$ for the observable metric. A transition occurs near $s_0$ if $|m'(s_0)|$ is large and localized.
Usage: Phase-transition-style behavior signals a regime boundary in learning dynamics or representation structure.
Valid Example: Accuracy on multi-step arithmetic may jump from near chance to high accuracy across a narrow range of model sizes.
Failure Case: Noisy measurements or wide confidence intervals can create an illusion of a transition that disappears with repeated trials.
Explicit ML Relevance: Phase-transition-style behavior suggests that monitoring must be granular near critical scales.

Regime Shift

Definition: A regime shift is a change in the dominant optimization or generalization behavior as scale increases, such as a transition from high-gradient-noise training to low-noise, curvature-dominated training.
Assumptions: The training protocol is held fixed while scale varies, and the observed change is not due to hyperparameter retuning.
Notation: Use $s$ for scale and $\mathcal{R}_1, \mathcal{R}_2$ for reg\times. A shift occurs when a diagnostic statistic changes qualitatively, for example $\t\\text{SNR}(\nabla f)$ crossing a threshold.
Usage: Regime shifts explain why small-scale extrapolations often underpredict large-scale capabilities.
Valid Example: Large-batch training becomes stable only after a certain model size because gradient noise decreases with scale.
Failure Case: If optimizer changes with scale, the observed shift may be confounded and not attributable to inherent model scaling.
Explicit ML Relevance: Regime shifts inform when to adjust learning rates, batch sizes, or regularization strategies at scale.

Representation Geometry

Definition: Representation geometry refers to the geometric structure of feature vectors $z_i \in \mathbb{R}^d$, measured by distances, angles, and subspace structure induced by the learned model.
Assumptions: Representations are compared in a fixed embedding space and are normalized or scaled consistently.
Notation: Use $Z = [z_1, \ldots, z_n]$ for representations and $G = Z Z$ for the Gram matrix.
Usage: Geometry determines linear separability, cluster structure, and transferability to downstream tasks.
Valid Example: As model size grows, the cosine similarity between representations of synonymous sentences increases, indicating semantic clustering.
Failure Case: If representations collapse to a single point, geometry loses discriminative power and downstream tasks fail.
Explicit ML Relevance: Representation geometry connects scaling to downstream performance via linear probing and clustering analyses.

Latent Structure

Definition: Latent structure is the hidden factorization of data into lower-dimensional variables $h$ such that $x = g(h, \epsilon)$ for some generative function $g$ and noise $\epsilon$.
Assumptions: The data distribution admits a low-dimensional generative explanation and the model has capacity to represent it.
Notation: Use $h$ for latent variables and $x$ for observed data. Avoid using $z$ for both latent variables and representations in the same derivation.
Usage: Discovering latent structure enables disentanglement and improves sample efficiency.
Valid Example: In vision models, latent structure might correspond to pose, lighting, and object identity.
Failure Case: If the assumed latent structure is incorrect (for example, non-factorized or strongly entangled), learned representations can be unstable or misleading.
Explicit ML Relevance: Emergent capabilities often reflect the model discovering latent structure that was previously inaccessible at smaller scales.

Sparse Activation

Definition: Sparse activation means that for a representation vector $z \in \mathbb{R}^d$, only a small fraction of components are non-zero or significantly non-zero, typically $\|z\|_0 \ll d$ or $\|z\|_1 / \|z\|_2$ is small.
Assumptions: Activations are measured after a fixed nonlinearity and thresholding is consistent across samples.
Notation: Use $\|z\|_0$ for the number of non-zero elements and $k$ for sparsity level.
Usage: Sparse activation indicates specialized feature detectors and reduces interference between features.
Valid Example: Mixture-of-experts models often route tokens to a small subset of experts, creating sparse activation patterns.
Failure Case: Excessive sparsity can cause underutilization of capacity and brittle generalization.
Explicit ML Relevance: Sparse activation is linked to interpretability and scaling efficiency in large models.

Hierarchical Composition

Definition: Hierarchical composition is the repeated composition of functions, $f(x) = f_L(\cdots f_2(f_1(x))\cdots)$, where each layer builds increasingly abstract representations.
Assumptions: Each layer is a non-linear map and has sufficient capacity to transform inputs without collapsing them.
Notation: Use $f_1, \ldots, f_L$ for layers, and $L$ for depth. Avoid using $L$ for loss in this section.
Usage: Hierarchical composition explains why depth can yield exponentially many linear regions in piecewise-linear networks.
Valid Example: A ReLU network with depth $L$ can represent functions with $O(2)$ linear regions in 1D input space.
Failure Case: If layers are linear or saturating, composition reduces to a single linear map and depth yields no additional expressivity.
Explicit ML Relevance: Emergent behaviors can arise from deep compositional structure enabling feature reuse across layers.

Capacity Threshold

Definition: A capacity threshold is the minimum model capacity required to represent a target function class, often quantified by VC dimension or effective rank.
Assumptions: The target function class is well defined, and capacity is measured by a consistent complexity measure (VC dimension, Rademacher complexity, or parameter count under regularization).
Notation: Use $\t\\text{VC}( \mathcal{H})$ for the VC dimension of hypothesis class $\mathcal{H}$, and $k$ for the task complexity threshold.
Usage: If capacity is below threshold, the model cannot realize the task; above threshold, it can in principle represent it, though optimization may still fail.
Valid Example: Linear separators in $\mathbb{R}^d$ have VC dimension $d+1$; tasks requiring shattering of $m > d+1$ points exceed capacity.
Failure Case: Increasing capacity without sufficient data can lead to overfitting and poor generalization despite representational sufficiency.
Explicit ML Relevance: Capacity thresholds explain why some tasks only become solvable when models scale past a size or depth limit.

Double Descent (Revisited)

Definition: Double descent is the phenomenon where test error decreases for small models, increases near the interpolation threshold, and decreases again for overparameterized models.
Assumptions: The estimator is interpolating (zero training error), the data contains noise, and the model class grows by increasing parameters or features.
Notation: Use $n$ for number of samples, $d$ for model dimension, and $R(d)$ for test risk as a function of $d$.
Usage: Double descent indicates that model size affects variance differently in underparameterized and overparameterized reg\times.
Valid Example: Minimum-norm linear regression with Gaussian features shows a risk peak near $d \a\\approx n$ and improved risk for $d \gg n$.
Failure Case: With strong regularization or early stopping, the interpolation peak may be suppressed and double descent may not appear.
Explicit ML Relevance: Double descent helps explain why very large models can generalize well despite perfect training fit.

Overparameterized Regime

Definition: The overparameterized regime occurs when model dimension $d$ exceeds the number of training samples $n$ or effective degrees of freedom, enabling zero training error.
Assumptions: The model class can interpolate the training data, and optimization finds an interpolating solution.
Notation: Use $d > n$ to denote overparameterization. Do not conflate parameter count $P$ with intrinsic dimension $d$.
Usage: In this regime, inductive bias (for example, minimum norm) becomes the main driver of generalization.
Valid Example: Deep networks with millions of parameters trained on datasets with tens of thousands of samples often operate overparameterized.
Failure Case: If optimization fails to interpolate (due to bad initialization or optimization instability), overparameterization does not guarantee zero training error.
Explicit ML Relevance: Overparameterization is standard in modern deep learning and is central to scaling behavior.

Saturation Effects

Definition: Saturation effects are diminishing returns where additional scaling produces negligible improvement because the metric approaches a floor $b$, such that $M(s) \to b$ as $s \to \infty$.
Assumptions: There exists an irreducible error floor, often due to label noise, domain mismatch, or evaluation limits.
Notation: Use $b$ for the floor and $M(s)$ for the metric.
Usage: Saturation indicates that the current data or objective limits further gains and that scaling alone is inefficient.
Valid Example: On a noisy dataset with 5 percent label noise, test error may saturate around 5 percent even with huge models.
Failure Case: If new data or better objectives are introduced, the apparent saturation can disappear, showing that the previous floor was not fundamental.
Explicit ML Relevance: Saturation effects inform when to shift focus from scaling to data curation or objective redesign.

Correlation Structure

Definition: Correlation structure refers to the covariance matrix $\Sigma = \mathbb{E}[x x]$ of inputs or features and its spectral properties, such as eigenvalue decay.
Assumptions: Inputs are centered and second moments exist. The covariance is estimated from sufficient data.
Notation: Use $\Sigma$ for covariance and $\lambda_i$ for eigenvalues ordered $\lambda_1 \geq \cdots \geq \lambda_d$.
Usage: Strong correlations (rapid eigenvalue decay) indicate low effective dimensionality and can improve sample efficiency.
Valid Example: Natural image patches often exhibit a power-law eigenvalue spectrum, implying low effective rank.
Failure Case: If covariance is near-isotropic (flat spectrum), low-rank assumptions fail and dimensionality reduction yields minimal gains.
Explicit ML Relevance: Correlation structure determines effective rank and influences scaling laws and generalization.

Feature Locking

Definition: Feature locking is the phenomenon where early-learned features become dominant and resist change during further training or scaling, effectively constraining representation updates.
Assumptions: Optimization is path dependent (for example, non-convex objectives), and training is performed with finite learning rate and limited regularization.
Notation: Use $\phi_t$ for features at time $t$ and $\Delta \phi_t$ for feature updates. Feature locking implies $\|\Delta \phi_t\|$ remains small after an early phase.
Usage: Feature locking can explain why models fail to adapt to new data distributions without re-initialization or curriculum changes.
Valid Example: A vision model trained first on a biased dataset can lock onto background cues, and later fine-tuning may not remove this bias.
Failure Case: If training includes strong regularization or explicit feature re-learning mechanisms, feature locking may not occur.
Explicit ML Relevance: Feature locking affects transfer learning, continual learning, and fairness when early learned biases persist.

Combinatorial Complexity

Definition: Combinatorial complexity measures the number of distinct configurations or functions representable by a model, often scaling exponentially with depth or width.
Assumptions: The model uses non-linearities that enable combinatorial composition and the parameters are unconstrained.
Notation: Use $\mathcal{N}(L)$ for the number of linear regions or configurations as a function of depth $L$.
Usage: Combinatorial growth explains why deep models can represent highly complex functions with moderate parameter counts.
Valid Example: A ReLU network with depth $L$ and width $m$ can implement $O(m)$ linear regions in 1D, indicating exponential expressivity in depth.
Failure Case: If activations saturate (for example, sigmoid near 0 or 1), combinatorial complexity collapses and depth yields limited gains.
Explicit ML Relevance: Combinatorial complexity is a core reason emergence is associated with depth and scale.

Theorems

Power-Law Scaling Bound

Formal statement. Assume a metric $M(s) = a s^{-\a\\alpha} + b$ with $a > 0$, $\a\\alpha > 0$, $b \geq 0$, and scale $s > 0$. Then for any $s_2 > s_1 > 0$, \[ M(s_2) - b \leq \left(\frac{s_1}{s_2}\right)^{\a\\alpha} (M(s_1) - b). \]

Full formal proof. By definition, \[ M(s_1) - b = a s_1^{-\a\\alpha}, \quad M(s_2) - b = a s_2^{-\a\\alpha}. \] Because $a > 0$ and $s_2 > s_1$, we have \[ \frac{M(s_2) - b}{M(s_1) - b} = \frac{a s_2^{-\a\\alpha}}{a s_1^{-\a\\alpha}} = \left(\frac{s_1}{s_2}\right)^{\a\\alpha}. \] Multiplying both sides by $M(s_1) - b$ yields the bound: \[ M(s_2) - b = \left(\frac{s_1}{s_2}\right)^{\a\\alpha} (M(s_1) - b). \] Since equality holds, the inequality is true. $\square$

Interpretation. The improvement in the metric decays as a power of scale. Doubling scale yields a multiplicative improvement of $2^{-\a\\alpha}$ in the distance to the floor $b$.

Explicit ML relevance. This bound formalizes the diminishing returns observed in empirical scaling laws for language models and helps estimate the required scale for a target performance.

Bias-Variance Scaling Decomposition

Formal statement. Let $y = f(x) + \epsilon$ with $\mathbb{E}[\epsilon] = 0$ and $\mathbb{E}[\epsilon^2] = \sigma^2$. For an estimator $\hat{f}(x)$ trained on a random dataset $\mathcal{D}$, the expected squared error decomposes as \[ \mathbb{E}_{\mathcal{D}, \epsilon}[(\hat{f}(x) - y)^2] = \underbrace{(\mathbb{E}_{\mathcal{D}}[\hat{f}(x)] - f(x))^2}_{\t\\text{bias}^2} + \underbrace{\mathbb{E}_{\mathcal{D}}[(\hat{f}(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)])^2]}_{\t\\text{variance}} + \sigma^2. \]

Full formal proof. We expand the error: \[ \hat{f}(x) - y = \hat{f}(x) - f(x) - \epsilon. \] Square and take expectations: \[ \mathbb{E}[(\hat{f}(x) - y)^2] = \mathbb{E}[(\hat{f}(x) - f(x))^2] + \mathbb{E}[\epsilon^2] - 2\mathbb{E}[(\hat{f}(x) - f(x))\epsilon]. \] Because $\epsilon$ is independent of $\hat{f}(x)$ and has zero mean, the cross term is zero: \[ \mathbb{E}[(\hat{f}(x) - f(x))\epsilon] = \mathbb{E}[\hat{f}(x) - f(x)] \mathbb{E}[\epsilon] = 0. \] Thus, \[ \mathbb{E}[(\hat{f}(x) - y)^2] = \mathbb{E}[(\hat{f}(x) - f(x))^2] + \sigma^2. \] Now decompose the first term using $\hat{f}(x) - f(x) = (\hat{f}(x) - \mathbb{E}[\hat{f}(x)]) + (\mathbb{E}[\hat{f}(x)] - f(x))$: \[ \mathbb{E}[(\hat{f}(x) - f(x))^2] = \mathbb{E}[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2] + (\mathbb{E}[\hat{f}(x)] - f(x))^2, \] because the cross term has zero expectation. Combining yields the result. $\square$

Interpretation. Error splits into irreducible noise, bias from insufficient capacity, and variance from finite data.

Explicit ML relevance. This decomposition explains why scaling data reduces variance while scaling model size reduces bias until variance dominates.

Double Descent in Overparameterized Models

Formal statement. Consider linear regression with data $X \in \mathbb{R}^{n \times d}$ whose rows are i.i.d. $\mathcal{N}(0, I_d)$, labels $y = X \beta + \epsilon$ with $\epsilon \sim \mathcal{N}(0, \sigma^2 I_n)$, and the minimum-norm interpolator $\hat{\beta} = X (X X)^{-1} y$ for $d > n$ and $\hat{\beta} = (X X)^{-1} X y$ for $d < n$. Then the expected prediction risk at a fresh test point $x \sim \mathcal{N}(0, I_d)$ satisfies: \[ \mathbb{E}[(x \hat{\beta} - x \beta)^2] = \sigma^2 \cdot \frac{d}{n-d-1} + \mathbb{E}[\|(I - P)\beta\|^2] \quad \t\\text{for } d < n-1, \] and \[ \mathbb{E}[(x \hat{\beta} - x \beta)^2] = \sigma^2 \cdot \frac{n}{d-n-1} + \mathbb{E}[\|(I - P)\beta\|^2] \quad \t\\text{for } d > n+1, \] where $P$ is the projection onto the column space of $X$. The variance term diverges as $d \to n$ and decreases for $d > n$, yielding a double-descent curve.

Full formal proof. We derive the prediction error. For a test point $x$, the prediction error is \[ x \hat{\beta} - x \beta = x (\hat{\beta} - \beta). \] Conditioning on $X$, we have \[ \mathbb{E}_x[(x (\hat{\beta} - \beta))^2 \mid X] = \mathbb{E}_x[(\hat{\beta} - \beta) x x (\hat{\beta} - \beta) \mid X] = \|\hat{\beta} - \beta\|^2, \] since $\mathbb{E}[x x] = I_d$. Thus \[ \mathbb{E}[(x \hat{\beta} - x \beta)^2] = \mathbb{E}[\|\hat{\beta} - \beta\|^2]. \] We now compute $\hat{\beta} - \beta$. For $d < n$, \[ \hat{\beta} = (X X)^{-1} X (X \beta + \epsilon) = \beta + (X X)^{-1} X \epsilon. \] Thus \[ \hat{\beta} - \beta = (X X)^{-1} X \epsilon. \] Then \[ \mathbb{E}[\|\hat{\beta} - \beta\|^2 \mid X] = \mathbb{E}[\epsilon X (X X)^{-2} X \epsilon \mid X] = \sigma^2 \operatorname{tr}(X (X X)^{-2} X). \] Using $\operatorname{tr}(X (X X)^{-2} X) = \operatorname{tr}((X X)^{-1})$, we get \[ \mathbb{E}[\|\hat{\beta} - \beta\|^2 \mid X] = \sigma^2 \operatorname{tr}((X X)^{-1}). \] For Gaussian design with $X$ having i.i.d. $\mathcal{N}(0,1)$ rows, $X X$ is Wishart with $n$ degrees of freedom. The expectation of its inverse is \[ \mathbb{E}[(X X)^{-1}] = \frac{I_d}{n-d-1} \quad \t\\text{for } n > d+1. \] Therefore, \[ \mathbb{E}[\|\hat{\beta} - \beta\|^2] = \sigma^2 \frac{d}{n-d-1}. \] For $d > n$, the minimum-norm interpolator is \[ \hat{\beta} = X (X X)^{-1} y = X (X X)^{-1} (X \beta + \epsilon). \] Let $P = X (X X)^{-1} X$, the projection onto the row space of $X$. Then \[ \hat{\beta} = P \beta + X (X X)^{-1} \epsilon, \] so \[ \hat{\beta} - \beta = (P - I)\beta + X (X X)^{-1} \epsilon. \] The error splits into bias and variance: \[ \mathbb{E}[\|\hat{\beta} - \beta\|^2] = \mathbb{E}[\|(I - P)\beta\|^2] + \mathbb{E}[\|X (X X)^{-1} \epsilon\|^2]. \] For the variance term, conditioning on $X$: \[ \mathbb{E}[\|X (X X)^{-1} \epsilon\|^2 \mid X] = \sigma^2 \operatorname{tr}(X (X X)^{-2} X) = \sigma^2 \operatorname{tr}((X X)^{-1}). \] For Gaussian design, $X X$ is Wishart with $d$ degrees of freedom, and \[ \mathbb{E}[(X X)^{-1}] = \frac{I_n}{d-n-1} \quad \t\\text{for } d > n+1. \] Thus \[ \mathbb{E}[\|X (X X)^{-1} \epsilon\|^2] = \sigma^2 \frac{n}{d-n-1}. \] Combining yields the statement. As $d \to n$, the variance term diverges; for $d > n$, it decreases with $d$, producing double descent. $\square$

Interpretation. Under isotropic Gaussian design, the variance term has a pole at interpolation and decreases in the overparameterized regime, explaining the second descent.

Explicit ML relevance. This theorem provides a concrete mechanism for why very large models can generalize better than medium-sized ones despite perfect training fit.

Capacity Threshold Theorem

Formal statement. Let $\mathcal{H}$ be a hypothesis class with VC dimension $d_{\t\\text{VC}}$. For any integer $m$, if $m \leq d_{\t\\text{VC}}$, then there exists a set of $m$ points that $\mathcal{H}$ can shatter; if $m > d_{\t\\text{VC}}$, then no set of $m$ points is shattered by $\mathcal{H}$.

Full formal proof. By definition of VC dimension, $d_{\t\\text{VC}}$ is the largest integer such that some set of size $d_{\t\\text{VC}}$ is shattered by $\mathcal{H}$. Therefore, for all $m \leq d_{\t\\text{VC}}$, there exists a set of size $m$ that is shattered, because any subset of a shattered set is also shattered: for any labeling on the subset, \textend it to the full shattered set and then restrict the hypothesis to the subset. Conversely, for $m > d_{\t\\text{VC}}$, suppose there existed a set of size $m$ that is shattered. Then $\mathcal{H}$ would shatter a set of size $m$, contradicting maximality of $d_{\t\\text{VC}}$. Thus no such set exists. $\square$

Interpretation. A hypothesis class has a sharp capacity threshold: it can represent all labelings up to size $d_{\t\\text{VC}}$, but not beyond.

Explicit ML relevance. Capacity thresholds explain why certain tasks become solvable only when model capacity crosses a critical size.

Phase Transition in Linear Models

Formal statement. The class of linear separators in $\mathbb{R}^d$ has VC dimension $d+1$. Therefore, for any set of $m \leq d+1$ points in general position, every labeling is realizable by a linear separator, whereas for some sets of $m = d+2$ points, not all labelings are realizable.

Full formal proof. First, we show that $d+1$ points in general position can be shattered. Let $x_1, \ldots, x_{d+1}$ be affinely independent. For any labeling $y_i \in \{-1, +1\}$, there exists an affine hyperplane $w x + b = 0$ that separates the labeled points. This follows because the convex hulls of the positively and negatively labeled points are disjoint for affinely independent points, and by the separating hyperplane theorem there exists a hyperplane separating them. Hence the class shatters $d+1$ points, so $d_{\t\\text{VC}} \geq d+1$.

N\text, we show that $d_{\t\\text{VC}} \leq d+1$. Consider any set of $m = d+2$ points. By Radon’s theorem, the set can be partitioned into two subsets whose convex hulls intersect. Therefore, there exists a labeling that assigns opposite labels to the two subsets yet cannot be separated by a hyperplane, because any hyperplane separating them would also separate their convex hulls, contradicting the intersection. Thus not all labelings are realizable, and the class cannot shatter $d+2$ points. Hence $d_{\t\\text{VC}} = d+1$. $\square$

Interpretation. The transition from realizable to non-realizable labelings occurs around $m = d+1$, a phase-transition-style boundary in linear separability.

Explicit ML relevance. This theorem explains why linearly separable behavior can suddenly appear as model dimension crosses a threshold.

Emergence from Composition Depth

Formal statement. Consider a ReLU network in one dimension with depth $L$ and width $m$ at each layer. There exists a choice of parameters such that the network represents a piecewise-linear function with at least $m$ linear regions.

Full formal proof. We prove by induction on depth $L$. For $L = 1$, a width-$m$ ReLU network is a sum of $m$ hinge functions and can create at least $m$ linear regions on $\mathbb{R}$. Assume the claim holds for depth $L-1$, so there exists a network producing a function with at least $m^{L-1}$ linear regions. Now construct a depth-$L$ network by composing the depth-$L-1$ network with a width-$m$ ReLU layer that introduces $m$ new breakpoints within each existing region. Because ReLU is piecewise linear and the previous function is linear on each region, the new layer can be parameterized to create $m$ subregions inside each of the $m^{L-1}$ regions. Therefore the total number of regions is at least $m \cdot m^{L-1} = m$. This completes the induction. $\square$

Interpretation. Depth increases expressivity exponentially through composition, enabling emergent behaviors not present in shallow networks.

Explicit ML relevance. This theorem formalizes why deeper architectures can exhibit sudden capability gains as depth increases.

Correlation Structure and Effective Rank

Formal statement. Let $\Sigma \in \mathbb{R}^{d \times d}$ be a covariance matrix with eigenvalues $\lambda_1 \geq \cdots \geq \lambda_d \geq 0$. Define the effective rank as $r_{\t\\text{eff}} = \frac{(\sum_i \lambda_i)^2}{\sum_i \lambda_i^2}$. Then $1 \leq r_{\t\\text{eff}} \leq d$, with equality $r_{\t\\text{eff}} = d$ if and only if all $\lambda_i$ are equal, and $r_{\t\\text{eff}} = 1$ if and only if only one $\lambda_i$ is non-zero.

Full formal proof. Let $s_1 = \sum_i \lambda_i$ and $s_2 = \sum_i \lambda_i^2$. By Cauchy-Schwarz, \[ s_1^2 = \left(\sum_i \lambda_i \cdot 1\right)^2 \leq \left(\sum_i \lambda_i^2\right) \left(\sum_i 1^2\right) = s_2 d, \] so $r_{\t\\text{eff}} = s_1^2 / s_2 \leq d$. Since $\lambda_i \geq 0$, we have $s_1^2 \geq s_2$, with equality only when all but one $\lambda_i$ are zero. Thus $r_{\t\\text{eff}} \geq 1$. The equality cases follow: if all $\lambda_i$ equal $c$, then $s_1 = dc$, $s_2 = dc^2$, hence $r_{\t\\text{eff}} = d$. If only one $\lambda_i$ is non-zero, then $s_1^2 = s_2$, hence $r_{\t\\text{eff}} = 1$. $\square$

Interpretation. Effective rank measures how many dimensions carry meaningful variance and thus quantifies correlation structure.

Explicit ML relevance. Effective rank predicts sample complexity and explains why correlated data can yield better generalization at fixed sample size.

Saturation Bound Under Fixed Data

Formal statement. Suppose data are generated by $y = f(x) + \epsilon$ with $\mathbb{E}[\epsilon^2] = \sigma^2$. For any estimator $\hat{f}$ trained on a fixed dataset of size $N$, the expected test error satisfies \[ \mathbb{E}[(\hat{f}(x) - y)^2] \geq \sigma^2. \] If the estimator interpolates the data and the noise is irreducible, then additional model capacity cannot reduce error below $\sigma^2$.

Full formal proof. From the bias-variance decomposition: \[ \mathbb{E}[(\hat{f}(x) - y)^2] = \t\\text{bias}^2 + \t\\text{variance} + \sigma^2. \] Both bias and variance are non-negative, so \[ \mathbb{E}[(\hat{f}(x) - y)^2] \geq \sigma^2. \] If $\sigma^2 > 0$, no estimator can reduce the expected error below $\sigma^2$, regardless of model capacity. Thus error saturates at the noise floor. $\square$

Interpretation. With fixed data and irreducible noise, scaling parameters cannot beat the noise floor.

Explicit ML relevance. This bound explains why performance saturates when data quality, not model size, is the limiting factor.

Representation Collapse Condition

Formal statement. Let $z_1, \ldots, z_n \in \mathbb{R}^d$ be learned representations that minimize the objective \[ J(Z) = \sum_{i=1}^n \|z_i - c\|^2, \] for some fixed $c$. Then the unique minimizer satisfies $z_i = c$ for all $i$, and the representation covariance has rank zero, indicating collapse.

Full formal proof. For each $i$, the term $\|z_i - c\|^2$ is minimized uniquely at $z_i = c$. Since the objective is a sum of independent non-negative terms, the global minimizer must satisfy $z_i = c$ for all $i$. The covariance matrix is \[ \Sigma = \frac{1}{n} \sum_{i=1}^n (z_i - \bar{z})(z_i - \bar{z}), \] where $\bar{z} = c$. Thus $z_i - \bar{z} = 0$ for all $i$, so $\Sigma = 0$ and has rank zero. $\square$

Interpretation. Without a term that spreads representations, the optimal solution collapses to a single point.

Explicit ML relevance. This condition explains why contrastive objectives require negative samples or variance-preserving terms to avoid collapse.

Scaling Instability Theorem

Formal statement. Let $f$ be $L$-smooth, meaning $\|\nabla f(x) - \nabla f(y)\| \leq L \|x - y\|$. Gradient descent with step size $\eta$ satisfies \[ f(x_{t+1}) \leq f(x_t) - \left(\eta - \frac{L \eta^2}{2}\right) \|\nabla f(x_t)\|^2. \] If $\eta > 2/L$, then the coefficient becomes negative and descent is not guaranteed. Therefore, if $L$ increases with scale while $\eta$ is fixed, training becomes unstable beyond the scale where $\eta > 2/L$.

Full formal proof. By L-smoothness, for any $x, y$, \[ f(y) \leq f(x) + \nabla f(x) (y - x) + \frac{L}{2} \|y - x\|^2. \] Set $y = x - \eta \nabla f(x)$: \[ f(x - \eta \nabla f(x)) \leq f(x) - \eta \|\nabla f(x)\|^2 + \frac{L \eta^2}{2} \|\nabla f(x)\|^2. \] Thus \[ f(x_{t+1}) \leq f(x_t) - \left(\eta - \frac{L \eta^2}{2}\right) \|\nabla f(x_t)\|^2. \] If $\eta > 2/L$, then $\eta - L \eta^2/2 < 0$, so the bound allows increases in $f$, and descent is not guaranteed. If $L$ grows with scale while $\eta$ is fixed, the condition $\eta \leq 2/L$ eventually fails, implying potential instability at larger scales. $\square$

Interpretation. Scaling can destabilize training if learning rates are not adapted to increasing smoothness constants.

Explicit ML relevance. This theorem explains why large models often require smaller learning rates or adaptive optimizers for stable training.

Worked Examples

Example 1 — Empirical Power-Law Fit

In this setup, you gather a family of models trained with a consistent pipeline and measure a single metric such as validation loss across a wide range of parameter counts or data sizes. The reasoning is to fit a power-law curve of the form $M(s) = a s^{-\a\\alpha} + b$ by regressing $\log(M(s) - b)$ on $\log s$, which reveals the exponent $\a\\alpha$ and the error floor $b$. The interpretation is that the slope on a log-log plot encodes how costly improvements will be as scale increases, while the intercept suggests a regime-dependent floor that may reflect noise or evaluation limits. A common misconception is that the fitted curve proves causality or generalizes to any training change; in reality, altering tokenization, data mix, or optimizer can shift $\a\\alpha$ and invalidate the fit. In a what-if scenario where the dataset quality improves, the same scaling curve can shift downward, making the previously estimated floor $b$ non-binding, which signals that scaling alone was not the limiting factor. The explicit ML relevance is that reliable power-law fits inform budget allocation for large-scale language models and allow teams to estimate the cost of achieving a target perplexity before committing to a full training run.

Example 2 — Parameter Scaling in Linear Models

The setup considers a linear regression model with feature dimension $d$ and a fixed training dataset, then scales $d$ by adding features or basis functions while keeping the data distribution fixed. The reasoning is to study how risk decomposes into bias and variance as $d$ increases, explicitly tracking the interpolation threshold near $d \a\\approx n$, where $n$ is the sample size. The interpretation is that, in the underparameterized regime, increasing $d$ reduces bias while variance grows; beyond the interpolation threshold, implicit regularization (such as minimum norm solutions) can reduce variance, producing a second descent. A common misconception is to think that more parameters always mean overfitting; the linear model shows that the inductive bias and data geometry determine whether overparameterization helps. In a what-if scenario where you add strong regularization or early stopping, the double-descent peak may flatten, indicating that the observed scaling behavior is as much about optimization bias as it is about model size. The explicit ML relevance is that many deep models behave like their linearized counterparts in certain reg\times, so these linear scaling insights help explain why large neural networks can generalize even when they interpolate the training data.

Example 3 — Double Descent Simulation

The setup simulates a dataset with controlled noise, trains models with increasing capacity, and plots test error as capacity passes the interpolation threshold. The reasoning is to ensure the same data distribution, noise level, and training pipeline while varying model dimension, so that changes in test error can be attributed to capacity rather than confounds. The interpretation is a curve that decreases, spikes near $d \a\\approx n$, then decreases again as $d$ grows, consistent with the theoretical double descent. A common misconception is to interpret the peak as an unavoidable failure of large models; in reality, the peak depends on the training procedure and can be moderated by regularization or improved data. In a what-if scenario where the data noise is reduced or label noise is removed, the peak diminishes and the second descent b\begins earlier, showing how data quality shifts the threshold. The explicit ML relevance is that monitoring for a double-descent peak helps practitioners choose model sizes that avoid unstable generalization reg\times and informs whether to scale up or instead improve data.

Example 4 — Effective Rank Under Correlated Features

The setup constructs data with a controlled covariance spectrum, such as a power-law decay in eigenvalues, and computes the effective rank $r_{\t\\text{eff}}$ for varying correlation strength. The reasoning is that effective rank summarizes how many directions carry meaningful variance, which in turn shapes sample efficiency and generalization. The interpretation is that as correlations increase, $r_{\t\\text{eff}}$ drops, effectively reducing dimensionality, which can allow smaller models to perform well despite high ambient dimension. A common misconception is to equate dimensionality with feature count; effective rank shows that correlation structure is the true driver of complexity. In a what-if scenario where correlations become weaker or the data distribution shifts to a flatter spectrum, $r_{\t\\text{eff}}$ increases and models require more capacity to achieve the same performance. The explicit ML relevance is that feature correlation explains why some domains scale faster than others and why transfer learning can be effective when data is intrinsically low-rank.

Example 5 — Phase Transition in Overparameterized Regression

The setup trains a regression model while sweeping the ratio $d/n$ across the interpolation threshold and measures training error, test error, and stability metrics. The reasoning is to identify whether a sharp transition occurs at a critical ratio, consistent with phase-transition-style behavior. The interpretation is that the system may abruptly shift from underfitting to perfect interpolation with a narrow change in $d/n$, accompanied by a large spike in variance. A common misconception is to view the transition as a purely theoretical artifact; in practice, the sharpness depends on noise, data distribution, and optimization, but clear regime changes are often observable. In a what-if scenario where the training algorithm is changed to a constrained optimizer or where explicit regularization is introduced, the transition can smooth out, indicating that optimization choices can soften emergent regime boundaries. The explicit ML relevance is that large-scale model training can encounter sharp regime changes, so practitioners must monitor not just mean loss but also stability indicators when scaling capacity.

Example 6 — Representation Collapse in Deep Networks

The setup uses a deep encoder trained with a reconstruction or contrastive objective and intentionally removes variance-preserving terms, then inspects the representation covariance to check for collapse. The reasoning is that without explicit incentives to maintain diversity, the optimal solution can map all inputs to a single point, minimizing reconstruction error or trivial alignment criteria. The interpretation is a degenerate representation geometry with near-zero covariance and low effective rank, which destroys downstream utility. A common misconception is that stronger regularization always prevents collapse; in contrast, some regularizers can accelerate collapse by shrinking representations uniformly. In a what-if scenario where you add a variance term, a stop-gradient pathway, or negative samples, the collapse disappears, showing how objective design determines representation health. The explicit ML relevance is that collapse is a real risk in self-supervised learning and representation learning at scale, and it explains why contrastive and variance-preserving methods are required for stable emergent feature discovery.

Example 7 — Sparse Activation and Capacity

The setup analyzes sparse activation in a mixture-of-experts or gated network by measuring how many units activate per token and how that scales with model width. The reasoning is that sparsity reduces interference between features and effectively increases the usable capacity, because different data regions activate different subsets of parameters. The interpretation is that sparse activation can yield emergent performance gains even without increasing total parameter count, by improving specialization. A common misconception is that sparsity always improves efficiency; in practice, overly sparse routing can underutilize capacity and increase variance in optimization. In a what-if scenario where routing is softened or expert count is reduced, sparsity decreases and the model may lose specialized capabilities, revealing a trade-off between specialization and coverage. The explicit ML relevance is that sparse activation is a core scaling strategy for large language models and helps explain why capability jumps can occur when routing thresholds or expert counts cross critical values.

Example 8 — Hierarchical Composition and Emergence

The setup compares shallow and deep networks on tasks requiring compositional structure, such as hierarchical parsing or multi-step reasoning, while keeping parameter count comparable. The reasoning is that depth enables repeated composition, which exponentially increases the number of linear regions and the capacity for hierarchical features. The interpretation is that deeper models can solve tasks that shallow models cannot, even when overall parameter count is similar, indicating that emergence can be driven by depth rather than size alone. A common misconception is to attribute all emergent behavior to scale; here, depth acts as a distinct axis that can create new capabilities. In a what-if scenario where depth is increased but width is reduced to keep parameters fixed, performance may still improve on compositional tasks, demonstrating the primacy of hierarchical structure. The explicit ML relevance is that architecture choices, not only size, govern emergent behaviors in transformers and deep vision models.

Example 9 — Scaling Instability Under Limited Data

The setup holds data fixed while increasing model size and keeps the learning rate constant, then measures training stability, gradient norms, and divergence events. The reasoning is that the smoothness constant $L$ can increase with model size, shrinking the stable learning-rate window, so fixed learning rates can lead to instability at larger scale. The interpretation is that scaling without re-tuning can produce sudden divergence or oscillation, which can masquerade as emergent behavior if not diagnosed. A common misconception is that larger models always require less tuning because they generalize better; in fact, they can be more fragile to optimization hyperparameters. In a what-if scenario where learning rates are scaled down or adaptive optimizers are introduced, the instability may disappear, showing that the issue was optimization, not representational capacity. The explicit ML relevance is that safe scaling requires learning-rate schedules and stability checks that adapt to model size and data limits.

Example 10 — Loss Scaling in Language Models

The setup collects training runs of language models across a range of sizes and data budgets and fits loss curves, holding tokenization and evaluation constant. The reasoning is to disentangle parameter scaling from data scaling and isolate the compute-optimal frontier where additional parameters and data yield proportional gains. The interpretation is that the loss follows a predictable power-law in total compute, but only when the model is trained to the compute-optimal point; otherwise, undertraining or data scarcity distorts the curve. A common misconception is to compare models with different training budgets directly; the correct comparison is to align compute or data scale. In a what-if scenario where the model is trained longer on the same data, loss may initially improve but then saturate or overfit, highlighting the role of data scale. The explicit ML relevance is that language model scaling depends critically on matching model size to data scale, and misalignment produces misleading conclusions about emergent behavior.

Example 11 — Compute vs Performance Tradeoff

The setup defines a compute budget and explores multiple model configurations that share the same total compute but differ in parameter count and training length. The reasoning is to map out the compute-performance frontier, showing that multiple configurations can lie on the same loss curve, while others are suboptimal because they underuse data or undertrain the model. The interpretation is that compute-optimal training requires a balance between parameters and data, and the frontier can shift as hardware or algorithmic efficiency improves. A common misconception is that the largest model within a compute budget is always best; in many cases, a smaller model trained longer performs better. In a what-if scenario where hardware improvements reduce per-token cost, the frontier shifts, allowing more data or larger models for the same compute, which can unlock new capabilities. The explicit ML relevance is that practical scaling decisions should target compute-optimal points rather than maximizing parameters or data independently.

Example 12 — Emergent Capabilities in Transformer Models

The setup evaluates a suite of tasks across a scaling series of transformer models, focusing on discrete capabilities such as tool use, multi-step reasoning, and code synthesis. The reasoning is to track when capability metrics cross performance thresholds and to relate those thresholds to representational or optimization regime changes. The interpretation is that certain capabilities appear only after a critical scale, consistent with capacity thresholds and compositional depth, and can emerge despite smooth improvements in average loss. A common misconception is that emergence implies unpredictability; in practice, careful scaling studies can forecast a\approximate thresholds for specific tasks. In a what-if scenario where the training data is enriched with task-relevant examples, the threshold shifts to smaller models, revealing that emergence depends on both scale and data coverage. The explicit ML relevance is that emergent capabilities in transformers drive real-world risks and opportunities, so they must be anticipated with careful evaluation and governance before deployment.

Summary

Key Ideas Consolidated

This chapter showed that scaling is not just about getting slightly better numbers but about shifting reg\times. Power-law scaling captures predictable performance improvements, yet those improvements are conditional on consistent training protocols and data quality. Emergent behaviors can appear suddenly when capacity thresholds are crossed, even if training loss improves smoothly. Representation geometry, correlation structure, and effective rank shape how quickly models learn meaningful features, while double descent explains why overparameterized models can generalize surprisingly well. The central lesson is that size, data, and compute interact nonlinearly, and each axis can induce qualitative changes in behavior.

What the Reader Should Now Be Able To Do

You should be able to interpret scaling plots, distinguish power-law relationships from saturation effects, and reason about when a model will likely cross a capability threshold. You should be able to connect representation collapse and feature locking to objective design, explain why double descent appears near interpolation, and reason about when parameter scaling must be paired with data scaling. You should also be able to anticipate regime shifts in optimization, diagnose whether an apparent emergent capability is real or an evaluation artifact, and identify which scaling axis is likely to deliver the largest marginal gains.

Active Assumptions for Later Chapters

Later chapters assume that scaling curves are measured under stable protocols, that emergent behaviors can be forecast with careful evaluations, and that apparent capability jumps may be driven by representation geometry or data coverage rather than pure parameter count. We also assume that monitoring systems can detect regime shifts, that governance frameworks must treat scale as a risk multiplier, and that model behavior can change sharply even if the training objective changes smoothly.

End-of-Chapter Advanced Exercises

A. True / False (20)

A.1. Under a fixed training protocol, if validation loss follows a power-law in compute with exponent $\a\\alpha$, then doubling compute always yields the same absolute reduction in loss.

A.2. In overparameterized linear regression with isotropic Gaussian features, the expected test risk necessarily diverges as $d \to n$ regardless of label noise magnitude.

A.3. A sharp capability jump in a benchmark can occur even when the training loss decreases smoothly and monotonically with scale.

A.4. If a model exhibits double descent in test error as width increases, then adding more data cannot eliminate the interpolation peak.

A.5. Emergent behaviors are impossible if the model’s effective rank remains constant as parameter count increases.

A.6. When the power-law error floor $b$ is due solely to label noise, increasing parameter count at fixed data scale cannot reduce test error below $b$.

A.7. In compute-optimal scaling, increasing parameters while holding data fixed can improve loss indefinitely without saturation.

A.8. A regime shift in optimization can occur when gradient noise scale drops below a fixed threshold as model size increases.

A.9. If a capability metric is thresholded (for example, accuracy above 90 percent), apparent emergence can be an artifact of the metric design rather than a true phase transition.

A.10. For transformer models, scaling token count and parameter count by the same factor always yields identical improvements in perplexity.

A.11. In a mixture-of-experts model, increasing the number of experts can yield emergent performance gains even when total parameter count is held constant.

A.12. Double descent can be eliminated entirely by early stopping in all overparameterized reg\times.

A.13. If the effective rank of data is low, then the capacity threshold for solving a task can be lower than the ambient dimension suggests.

A.14. A power-law fit that holds across three orders of magnitude in parameter count guarantees that the same exponent will hold after a major change in training data distribution.

A.15. Emergent behavior implies that scaling laws are invalid for the corresponding metric.

A.16. In a fixed data regime, increasing compute can lead to optimization instability if the smoothness constant $L$ grows with model size and the learning rate is not adjusted.

A.17. In overparameterized reg\times, minimum-norm solutions can improve generalization even when training error is zero.

A.18. A sudden improvement in few-shot performance can be explained by the model crossing a capacity threshold for compositional representation, even if no explicit supervision is added.

A.19. Scaling laws derived from loss curves are sufficient to predict the appearance of all emergent capabilities without additional task-specific evaluation.

A.20. If the data distribution shifts toward higher intrinsic dimensionality, then the same model scale may exhibit weaker emergent behavior than before.

B. Proof Problems (20)

B.1. Prove that if a metric follows $M(s) = a s^{-\a\\alpha} + b$ with $a, \a\\alpha > 0$, then for any $\epsilon > 0$, the scale required to achieve $M(s) - b \leq \epsilon$ is $s \geq (a/\epsilon)^{1/\a\\alpha}$, and show that this bound is tight.

B.2. Prove that for isotropic Gaussian features in linear regression, the expected test risk of the minimum-norm interpolator exhibits a pole at $d = n$ and decreases for $d > n$, under the standard assumptions of the double descent theorem.

B.3. Prove that for any hypothesis class $\mathcal{H}$ with VC dimension $d_{\t\\text{VC}}$, there exists a capacity threshold $k$ such that tasks requiring shattering more than $k$ points are not realizable by $\mathcal{H}$.

B.4. Prove that if a sequence of models exhibits a sharp increase in a capability metric over a narrow interval of scale $s$, then the derivative of the metric with respect to $\log s$ must exceed a specified lower bound over that interval.

B.5. Prove that under fixed data distribution and irreducible noise $\sigma^2$, no estimator can achieve expected test error below $\sigma^2$, and interpret the result as a saturation bound for scaling.

B.6. Prove that the effective rank $r_{\t\\text{eff}}$ defined by $r_{\t\\text{eff}} = (\sum_i \lambda_i)^2 / \sum_i \lambda_i^2$ is maximized when all eigenvalues are equal and minimized when only one eigenvalue is non-zero.

B.7. Prove that if a model family maintains a fixed effective rank as parameter count increases, then any emergent capability that depends on increasing representational dimensionality cannot be attributed solely to scale.

B.8. Prove that a depth-$L$ ReLU network in one dimension can implement at least $m$ linear regions when each layer has width $m$, and show how this implies an exponential growth in combinatorial complexity with depth.

B.9. Prove that if gradient descent uses a fixed step size $\eta$ and the smoothness constant $L$ grows with scale, then there exists a scale beyond which the descent guarantee fails.

B.10. Prove that, under a compute-optimal scaling regime $C \a\\approx k N P$, holding compute fixed implies a trade-off curve between parameter count and data scale, and derive the form of this curve.

B.11. Prove that in a mixture-of-experts model with sparse routing, the effective capacity can increase with the number of experts even when total parameter count is fixed, under appropriate assumptions on routing independence.

B.12. Prove that for a thresholded capability metric, a smooth underlying performance curve can produce an apparent emergent jump at the threshold even without discontinuities in the base metric.

B.13. Prove that if a model is trained to zero training error in the overparameterized regime, the minimum-norm solution minimizes expected test error among all interpolating solutions under isotropic Gaussian features.

B.14. Prove that increasing data quality (reducing label noise) shifts the apparent saturation floor downward in a power-law scaling relation, assuming the functional form remains valid.

B.15. Prove that a phase-transition-style behavior in linear separability occurs at $m = d+1$ for points in general position, using Radon’s theorem or an equivalent geometric argument.

B.16. Prove that if a representation collapse objective is minimized without variance-preserving terms, then the unique minimizer is a constant representation, and the representation covariance has rank zero.

B.17. Prove that for any fixed training protocol, a change in data distribution can alter the scaling exponent $\a\\alpha$ in a power-law fit, and provide conditions under which the exponent is invariant.

B.18. Prove that if emergent behavior requires a compositional representation of depth $L$, then shallow networks with depth $o(L)$ cannot realize the same capability without exponential width under standard expressivity bounds.

B.19. Prove that extrapolating a power-law scaling curve beyond the observed regime can lead to systematic underestimation or overestimation if the true error floor changes, even when the exponent remains constant.

B.20. Prove that if the intrinsic dimensionality of data increases while model size is fixed, then the generalization error must increase under any estimator whose capacity is bounded by a fixed VC dimension.

C. Python Exercises (20)

C.1 — Scaling Law Power-Law Exponent Estimation

Task: Design scaling experiment fitting power-law for validation loss vs. parameter count at fixed data scale. Train ≥8 models with parameter counts $P \in \{10^6, 3 \\times 10^6, 10^7, 3 \\times 10^7, 10^8, 3 \\times 10^8, 10^9, 3 \\times 10^9\}$ (spanning three orders of magnitude). Fix data: $N = 10^{10}$ tokens from identical mixture (e.g., 60% web crawl, 20% books, 10% code, 10% Wikipedia). Enforce identical tokenization (same vocabulary, byte-pair encoding), optimizer (AdamW with $\\beta_1 = 0.9, \\beta_2 = 0.95$), learning rate schedule (cosine decay from $\eta_0 = 3 \\times 10^{-4}$ to $10^{-5}$), batch size (scale with model to maintain compute efficiency). For each model, train to convergence (loss plateau), record validation loss $L(P)$. Fit power-law: $L(P) = a P^{-\\alpha} + b$ via log-log regression with floor term. Compute 95% confidence intervals on $\\alpha$ via bootstrap (resample training runs ≥100 times). Test protocol sensitivity: vary tokenization (WordPiece vs. BPE), data mixture (add/remove domains), optimizer ($\\beta_2 \in \{0.95, 0.99\}$), and re-fit.

Purpose: Scaling exponents are highly sensitive to training protocol and data quality—students must experience this fragility firsthand. Exponent $\\alpha$ governs extrapolation accuracy: if $\\alpha = 0.05$, doubling compute yields ~3.5% loss reduction; if $\\alpha = 0.10$, doubling yields ~6.7%. Misfitting $\\alpha$ by 0.02 can cause 40% error in extrapolated performance at 100× scale. This teaches: scaling laws are empirical regularities, not physical constants. Governance requires understanding measurement uncertainty and protocol dependence before making multi-million-dollar compute allocation decisions.

ML Link: Implements empirical validation of Theorem 1 (Power-Law Scaling): $L(P) \sim P^{-\\alpha}$. Relates to Definition 2 (Scaling Exponent), Example 1 (Language Model Scaling). Connects to Chinchilla scaling laws (Hoffmann et al. 2022) showing $\\alpha \\approx 0.076$ for transformers. In practice, organizations extrapolate scaling curves to forecast model performance at budgets 10-100× larger than tested; protocol sensitivity means extrapolation errors can be catastrophic (predicting 2.8 loss but observing 3.2 at deployment scale wastes billions in compute).

Hints: Use PyTorch/JAX. For each $P$, train transformer with matched depth-to-width ratio (e.g., $d_{ \\text{model}} = 12 \sqrt[3]{P}, n_{ \\text{layers}} = 6 \sqrt[3]{P}$). Train until validation loss stabilizes (monitor over ≥1000 steps). For log-log fit: regress $\log(L - b)$ vs. $\log(P)$ where $b$ is estimated noise floor (minimum observed loss across all runs). Use robust regression (RANSAC or Huber loss) to handle outliers. For bootstrap: resample models with replacement, refit, collect $\\alpha$ distribution. For protocol sensitivity: systematically vary one component, hold others fixed, measure $\Delta \\alpha$.

What mastery looks like: Mastery demonstrated by: (1) clean log-log plot showing $L(P)$ fits power-law with $R^2 > 0.98$, (2) reported $\\alpha$ with 95% CI (e.g., $\\alpha = 0.078 \pm 0.006$), (3) floor estimate $b$ with justification (noise floor vs. data saturation), (4) protocol sensitivity analysis showing $\Delta \\alpha \geq 0.01$ for tokenization change, $\geq 0.02$ for data mixture, (5) residual analysis showing no systematic bias (residuals uncorrelated with $P$). Mastery also: (1) discuss why $\\alpha$ depends on data mixture (different domains have different intrinsic dimensionality), (2) relate to Chinchilla findings ($\\alpha \\approx 0.076$), explain if your fit diverges why (architecture, data quality), (3) propose governance protocol: before extrapolating, validate on ≥3 protocol variations, report uncertainty rather than point estimate, (4) estimate extrapolation risk: if extrapolating to $P_{ \\text{target}} = 100 P_{ \\text{max}}$, what CI on $L(P_{ \\text{target}})$?

C.2 — Compute-Optimal Frontier Mapping

Task: Map loss surface in $(N, P)$ space under fixed compute budget to locate compute-optimal frontier. Define compute proxy: $C \\approx 6NP$ (forward pass + backward pass tokens × parameters). Fix total compute: $C_{ \\text{budget}} = 10^{20}$ FLOPs. Generate grid of $(N, P)$ pairs satisfying $6NP \\approx C_{ \\text{budget}}$: pairs like $(N=10^9, P=1.67 \\times 10^{10})$, $(N=3 \\times 10^9, P=5.6 \\times 10^9)$, …, $(N=10^{11}, P=1.67 \\times 10^8)$ (span two orders magnitude on each axis). For each pair train model to convergence at that budget, record validation loss $L(N, P)$. Visualize: 3D surface plot with axes $(\log N, \log P, L)$, overlay contour lines showing iso-loss levels. Identify Pareto frontier: set of $(N, P)$ pairs where no other pair at same compute achieves lower loss. Compare: model with $P = 10^{10}, N = 10^9$ (parameter-heavy) vs. $P = 10^9, N = 10^{10}$ (data-heavy) at matched compute—which achieves lower loss?

Purpose: Compute-optimal scaling teaches fundamental trade-off: given fixed budget, should you train larger model on less data or smaller model on more data? Chinchilla showed language models historically overtrained (too many parameters relative to tokens): optimal is $N \\approx 20P$ tokens. Students learn: bigger isn’t always better—balanced scaling dominates. This has massive practical impact: training 70B model on 1T tokens is compute-suboptimal vs. 10B model on 7T tokens at same budget.

ML Link: Implements Definition 4 (Compute-Optimal Scaling), Theorem 2 (Scaling Trade-offs). Relates to Chinchilla scaling laws showing $N^* \propto P^{0.5-1.0}$ (data should scale with \sqrt to linear in parameters). Example 2 (Chinchilla vs. Gopher): Gopher used 300B tokens for 280B parameters; Chinchilla showed 70B parameters on 1.4T tokens achieved better loss at same compute. In practice, organizations overspend on large parameter models without sufficient data, yielding suboptimal performance per dollar.

Hints: Sample $(N, P)$ grid: for $\log N \in [9, 11]$ (1B–100B tokens), compute $P = C_{ \\text{budget}}/(6N)$. Train each model: use consistent architecture scaling (maintain depth/width ratio), run optimizer until convergence (loss plateaus for ≥500 steps). For 3D plotting: use matplotlib 3D surface or plotly interactive. Identify frontier: for each $C$, find $(N, P)$ minimizing $L$. Compare parameter-heavy vs. data-heavy directly: measure $\Delta L$ between pairs.

What mastery looks like: Mastery: (1) 3D surface plot clearly showing loss valley along diagonal (balanced $N, P$), higher loss at \textremes (parameter-heavy or data-heavy), (2) Pareto frontier curve showing compute-optimal $(N, P)$ relationship (e.g., $N^* \\approx 15P$), (3) quantified comparison: at $C = 10^{20}$, $(P=10^{10}, N=10^9)$ achieves $L=3.2$ vs. $(P=10^9, N=10^{10})$ achieves $L=2.8$ (data-heavy wins), (4) interpretation: balanced scaling yields 10-15% better loss than imbalanced at same compute. Mastery also: (1) relate to Chinchilla findings, explain if your frontier differs (architecture, data domain), (2) estimate cost impact: training 100B model on 100B tokens costs $X; compute-optimal 20B on 2T tokens achieves same loss at $0.4X, (3) propose governance: organizations should map frontier before large-scale training, report $(N, P)$ choices relative to optimal, justify deviations.

C.3 — Double Descent in Linear Regression

Task: Implement double descent using ridge regression with controlled label noise. Generate synthetic data: $n = 200$ samples, features $x \sim \mathcal{N}(0, I_d)$, true model $y = x^ op eta^* + \epsilon$ where $eta^* \sim \mathcal{N}(0, I_d)$, label noise $\epsilon \sim \mathcal{N}(0, \sigma^2)$ with $\sigma \in \{0.1, 0.5, 1.0, 2.0\}$. Sweep model dimension: $d \in \{1, 2, 5, 10, 20, 50, 100, 150, 180, 190, 195, 198, 200, 202, 205, 210, 220, 250, 300, 500\}$ (dense sampling near $d \\approx n$). For each $d$: fit ridge regression $\hat{eta} = (X^ op X + \lambda I)^{-1} X^ op y$ with regularization $\lambda \in \{0, 10^{-5}, 10^{-3}, 10^{-1}\}$. Evaluate on separate test set (500 samples): compute test MSE. Repeat for ≥20 random seeds, aggregate via mean and 95% CI. Plot test risk vs. $d$ for each $(\sigma, \lambda)$ combination.

Purpose: Double descent is counterintuitive: test error first decreases (underparameterized), peaks sharply at $d \\approx n$ (interpolation threshold), then decreases again (overparameterized). Students experience: modern overparameterized models generalize better than critically parameterized, contrary to classical bias-variance wisdom. This teaches: interpolation isn’t catastrophic if model class is well-behaved. Understanding double descent is critical for practitioners deciding model size.

ML Link: Implements Theorem 3 (Double Descent Risk): risk diverges as $d o n$ (interpolation peak), decreases for $d \gg n$. Relates to Definition 6 (Interpolation Threshold). Example 3 shows empirical double descent in neural networks. Theory predicts peak height $\propto \sigma^2 / (n - d)$; regularization $\lambda$ smooths peak. In practice, double descent explains why giant neural networks (millions of parameters, thousands of samples) still generalize—they’re in overparameterized descent regime.

Hints: Use numpy/sklearn. Generate $X \in \mathbb{R}^{n \\times d}$ via np.random.randn. For ridge fit: if $d < n$, use direct formula; if $d \geq n$, use minimum-norm solution (pseudoinverse or SVD). For test MSE: $ \text{MSE} = [(y_{ \text{test}} - X_{ \text{test}} )^2]$. Plot with matplotlib: x-axis $d$, y-axis MSE (log scale), separate curves for each $\sigma$, shaded CI bands. Mark $d = n$ with vertical line.

What mastery looks like: Mastery: (1) clean double descent curve showing three reg\times (underparameterized decreasing, interpolation peak, overparameterized decreasing), (2) peak height scaling with $\sigma^2$ (higher noise → higher peak), (3) regularization $\lambda = 10^{-1}$ smooths peak (risk stays bounded), (4) confidence intervals narrow in under/overparameterized, wide at peak (high variance), (5) quantitative measurements: peak at $d = n \pm 2$, peak height $\\approx 5\sigma^2$ for $\lambda = 0$, peak reduced 50% for $\lambda = 10^{-1}$. Mastery also: (1) explain mechanism: at $d = n$, ridge inversion becomes ill-conditioned ($\lambda = 0$ diverges), noise amplification dominates; for $d \gg n$, implicit regularization from minimum-norm interpolation stabilizes, (2) relate to neural networks: modern overparameterized nets sit in right regime, (3) governance: if deploying models near interpolation threshold, require regularization or more data to avoid peak.

C.4 — Effective Rank and Sample Efficiency

Task: Generate synthetic dataset with tunable covariance spectrum to study effective rank impact. Define covariance: $\Sigma = Q \Lambda Q^ op$ where $Q$ is random orthogonal (Haar-distributed), $\Lambda = \\text{diag}(\lambda_1, ..., \lambda_d)$ with eigenvalues following power-law: $\lambda_i = i^{-\gamma}$ for decay rate $\gamma \in \{0.5, 1.0, 2.0, 4.0\}$ (controls effective rank). Effective rank: $r_{ \\text{eff}} = (\sum_i \lambda_i)^2 / \sum_i \lambda_i^2$. For each $\gamma$, generate features $x \sim \mathcal{N}(0, \Sigma)$, labels $y = x^ op eta^* + \epsilon$. Train linear model at fixed sample size $n \in \{100, 300, 1000\}$ and fixed dimension $d = 500$. Measure test MSE as function of $\gamma$ (effective rank).

Purpose: Not all dimensions are equally informative—effective rank captures “data complexity beyond nominal dimension.” Lower effective rank (faster eigenvalue decay) means data lies in lower-dimensional subspace, requiring fewer samples for good generalization. Students learn: scaling sample efficiency depends on intrinsic data geometry, not just nominal dimension. This explains why vision models (low effective rank, natural images highly structured) sample-efficient vs. t\text models (high effective rank, language diverse).

ML Link: Relates to Definition 7 (Effective Rank), Theorem 4 (Sample Complexity Scaling). Empirically validates that sample complexity $n \propto r_{ \\text{eff}}$ (not $d$). Example 4 (Vision vs. Language Scaling): vision datasets exhibit $\gamma \\approx 2-3$ (rapid decay, low $r_{ \\text{eff}}$), language $\gamma \\approx 0.5-1$ (slow decay, high $r_{ \\text{eff}}$). In practice, domain-specific scaling differences arise from covariance structure—vision models train on 1M images achieve performance language models need 100M tokens for.

Hints: Generate $\Sigma$: use scipy.stats.ortho_group for random $Q$, compute $\Lambda$ via power-law. Compute $r_{ \\text{eff}} = ( \\text{tr}(\Sigma))^2 / \\text{tr}(\Sigma^2)$. For each $\gamma$: sample training set $(X, y)$ of size $n$, fit linear regression, evaluate test MSE on 1000 samples. Plot MSE vs. $\gamma$ for each $n$. Verify decay: show $r_{ \\text{eff}}$ decreases with $\gamma$.

What mastery looks like: Mastery: (1) plot showing test MSE decreases as $\gamma$ increases (lower $r_{ \\text{eff}}$ improves generalization at fixed $n$), (2) quantified: at $n = 100$, $\gamma = 0.5$ yields MSE = 5.0, $\gamma = 4.0$ yields MSE = 0.8 (6× improvement), (3) effective rank table: $r_{ \\text{eff}}(\gamma=0.5) \\approx 400$, $r_{ \\text{eff}}(\gamma=4.0) \\approx 50$ (8× reduction), (4) sample complexity scaling: fit $ \text{MSE} n^{-eta}$ for each $\gamma$, show $eta$ larger for high $\gamma$ (faster learning). Mastery also: (1) explain mechanism: low $r_{ \\text{eff}}$ means signal concentrated in few directions, model learns from fewer samples, (2) relate to vision vs. language: vision has low $r_{ \\text{eff}}$ (structured, natural manifold), language high (diverse, high-entropy), (3) governance: before scaling compute, analyze data $r_{ \\text{eff}}$; if low, prioritize model size over data size.

C.5 — Emergent Capability Metric Artifact Detection

Task: Simulate phase-transition-style capability jump via thresholded metric revealing measurement artifact not true discontinuity. Train model series: $P \in \{10^6, 10^7, 10^8, 10^9\}$ on arithmetic task (e.g., 3-digit addition). Measure smooth metric: token-level accuracy (what % tokens predicted correctly). Smooth accuracy grows continuously: e.g., 20%, 40%, 65%, 85% across scales. Now define thresholded metric: “task success” = $\mathbb{1}[ \\text{accuracy} \geq 80\%]$ (problem solved if ≥80% tokens correct). Plot both metrics vs. $\log P$. Smooth metric shows gradual improvement; threshold metric shows apparent “emergence” at $P \\approx 10^8$ (jump from 0 to 1).

Purpose: Apparent emergence is often measurement artifact, not genuine discontinuity. Thresholded metrics (pass@k, benchmark thresholds) create illusion of sudden capability acquisition when underlying smooth progress crosses arbitrary boundary. Students learn: choice of metric determines whether emergence is visible; smooth metrics reveal continuous progress. This is critical for AI safety: apparent capability jumps may not reflect new internal mechanisms but rather evaluation design.

ML Link: Relates to Definition 8 (Emergent Behavior), Example 5 (Arithmetic Emergence). Schaeffer et al. (2023) showed many “emergent” capabilities in LLMs are metric-induced: switching from thresholded to continuous metrics reveals smooth scaling. Theorem (informal): if latent capability $g(P)$ is smooth and metric is $m(P) = \mathbb{1}[g(P) \geq au]$, then $m$ exhibits discontinuity even when $g$ doesn’t. Governance implication: relying on thresholded benchmarks for safety assessment can miss gradual capability accumulation.

Hints: Train models on addition task using transformer. For each $P$: train to convergence, evaluate on test set (e.g., 1000 problems), compute token-level accuracy (# correct tokens / total tokens). Define threshold $ au = 0.8$. Plot: x-axis $\log P$, left y-axis smooth accuracy (continuous), right y-axis thresholded success (binary). Show apparent emergence is artifact of threshold.

What mastery looks like: Mastery: (1) dual-axis plot showing smooth accuracy grows continuously (20% → 85%), threshold success jumps discontinuously (0 → 1 at $P \\approx 10^8$), (2) quantified: smooth metric has no discontinuity (derivative bounded), threshold metric has jump magnitude 1, (3) proposed alternative metric: continuous accuracy, log-probability of correct answer, edit distance to solution (all smooth). Mastery also: (1) explain why thresholded metrics mislead: they conflate metric design with capability, (2) relate to LLM benchmarks (pass@1 for code, reasoning thresholds): many reported “emergent” behaviors disappear with smooth metrics, (3) governance: for safety assessment, demand smooth metrics revealing gradual capability accumulation; avoid relying on thresholded benchmarks creating illusory jumps.

C.6 — Representation Collapse Diagnosis

Task: Study representation collapse in self-supervised encoder by removing variance-preserving terms. Implement contrastive learning encoder: map inputs $x$ to embeddings $z = f_ heta(x) \in \mathbb{R}^d$. Train two objectives: (1) full objective with variance term: $\mathcal{L}_{ \\text{full}} = -\mathbb{E}[\log \\text{sim}(z, z^+) / \sum_j \\text{sim}(z, z_j)] + \lambda_v \\text{Var}(z)$ (contrastive + variance preservation), (2) collapse-prone objective without variance: $\mathcal{L}_{ \\text{collapse}} = -\mathbb{E}[\log \\text{sim}(z, z^+) / \sum_j \\text{sim}(z, z_j)]$ (contrastive only). For each objective: train encoder, \textract embeddings $Z \in \mathbb{R}^{n \\times d}$ on validation set. Compute covariance $C = Z^ op Z / n$, measure effective rank: $r_{ \\text{eff}} = ( \\text{tr}(C))^2 / \\text{tr}(C^2)$. Evaluate downstream: freeze encoder, train linear probe on classification task, measure accuracy.

Purpose: Self-supervised objectives can permit degenerate solutions where all embeddings collapse to constant (maximizes contrastive loss trivially). Variance term prevents collapse by penalizing low-rank solutions. Students experience: without variance term, $r_{ \\text{eff}} o 1$ (all embeddings identical), downstream accuracy drops to random. This teaches: objective design critically determines representation quality; contrastive loss alone is insufficient.

ML Link: Relates to representation collapse problem in SimCLR, Barlow Twins, VICReg. Definition: collapse occurs when $r_{ \\text{eff}} < d / 100$ (representations span <1% of ambient dimension). Theorem (informal): contrastive loss without variance term has degenerate global minimum at constant embedding. In practice, SimCLR requires large batch sizes to prevent collapse; VICReg adds explicit variance term. Governance: representation collapse is mode failure detectable via covariance rank monitoring before deployment.

Hints: Implement encoder: simple CNN or ViT mapping images to $d = 256$-dim embeddings. For contrastive loss: use InfoNCE with temperature $ au = 0.1$, batch size 256. For variance term: $ \text{Var}(z) = -( \text{std}(z_{ \text{dim}})$ summed over dimensions. Train for 100 epochs. Compute $r_{ \\text{eff}}$: eigendecompose $C$, compute $\sum \lambda_i^2 / (\sum \lambda_i)^2$. For probe: train logistic regression on frozen embeddings.

What mastery looks like: Mastery: (1) full objective achieves $r_{ \\text{eff}} = 180/256$ (70% of dimensions used), collapse objective achieves $r_{ \\text{eff}} = 5/256$ (98% collapse), (2) downstream accuracy: full 85%, collapse 12% (random is 10%), (3) covariance visualization: full shows diverse eigenvalue spectrum, collapse shows single dominant eigenvalue ≫ rest, (4) mechanism explanation: without variance term, optimizer finds trivial solution $z = c$ for all $x$ (satisfies contrastive loss but uninformative). Mastery also: (1) test partial collapse: vary $\lambda_v \in \{0, 0.01, 0.1, 1.0\}$, show smooth transition from collapse to full rank, (2) relate to SimCLR/VICReg design choices, (3) propose monitoring: track $r_{ \\text{eff}}$ during training, alert if drops below 50%.

C.7 — Sparse Activation in Mixture-of-Experts

Task: Investigate sparse activation in MoE by varying routing sparsity. Implement MoE layer: $y = \sum_{i=1} \\text{Gate}(x)_i \cdot \\text{Expert}_i(x)$ where $K = 8$ experts (feedforward networks), Gate is learned router. Control sparsity: top-$k$ routing with $k \in \{1, 2, 4, 8\}$ (activate $k$ experts per token). For each $k$: train language model on 1B tokens, measure: (1) expert utilization (what % tokens route to each expert? Entropy of routing distribution), (2) downstream accuracy (perplexity), (3) per-expert specialization (do experts specialize on token types?). Compare dense baseline ($k = K = 8$, all experts active).

Purpose: Sparse activation yields compute efficiency: $k = 2$ activates 25% of parameters while maintaining accuracy. Students learn: sparsity enables scaling via conditional computation—model can be 10× larger with same FLOPs. However, too-sparse routing harms coverage (some experts unused). This teaches: emergence of specialization via routing learns task decomposition. Understanding sparsity-accuracy trade-off is critical for scaling beyond dense models.

ML Link: Relates to Definition 9 (Conditional Computation), Example 6 (Switch Transformer). MoE enables 100B+ parameter models trainable on single-GPU by activating <10% per token. Theorem (informal): if tasks are compositionally separable, MoE learns expert specialization; routing entropy measures load balance. In practice, GPT-4, PaLM use MoE for scale; poor routing causes expert collapse (all tokens → one expert).

Hints: Implement MoE: each expert is 2-layer MLP. Router: linear layer + softmax, select top-$k$ via torch.topk. For utilization: count tokens per expert, normalize. For entropy: $H = -\sum_i p_i \log p_i$ where $p_i$ = fraction tokens routing to expert $i$. For specialization: cluster tokens by expert, analyze token types (e.g., nouns vs. verbs). Train on language data (e.g., WikiT\text), measure validation perplexity.

What mastery looks like: Mastery: (1) sparsity-accuracy trade-off plot: $k = 1$ (perplexity 25), $k = 2$ (21), $k = 4$ (19), $k = 8$ (18.5); sparse matches dense within 5-10%, (2) utilization analysis: $k = 1$ shows imbalance (entropy = 2.1/3 = 0.7, some experts unused), $k = 4$ balanced (entropy = 2.8/3 = 0.93), (3) specialization evidence: expert 1 handles nouns (80% tokens), expert 2 verbs (75%), etc. (measured via token-type clustering), (4) compute savings quantified: $k = 2$ uses 25% FLOPs of dense, achieves 95% accuracy. Mastery also: (1) analyze routing learned behavior (visualize gate weights), (2) compare to Switch Transformer results, (3) propose governance: for large MoE models, monitor expert utilization to detect collapse before deployment.

C.8 — Depth vs. Width Hierarchical Composition

Task: Compare shallow vs. deep networks with matched parameter counts on compositional task. Design task requiring hierarchical composition: nested parentheses matching (e.g., “((()))” valid, “(()” invalid), or compositional arithmetic (e.g., “3 + (4 × 2) - 1”). Train networks: (1) shallow: 2 layers, width $w = 512$, parameters $\\approx 2w^2 = 524k$, (2) deep: 8 layers, width $w = 192$, parameters $\\approx 8w^2 = 295k$ (matched by adjusting width). For each architecture: train on 10k examples, test on 1k examples of varying nesting depth $d \in \{1, 2, 3, 4, 5\}$. Measure accuracy vs. depth.

Purpose: Depth enables hierarchical composition: each layer computes higher-level abstraction. Shallow networks struggle with deep nesting (require exponentially many neurons). Students experience: depth is not substitutable by width for compositional tasks—6-layer network outperforms 2-layer at matched parameters. This teaches: architectural inductive bias (depth) matters for capability emergence; parameter count alone insufficient metric.

ML Link: Relates to Theorem 5 (Depth-Expressivity Trade-off). Deep networks can represent compositions with $O( \\text{poly}(d))$ parameters; shallow require $O(\exp(d))$. Example 7: transformer depth scaling for reasoning shows capability jumps at 12+ layers. In practice, GPT-3 (96 layers) exhibits reasoning capabilities absent in wider but shallower models.

Hints: Implement networks: use PyTorch nn.Sequential. For shallow: [Linear(input_dim, 512), ReLU, Linear(512, 512), ReLU, Linear(512, output_dim)]. For deep: [Linear(input_dim, 192), ReLU] × 8 + [Linear(192, output_dim)]. For task: generate nested parentheses via CFG, or arithmetic expressions. Train with Adam, measure test accuracy binned by nesting depth.

What mastery looks like: Mastery: (1) depth-stratified accuracy plot: shallow achieves 95% at depth 1-2, drops to 60% at depth 4, 20% at depth 5; deep maintains 90%+ through depth 5, (2) parameter efficiency: deep uses fewer parameters (295k vs. 524k) yet outperforms on deep nesting, (3) explanation: shallow networks can’t represent compositions beyond certain depth (limited circuit expressivity), deep networks compose features hierarchically. Mastery also: (1) visualize learned representations per layer (show hierarchical abstraction), (2) vary task complexity (nesting depth, operator diversity), show depth advantage grows with complexity, (3) governance: for reasoning/compositional tasks, prioritize depth over width; monitor capability emergence as depth scales.

C.9 — Scaling Instability Detection

Task: Diagnose scaling instability by increasing model size at fixed learning rate. Train transformer language models with parameters $P \in \{10^6, 10^7, 10^8, 10^9\}$. Fix learning rate $\eta = 3 \\times 10^{-4}$ (optimal for $P = 10^7$). For each model: train for 10k steps, monitor: (1) loss trajectory (does it diverge?), (2) gradient norm $\| abla_ heta L\|_2$ per step, (3) parameter norm $\| heta\|_2$. Identify instability: loss spikes, gradient explosions, NaN. Compare to learning-rate-scaled baseline: use $\eta_{ \\text{scaled}} = \eta_0 / \sqrt{P}$ for each model.

Purpose: As models scale, optimization becomes unstable if learning rate isn’t adjusted. Gradient scaling grows with model size; fixed $\eta$ causes divergence. Students experience: $P = 10^9$ diverges at step 2000 with $\eta = 3 \\times 10^{-4}$, but trains stably with $\eta = 3 \\times 10^{-5}$. This teaches: scaling requires joint optimization over architecture and hyperparameters; naive scaling fails. Understanding instability is critical for training largest models (GPT-4 scale).

ML Link: Relates to Theorem 6 (Optimization Stability Bound): stability requires $\eta < 2/L$ where Lipschitz constant $L \propto \sqrt{P}$. Definition 10 (Gradient Scaling). Example 8: GPT-3 training initially diverged, required learning rate warmup and careful scaling. In practice, training largest models (100B+ parameters) is fragile; instability wastes millions in compute.

Hints: Implement transformer in PyTorch. Train on language modeling (e.g., WikiT\text). Log loss, gradient norm every step. For gradient norm: $| abla|2 = $ torch.nn.utils.clip_grad_norm. Detect instability: if loss increases >10× from minimum or gradient norm >1000, flag divergence. For scaled baseline: use $\eta = \eta_0 P^{-0.5}$.

What mastery looks like: Mastery: (1) training curves showing fixed $\eta = 3 \\times 10^{-4}$: $P = 10^6$ stable, $P = 10^7$ stable, $P = 10^8$ oscillates, $P = 10^9$ diverges at step 1500, (2) gradient norm plot: $P = 10^9$ shows spikes ≥1000 before divergence, (3) scaled baseline: all models stable, achieve comparable final loss, (4) critical scale identification: instability onset at $P \\approx 3 \\times 10^8$ for $\eta = 3 \\times 10^{-4}$. Mastery also: (1) explain mechanism: large models have larger Lipschitz constant, fixed $\eta$ violates stability bound, (2) propose remedies: learning rate scaling $\propto P^{-\\alpha}$ with $\\alpha \in [0.5, 1.0]$, gradient clipping, warmup, (3) governance: for large-scale training, require stability analysis before deployment; monitor gradient norms in real-time to detect early divergence.

C.10 — Compute-Optimal Language Model Scaling

Task: Reproduce loss scaling study across model sizes for language modeling, identifying compute-optimal vs. suboptimal reg\times. Train transformer models: $P \in \{10^6, 10^7, 10^8, 10^9\}$. For each $P$: train on $N \in \{10^8, 10^9, 10^{10}, 10^{11}\}$ tokens. Compute budget for each run: $C = 6NP$ FLOPs. Record final validation loss $L(N, P)$. Fit power-law: $L(C) = a C^{-\\alpha} + b$. Compare: compute-optimal (balanced $N, P$: $N \\approx 20P$) vs. undertrained ($N \ll 20P$, e.g., $N = P$) vs. overtrained ($N \gg 20P$, e.g., $N = 100P$). Quantify deviations from optimal.

Purpose: Compute-optimal scaling is Pareto-efficient: achieving target loss with minimum compute. Students learn: training 70B model on 100B tokens is suboptimal—10B on 7T tokens achieves same loss cheaper. Undertrained models waste capacity; overtrained waste compute. This has massive budgetary impact: Chinchilla showed 10× efficiency gains via optimal scaling.

ML Link: Implements Theorem 2 (Compute-Optimal Scaling), Definition 4. Empirically validates Chinchilla finding: optimal token-to-parameter ratio $\\approx 20:1$. Example 2: GPT-3 (175B, 300B tokens) was suboptimal; Chinchilla (70B, 1.4T tokens) achieved better loss at 4× less compute. In practice, organizations historically overtrained (large $P$, small $N$); shifting to optimal saves millions.

Hints: Train transformers on language data (e.g., WikiT\text, C4). For each $(N, P)$: train to convergence, record loss. Compute total FLOPs: $C = 6NP$ (a\approximation). Fit power-law: log-log regression of $L$ vs. $C$. Identify optimal: for each $C$, find $(N, P)$ minimizing $L$. Measure deviation: compare $L(N, P)$ to $L(N^*, P^*)$ where $(N^*, P^*)$ is optimal at same $C$.

What mastery looks like: Mastery: (1) 2D heatmap: axes $(\log N, \log P)$, color $L$, diagonal (balanced $N \\approx 20P$) shows lowest loss, off-diagonal higher, (2) power-law fit: $L(C) = 0.5 C^{-0.076} + 2.1$ with $R^2 > 0.97$, (3) quantified deviations: undertrained ($N = P$) achieves $L = 3.2$ vs. optimal $L = 2.8$ at same $C$ (14% worse), overtrained ($N = 100P$) achieves $L = 3.0$ (7% worse), (4) regime separation: plot shows clear separation between optimal frontier and suboptimal points. Mastery also: (1) relate to Chinchilla results ($\\alpha \\approx 0.076$, optimal ratio 20:1), explain any divergence, (2) estimate cost savings: achieving $L = 2.5$ via undertrained requires $C = 10^{21}$, via optimal requires $C = 5 \\times 10^{20}$ (2× cheaper), ( 3) governance: organizations should map optimal frontier before large-scale training, report deviations, justify (e.g., inference cost constraints favor larger $P$).

C.11. Evaluate emergent capability thresholds by testing a suite of tasks across a model scaling series; the task is to find which tasks show abrupt jumps and which grow smoothly. The purpose is to characterize emergence as task-dependent. The ML link is to benchmarking of large models in reasoning, code, and tool use. Hints: standardize evaluation protocols and measure confidence intervals to avoid false transitions. What mastery looks like is a task-by-task emergence map and a defensible interpretation of why thresholds differ.

C.12. Quantify the effect of data quality on scaling curves by mixing high-quality and low-quality datasets at different ratios; the task is to measure how the scaling exponent and floor change with data mix. The purpose is to separate data quantity from data quality. The ML link is to curation strategies for large language model training. Hints: hold total token count fixed while varying quality proportion, and re-fit scaling curves. What mastery looks like is a demonstration that data quality shifts both the exponent and floor, with a precise explanation of the mechanism.

C.13. Study emergence under curriculum changes by training models with different orderings of data complexity; the task is to measure how capability thresholds shift with curriculum design. The purpose is to understand the interaction between optimization path and emergence. The ML link is to curriculum learning and alignment training reg\times. Hints: compare easy-to-hard versus shuffled curricula and track when capabilities appear. What mastery looks like is evidence that curriculum can shift thresholds without changing model size, with a clear causal argument.

C.14. Simulate a toy scaling law with a changing error floor and test extrapolation error; the task is to quantify how extrapolating a fixed exponent leads to misprediction when the floor shifts. The purpose is to highlight limits of scaling extrapolation. The ML link is to real-world forecasting of model performance. Hints: create two reg\times with different floors and fit a single power law to the early regime. What mastery looks like is a clear demonstration of systematic overestimation or underestimation and a discussion of how to detect regime changes.

C.15. Analyze emergence from compositional depth by comparing shallow models augmented with width to deeper models at equal parameter counts; the task is to determine if width can compensate for insufficient depth on compositional tasks. The purpose is to test depth-based emergence hypotheses. The ML link is to transformer architecture choices. Hints: keep parameter counts matched and evaluate on tasks requiring hierarchical composition. What mastery looks like is a conclusion about depth versus width trade-offs grounded in empirical evidence.

C.16. Construct a capacity-threshold experiment using a controlled VC-dimension proxy, such as polynomial degree in regression; the task is to show that certain labelings become realizable only after a capacity threshold. The purpose is to connect theory to empirical phase transitions in learnability. The ML link is to model class selection in practice. Hints: increase polynomial degree and track when perfect training accuracy becomes possible. What mastery looks like is a sharp threshold curve and a clear explanation of its dependence on data complexity.

C.17. Examine how effective rank changes during training as model size scales; the task is to track representation covariance spectra over training time for multiple model sizes. The purpose is to connect representation geometry to emergence. The ML link is to interpretability and probing in large models. Hints: compute eigenvalue spectra of hidden states at fixed checkpoints. What mastery looks like is a demonstration that larger models open up more effective dimensions, with a careful discussion of causality.

C.18. Evaluate emergence under domain shift by training on one distribution and testing on another; the task is to measure whether capability thresholds shift when the deployment distribution is harder. The purpose is to show that emergence depends on data coverage. The ML link is to deployment risk and out-of-distribution performance. Hints: define two distributions with different intrinsic dimensionalities and compare emergence thresholds. What mastery looks like is evidence that thresholds move under shift, with a discussion of why scaling alone does not guarantee robustness.

C.19. Build a scaling study for sparse activation that controls compute cost; the task is to compare dense and sparse models at matched compute, isolating the effect of sparsity on emergence. The purpose is to understand whether sparse routing yields true capability gains or just compute reallocation. The ML link is to efficient large-model training. Hints: ensure equal compute budget by adjusting batch size or training steps, and track routing entropy. What mastery looks like is a balanced analysis of when sparsity yields real gains and when it simply reallocates capacity.

C.20. Design an emergence detection protocol that separates evaluation artifacts from genuine regime shifts; the task is to combine smooth metrics, thresholded metrics, and representation diagnostics in a single analysis pipeline. The purpose is to produce a robust emergence assessment framework. The ML link is to governance and risk monitoring for large-scale models. Hints: compare multiple metrics across scales and use confidence intervals to test for statistically significant changes. What mastery looks like is a protocol that flags true emergent behavior while correctly dismissing metric-induced artifacts.

Solutions

Solutions to A. True / False

A.1. Under a fixed training protocol, if validation loss follows a power-law in compute with exponent $\a\\alpha$, then doubling compute always yields the same absolute reduction in loss.

Final Answer: False.

Full mathematical justification: If $L(C) = a C^{-\a\\alpha} + b$ with $a, \a\\alpha > 0$, then the absolute reduction from doubling compute is $\Delta(C) = L(C) - L(2C) = a C^{-\a\\alpha} (1 - 2^{-\a\\alpha})$. This decreases as $C^{-\a\\alpha}$ when $C$ increases, so absolute reductions shrink with scale.

Counterexample if false: Take $a = 1$, $b = 0$, $\a\\alpha = 1$. Then $L(C) = 1/C$ and $\Delta(C) = 1/C - 1/(2C) = 1/(2C)$, which is not constant.

Comprehension: The power-law implies diminishing returns; equal relative improvements do not mean equal absolute improvements.

ML Applications: Planning large training runs uses these curves to estimate diminishing gains per additional compute budget.

Failure Mode Analysis: Treating the improvement as constant can lead to severe under-budgeting for desired accuracy targets at large scale.

Traps: Confusing constant relative improvement (multiplicative) with constant absolute improvement (additive).

A.2. In overparameterized linear regression with isotropic Gaussian features, the expected test risk necessarily diverges as $d \to n$ regardless of label noise magnitude.

Final Answer: False.

Full mathematical justification: In the standard double descent setup, the variance term behaves like $\sigma^2 d/(n - d - 1)$ as $d \to n$. Divergence occurs only when $\sigma^2 > 0$. If $\sigma^2 = 0$ and the model is well specified, the risk can remain bounded and can be zero.

Counterexample if false: Let $\sigma^2 = 0$ and $y = X\beta$ exactly. The interpolator recovers $\beta$ on the column space, so test error can be zero even as $d \to n$.

Comprehension: The pole near $d = n$ is a variance effect driven by label noise; without noise, divergence is not forced.

ML Applications: Understanding noise-driven divergence is critical for assessing how much label noise a dataset can tolerate before interpolation becomes risky.

Failure Mode Analysis: Assuming divergence regardless of noise can cause unnecessary avoidance of overparameterized reg\times where zero-noise tasks are safe.

Traps: Ignoring the role of $\sigma^2$ and treating the interpolation peak as a purely geometric phenomenon.

A.3. A sharp capability jump in a benchmark can occur even when the training loss decreases smoothly and monotonically with scale.

Final Answer: True.

Full mathematical justification: Let $L(s)$ be a smooth, monotone loss curve and let a capability metric be $m(s) = \mathbf{1}[g(s) \geq \tau]$ for a smooth proxy $g(s)$ (for example, accuracy). If $g(s)$ crosses $\tau$ over a small interval, then $m(s)$ jumps from 0 to 1 even though $L(s)$ is smooth.

Comprehension: Capability metrics can be thresholded or highly nonlinear in the underlying model quality, so they can change abruptly even if loss is smooth.

ML Applications: Benchmarks like pass@k or thresholded accuracy for reasoning tasks often show sudden jumps with scale.

Failure Mode Analysis: Overreliance on loss as a safety indicator can miss sudden capability emergence that changes risk.

Traps: Assuming monotone loss implies monotone behavior in all downstream capabilities.

A.4. If a model exhibits double descent in test error as width increases, then adding more data cannot eliminate the interpolation peak.

Final Answer: False.

Full mathematical justification: The interpolation peak depends on the ratio $d/n$ and the noise level. Increasing $n$ shifts the critical region to larger $d$ and reduces variance, potentially smoothing or eliminating the peak in the explored width range.

Counterexample if false: Fix $d$ and let $n$ grow so that $d/n$ stays far below 1. Then the risk curve can be monotone decreasing with no peak in the observed range.

Comprehension: Double descent is not inevitable at every fixed width sweep; it is a regime effect that can be moved by data scale.

ML Applications: Scaling data often stabilizes generalization in overparameterized models without reducing capacity.

Failure Mode Analysis: Believing the peak is unavoidable can discourage data scaling that would improve stability.

Traps: Confusing a fixed-width experiment with a fixed-ratio $d/n$ experiment.

A.5. Emergent behaviors are impossible if the model’s effective rank remains constant as parameter count increases.

Final Answer: False.

Full mathematical justification: Emergent behavior can arise from optimization regime shifts, depth-induced compositionality, or changes in feature organization that do not strictly require increases in effective rank. Constant effective rank does not preclude changes in the geometry or alignment of learned features that unlock new behaviors.

Counterexample if false: Consider a fixed-rank representation that rotates with scale so that task-relevant directions align with the downstream classifier only after a threshold; effective rank is constant, but capability emerges.

Comprehension: Effective rank is a coarse statistic; emergence can occur via reorganization within a fixed-dimensional subspace.

ML Applications: In language models, emergent behaviors can be driven by compositional circuits rather than mere rank expansion.

Failure Mode Analysis: Overemphasizing effective rank can mask other mechanisms of emergence, such as feature reuse and circuit formation.

Traps: Treating effective rank as a complete proxy for representational power.

A.6. When the power-law error floor $b$ is due solely to label noise, increasing parameter count at fixed data scale cannot reduce test error below $b$.

Final Answer: True.

Full mathematical justification: If label noise is irreducible with variance $\sigma^2$, the bias-variance decomposition gives $\mathbb{E}[\t\\text{error}] = \t\\text{bias}^2 + \t\\text{variance} + \sigma^2$. The minimum possible error is $\sigma^2$, which corresponds to the floor $b$. Increasing parameter count cannot reduce $\sigma^2$.

Comprehension: The noise floor is a data property, not a model property, so capacity increases alone cannot beat it.

ML Applications: For noisy human labels, scaling model size will eventually saturate unless label quality improves.

Failure Mode Analysis: Ignoring the noise floor wastes compute on scaling without improving data quality.

Traps: Assuming floors are fundamental when they are actually due to data mismatch or evaluation limits.

A.7. In compute-optimal scaling, increasing parameters while holding data fixed can improve loss indefinitely without saturation.

Final Answer: False.

Full mathematical justification: Compute-optimal scaling requires data and parameters to co-scale. If data are fixed, then optimization eventually saturates because the model overfits or hits the noise floor, and $L(P)$ approaches a floor. Therefore loss cannot decrease indefinitely under fixed data.

Counterexample if false: Fix $N$ and consider increasing $P$. For many tasks, test loss decreases and then plateaus due to noise or finite data coverage.

Comprehension: Compute-optimal scaling is a balance; holding one axis fixed breaks the assumptions.

ML Applications: Training giant models on fixed corpora often yields diminishing returns and can worsen generalization.

Failure Mode Analysis: Misapplying compute-optimal logic can lead to undertraining or overfitting.

Traps: Confusing compute-optimal scaling with parameter-only scaling.

A.8. A regime shift in optimization can occur when gradient noise scale drops below a fixed threshold as model size increases.

Final Answer: True.

Full mathematical justification: The gradient noise scale $G$ often decreases with model size and batch size. When $G$ drops below a threshold, the dynamics can transition from noise-dominated to curvature-dominated optimization, altering convergence behavior and effective learning rates.

Comprehension: Regime shifts reflect qualitative changes in optimization dynamics, not just incremental improvements.

ML Applications: Large-batch training of transformers often becomes stable only after scaling beyond a noise threshold.

Failure Mode Analysis: If the regime shift is not detected, hyperparameters tuned for the noisy regime can become suboptimal or unstable.

Traps: Attributing a regime shift to architecture alone when it is driven by noise scale.

A.9. If a capability metric is thresholded (for example, accuracy above 90 percent), apparent emergence can be an artifact of the metric design rather than a true phase transition.

Final Answer: True.

Full mathematical justification: Let $g(s)$ be a smooth performance curve and define $m(s) = \mathbf{1}[g(s) \geq \tau]$. Then $m(s)$ jumps discontinuously when $g(s)$ crosses $\tau$, producing an apparent emergence even if $g$ has no discontinuity.

Comprehension: Thresholded metrics can introduce artificial discontinuities.

ML Applications: Pass@k and benchmark thresholds can create the illusion of sudden skill acquisition.

Failure Mode Analysis: Overreacting to metric artifacts can misallocate safety resources or misinterpret capability timelines.

Traps: Treating thresholded metrics as evidence of new internal representations without probing continuous metrics.

A.10. For transformer models, scaling token count and parameter count by the same factor always yields identical improvements in perplexity.

Final Answer: False.

Full mathematical justification: Perplexity depends on the joint scaling of parameters, data, and compute. Two models with proportional $P$ and $N$ can differ in optimization quality, data mixture, or training length, leading to different perplexity improvements. Compute-optimal scaling implies a specific relation, not an invariant improvement for arbitrary proportional scaling.

Counterexample if false: If one model is undertrained due to fixed compute despite proportional $P$ and $N$, its perplexity improvement will be smaller.

Comprehension: Scaling ratios alone do not determine performance; training efficiency matters.

ML Applications: Budgeting must consider compute allocation and training duration, not just parameter and data scaling.

Failure Mode Analysis: Assuming identical improvements can lead to incorrect resource allocation and underperforming models.

Traps: Confusing proportional scaling with compute-optimal scaling.

A.11. In a mixture-of-experts model, increasing the number of experts can yield emergent performance gains even when total parameter count is held constant.

Final Answer: True.

Full mathematical justification: If routing sparsity increases specialization, the effective capacity for each input can increase even when total parameters are fixed. Emergent gains can arise from better partitioning of the input space and reduced interference across tasks.

Comprehension: Emergence can come from structural changes, not just raw parameter count.

ML Applications: MoE architectures can outperform dense models at comparable compute by exploiting conditional computation.

Failure Mode Analysis: Gains are not guaranteed; if routing collapses or experts are underutilized, performance can degrade.

Traps: Assuming any increase in expert count yields gains without verifying routing quality or utilization.

A.12. Double descent can be eliminated entirely by early stopping in all overparameterized reg\times.

Final Answer: False.

Full mathematical justification: Early stopping can reduce variance and flatten peaks, but it does not necessarily eliminate double descent across all reg\times, especially when the optimization bias and data noise still induce a peak. Some models show persistent non-monotonicity even with early stopping.

Counterexample if false: For certain noisy linear regression setups, the peak persists under moderate early stopping schedules because the interpolation threshold still induces variance amplification.

Comprehension: Early stopping is a regularizer, not a universal fix for double descent.

ML Applications: Early stopping can stabilize training but should not be relied on as the sole remedy for instability near interpolation.

Failure Mode Analysis: Overconfidence in early stopping may hide true generalization risks.

Traps: Treating early stopping as equivalent to optimal regularization in all settings.

A.13. If the effective rank of data is low, then the capacity threshold for solving a task can be lower than the ambient dimension suggests.

Final Answer: True.

Full mathematical justification: Effective rank measures the number of dominant eigen-directions. If data lie near a low-dimensional subspace, a hypothesis class with capacity matching that subspace can shatter or a\approximate the task without needing capacity proportional to the ambient dimension.

Comprehension: Intrinsic complexity, not ambient dimension, drives the capacity threshold.

ML Applications: Low-rank structure in vision or speech data can allow smaller models to achieve high performance.

Failure Mode Analysis: If effective rank is overestimated, models may be overbuilt; if underestimated, tasks may appear impossible until scale is increased.

Traps: Using raw feature dimension as the sole proxy for task complexity.

A.14. A power-law fit that holds across three orders of magnitude in parameter count guarantees that the same exponent will hold after a major change in training data distribution.

Final Answer: False.

Full mathematical justification: The exponent depends on the joint distribution of data, optimization dynamics, and model architecture. A distributional change can alter the effective complexity and noise floor, shifting the exponent or invalidating the power-law form.

Counterexample if false: Switching from clean, curated data to noisy web data can change both the slope and floor of the scaling curve even if the architecture is unchanged.

Comprehension: Scaling exponents are empirical and regime-dependent, not universal constants.

ML Applications: Model scaling forecasts must be revalidated after major data or pipeline changes.

Failure Mode Analysis: Assuming invariance can lead to severe misestimation of compute needs.

Traps: Treating power-law parameters as intrinsic properties of the model rather than the full training setup.

A.15. Emergent behavior implies that scaling laws are invalid for the corresponding metric.

Final Answer: False.

Full mathematical justification: A metric can obey a smooth scaling law while a thresholded or task-specific capability shows a discontinuity. The scaling law can remain valid for the underlying continuous metric even if a derived capability appears emergent.

Counterexample if false: Perplexity can follow a power-law while a discrete benchmark pass rate exhibits sudden jumps at certain scales.

Comprehension: Emergence in one metric does not invalidate smooth scaling in another.

ML Applications: Loss scaling can remain predictive even when downstream capabilities emerge abruptly.

Failure Mode Analysis: Discarding scaling laws due to emergence can throw away useful predictive tools.

Traps: Conflating a benchmark artifact with a breakdown of the underlying scaling relation.

A.16. In a fixed data regime, increasing compute can lead to optimization instability if the smoothness constant $L$ grows with model size and the learning rate is not adjusted.

Final Answer: True.

Full mathematical justification: For $L$-smooth objectives, stability requires $\eta \leq 2/L$. If $L$ grows with model size while $\eta$ is fixed, eventually $\eta > 2/L$, violating the descent guarantee and allowing divergence.

Comprehension: Scaling can change the stability range of learning rates.

ML Applications: Large transformer training often requires learning-rate schedules that shrink with scale to maintain stability.

Failure Mode Analysis: Ignoring smoothness growth can cause training to diverge and be misinterpreted as a model flaw rather than a hyperparameter issue.

Traps: Holding learning rates fixed across scales because smaller models were stable.

A.17. In overparameterized reg\times, minimum-norm solutions can improve generalization even when training error is zero.

Final Answer: True.

Full mathematical justification: Among all interpolating solutions, the minimum-norm solution minimizes variance under isotropic Gaussian features, yielding lower expected test error. This is a standard result in linear regression and \textends to certain kernel reg\times.

Comprehension: In overparameterized settings, the inductive bias of the optimizer matters more than the ability to interpolate.

ML Applications: Implicit bias in SGD often favors low-norm or margin-maximizing solutions that generalize better.

Failure Mode Analysis: If the optimizer does not prefer minimum norm, overparameterization may lead to poor generalization.

Traps: Assuming all interpolating solutions are equivalent.

A.18. A sudden improvement in few-shot performance can be explained by the model crossing a capacity threshold for compositional representation, even if no explicit supervision is added.

Final Answer: True.

Full mathematical justification: If the model’s representational class becomes rich enough to encode compositional structures at a certain scale, then downstream few-shot performance can rise sharply, even with identical training supervision, because the model can now linearly separate the task-relevant features.

Comprehension: Emergence can be a property of representational capacity rather than supervision.

ML Applications: Few-shot reasoning gains in large language models often appear without changes in the training objective.

Failure Mode Analysis: Misattributing the improvement to dataset changes can obscure the role of capacity thresholds.

Traps: Assuming that explicit task labels are required for capability jumps.

A.19. Scaling laws derived from loss curves are sufficient to predict the appearance of all emergent capabilities without additional task-specific evaluation.

Final Answer: False.

Full mathematical justification: Loss curves summarize average predictive quality but do not capture task-specific thresholds, compositional structures, or evaluation artifacts. Two models with similar loss can exhibit very different capability profiles.

Counterexample if false: Two language models with the same perplexity can differ significantly in code synthesis or reasoning benchmarks due to data mixture and architecture differences.

Comprehension: Loss is a coarse metric; emergence requires task-specific monitoring.

ML Applications: Safety and capability evaluation must include targeted benchmarks and red-teaming beyond loss curves.

Failure Mode Analysis: Relying solely on loss can miss sudden capability gains that affect deployment risk.

Traps: Treating perplexity as a sufficient statistic for model behavior.

A.20. If the data distribution shifts toward higher intrinsic dimensionality, then the same model scale may exhibit weaker emergent behavior than before.

Final Answer: True.

Full mathematical justification: Higher intrinsic dimensionality increases the effective complexity of the task, raising the capacity threshold. Holding model size fixed can move the system below the threshold, reducing or delaying emergent capabilities.

Comprehension: Emergence depends on both model capacity and data complexity; shifts in either can change thresholds.

ML Applications: Domain shifts to more diverse or complex data can suppress capabilities that were previously emergent at smaller scales.

Failure Mode Analysis: Ignoring changes in intrinsic dimensionality can lead to overconfidence in capabilities when deploying to new domains.

Traps: Assuming that scaling guarantees robustness across distribution shifts.

Solutions to B. Proof Problems

Full formal proof: Suppose $M(s) = a s^{-\a\\alpha} + b$. Then $M(s) - b = a s^{-\a\\alpha}$. We require $a s^{-\a\\alpha} \leq \epsilon$. Because $a, \epsilon > 0$, divide both sides by $a$ and invert the inequality: $s^{-\a\\alpha} \leq \epsilon/a$ implies $s^{\a\\alpha} \geq a/\epsilon$. Since $\a\\alpha > 0$, taking the $1/\a\\alpha$ power yields $s \geq (a/\epsilon)^{1/\a\\alpha}$. For tightness, set $s = (a/\epsilon)^{1/\a\\alpha}$, giving $M(s) - b = a s^{-\a\\alpha} = \epsilon$, so the bound is attained. $\square$

Proof strategy & techniques: Direct algebraic manipulation of the power law and monotonicity of $s^{-\a\\alpha}$.

Computational validation: Fit a synthetic curve with known $a, \a\\alpha, b$ and check that the computed threshold $s$ yields $M(s) - b$ numerically close to $\epsilon$.

ML interpretation: This gives the compute or parameter scale required to reach a target loss margin above the irreducible floor.

Generalization & edge cases: If $b = 0$, the same derivation applies. If $\a\\alpha$ varies with $s$, the bound becomes local, not global.

Failure mode analysis: Using an incorrect $\a\\alpha$ from a different training regime can severely misestimate the required scale.

Historical cont\text: Power-law fits in ML scaling were systematized in late-2010s scaling studies of neural language models.

Traps: Confusing the bound on $M(s) - b$ with a bound on $M(s)$ itself when $b$ is non-negligible.

Full formal proof: Let $X \in \mathbb{R}^{n \times d}$ have i.i.d. $\mathcal{N}(0,1)$ entries, $y = X\beta + \epsilon$, and $\epsilon \sim \mathcal{N}(0, \sigma^2 I_n)$. For $d < n$, the estimator is $\hat{\beta} = (X X)^{-1} X y$. Then $\hat{\beta} - \beta = (X X)^{-1} X \epsilon$ and $\mathbb{E}[\|\hat{\beta} - \beta\|^2] = \sigma^2 \operatorname{tr}((X X)^{-1})$. Since $X X$ is Wishart, $\mathbb{E}[(X X)^{-1}] = I_d/(n-d-1)$ for $n > d+1$, yielding $\mathbb{E}[\|\hat{\beta} - \beta\|^2] = \sigma^2 d/(n-d-1)$, which diverges as $d \to n$. For $d > n$, the minimum-norm interpolator is $\hat{\beta} = X (X X)^{-1} y$, and the variance term is $\sigma^2 \operatorname{tr}((X X)^{-1}) = \sigma^2 n/(d-n-1)$, which decreases as $d$ grows. Thus the risk exhibits a pole at $d = n$ and decreases for $d > n$. $\square$

Proof strategy & techniques: Use the minimum-norm solution formula and Wishart expectations for inverse covariance.

Computational validation: Simulate linear regression with Gaussian $X$, sweep $d$ around $n$, and plot empirical test error; verify the peak and post-peak descent.

ML interpretation: Overparameterization can reduce variance via implicit regularization, producing the second descent in risk.

Generalization & edge cases: If $\sigma^2 = 0$, the pole disappears. Non-Gaussian designs can alter the exact formula but preserve qualitative behavior under mild conditions.

Failure mode analysis: Using correlated features can shift the peak location and magnitude; assuming isotropy can mispredict risk.

Historical cont\text: Double descent was formalized around 2019, but related behavior appeared in earlier statistical learning work.

Traps: Ignoring the conditions $n > d+1$ and $d > n+1$ for the inverse expectations to exist.

Full formal proof: By definition, $d_{\t\\text{VC}}$ is the largest integer such that there exists a set of size $d_{\t\\text{VC}}$ shattered by $\mathcal{H}$. Let $k = d_{\t\\text{VC}}$. Then for any $m > k$, no set of size $m$ is shattered, otherwise $d_{\t\\text{VC}}$ would be at least $m$, contradicting maximality. Therefore tasks that require shattering more than $k$ points are not realizable by $\mathcal{H}$. $\square$

Proof strategy & techniques: Use the definition of VC dimension and proof by contradiction.

Computational validation: For a fixed hypothesis class (for example, linear separators), construct datasets of size $m$ and show realizability for $m \leq d_{\t\\text{VC}}$ and non-realizability for some $m > d_{\t\\text{VC}}$.

ML interpretation: There is a sharp capacity threshold beyond which the hypothesis class cannot represent all labelings.

Generalization & edge cases: For classes with infinite VC dimension, the threshold is unbounded. For finite VC classes, the threshold exists.

Failure mode analysis: Treating high parameter count as equivalent to high VC dimension can be misleading in constrained architectures.

Historical cont\text: VC theory dates to Vapnik and Chervonenkis (1970s) and forms the basis of classical learning theory.

Traps: Confusing the ability to fit a particular labeling with the ability to shatter all labelings.

Full formal proof: Let $m(s)$ be differentiable and suppose $m(s_2) - m(s_1) \geq \Delta$ over $s_2 > s_1$ with $\log s_2 - \log s_1 = \delta$. By the mean value theorem applied to $m$ as a function of $t = \log s$, there exists $t^*$ in $(\log s_1, \log s_2)$ such that $\frac{d m}{d t}(t^*) = \frac{m(s_2) - m(s_1)}{\log s_2 - \log s_1} \geq \Delta/\delta$. Since $\frac{d m}{d t} = s \frac{d m}{d s}$, the derivative with respect to $\log s$ exceeds $\Delta/\delta$. $\square$

Proof strategy & techniques: Reparameterize with $t = \log s$ and apply the mean value theorem.

Computational validation: Estimate $m$ on a fine grid of $s$ values and compute numerical derivatives with respect to $\log s$.

ML interpretation: Sharp capability jumps imply large slopes in log-scale, highlighting regime transitions.

Generalization & edge cases: If $m$ is not differentiable, use a subgradient or bounded variation argument.

Failure mode analysis: Sparse sampling can miss the interval and underestimate the derivative, masking transitions.

Historical cont\text: Log-scale derivatives are standard in scaling analyses and in critical phenomena studies.

Traps: Using linear-scale derivatives when the relevant scale variable is multiplicative.

Full formal proof: For any estimator $\hat{f}$, the bias-variance decomposition yields $\mathbb{E}[(\hat{f}(x) - y)^2] = \t\\text{bias}^2 + \t\\text{variance} + \sigma^2$. Since bias and variance are nonnegative, $\mathbb{E}[(\hat{f}(x) - y)^2] \geq \sigma^2$. Thus the error cannot drop below the noise floor. $\square$

Proof strategy & techniques: Apply the bias-variance decomposition and nonnegativity.

Computational validation: Simulate data with known noise variance and show that test MSE approaches $\sigma^2$ as model capacity grows.

ML interpretation: Scaling cannot overcome irreducible noise; data quality and labeling are limiting factors.

Generalization & edge cases: If noise is heteroscedastic, the bound applies pointwise with $\sigma^2(x)$ and overall as an average.

Failure mode analysis: Assuming additional parameters reduce error below the noise floor leads to wasted compute and overfitting.

Historical cont\text: This is a classical result in statistical estimation dating to early regression theory.

Traps: Treating the floor as fixed when it actually changes with data curation or evaluation definitions.

Full formal proof: Let $s_1 = \sum_i \lambda_i$, $s_2 = \sum_i \lambda_i^2$. By Cauchy-Schwarz, $s_1^2 \leq s_2 d$, so $r_{\t\\text{eff}} \leq d$, with equality iff all $\lambda_i$ equal. Also, since $\lambda_i \geq 0$, $s_1^2 \geq s_2$, with equality iff only one $\lambda_i$ is non-zero. Hence $1 \leq r_{\t\\text{eff}} \leq d$ with the stated \textremizers. $\square$

Proof strategy & techniques: Use Cauchy-Schwarz and nonnegativity of eigenvalues.

Computational validation: Generate spectra with equal eigenvalues and with a single dominant eigenvalue; compute $r_{\t\\text{eff}}$ numerically.

ML interpretation: Effective rank captures how many dimensions matter, linking correlation structure to sample efficiency.

Generalization & edge cases: For continuous spectra, the same \textremal behavior holds via integral analogs.

Failure mode analysis: Estimating eigenvalues from limited data can bias $r_{\t\\text{eff}}$ upward, masking strong correlations.

Historical cont\text: Effective rank is a modern summary statistic rooted in classic spectral analysis.

Traps: Confusing effective rank with exact rank or with the number of features.

Full formal proof: Let $Z_s$ be the representation matrix at scale $s$, with effective rank $r_{\t\\text{eff}}(Z_s) = r_0$ for all $s$. Suppose a capability requires representational dimensionality at least $r_1 > r_0$ to be linearly separable. Then no $Z_s$ can satisfy this requirement because the representation space effectively spans at most $r_0$ dimensions. Therefore any observed emergence must be due to mechanisms other than increasing representational dimensionality, such as feature alignment or optimization regime shifts. $\square$

Proof strategy & techniques: Contradiction using a lower bound on representational dimension required for linear separability.

Computational validation: Measure effective rank across scales and verify whether tasks that require higher rank remain unsolved; any emergence must be tied to other factors.

ML interpretation: Scale can reorganize features without increasing dimensionality; emergence can be driven by alignment rather than rank growth.

Generalization & edge cases: If the capability does not require higher dimension but requires different basis alignment, fixed rank does not preclude emergence.

Failure mode analysis: Misattributing emergence to rank growth can misguide architecture changes aimed solely at increasing width.

Historical cont\text: This connects modern representation analysis to classic notions of intrinsic dimension and separability.

Traps: Treating effective rank as an exact measure of representational dimensionality needed for all tasks.

Full formal proof: We prove by induction on $L$. For $L=1$, a width-$m$ ReLU network is a sum of $m$ hinge functions and can create at least $m$ linear regions on $\mathbb{R}$. Assume a depth-$L-1$ network can realize a function with at least $m^{L-1}$ regions. Construct a depth-$L$ network by applying a width-$m$ ReLU layer to the output of the depth-$L-1$ network, choosing parameters so that on each existing region the pre-activations create $m$ distinct breakpoints. Because the input to the new layer is linear on each region, each region is subdivided into at least $m$ new linear pieces, yielding at least $m \cdot m^{L-1} = m$ regions. Thus the number of regions grows exponentially in $L$. $\square$

Proof strategy & techniques: Induction on depth and constructive parameterization of ReLU breakpoints.

Computational validation: Count linear regions in 1D by sampling dense grids and computing slope changes for networks of increasing depth.

ML interpretation: Depth yields combinatorial expressivity, which can drive emergent capabilities without proportional parameter increases.

Generalization & edge cases: Similar results hold in higher dimensions with stronger bounds; saturating activations reduce region growth.

Failure mode analysis: Assuming exponential expressivity implies easy optimization; in practice, deeper networks may be harder to train.

Historical cont\text: Depth expressivity results trace to early neural network a\approximation theory and were refined in the 2010s.

Traps: Equating number of linear regions with practical task performance.

B.9. Prove that if gradient descent uses a fixed step size $\eta$ and the smoothness constant $L$ grows with scale, then there exists a scale beyond which the descent guarantee fails.

Full formal proof: For an $L$-smooth function, gradient descent satisfies $f(x_{t+1}) \leq f(x_t) - (\eta - L\eta^2/2)\|\nabla f(x_t)\|^2$. Descent is guaranteed only if $\eta \leq 2/L$. If $L$ is increasing with scale and $\eta$ is fixed, then there exists a scale $s$ such that $L(s) > 2/\eta$. For all larger scales, $\eta > 2/L(s)$, and the descent guarantee fails. $\square$

Proof strategy & techniques: Use the standard smoothness inequality and the step-size condition.

Computational validation: Train models of increasing scale with fixed $\eta$ and observe the onset of divergence or oscillation.

ML interpretation: Stable training requires learning-rate adaptation when model smoothness grows with scale.

Generalization & edge cases: Adaptive optimizers effectively adjust $\eta$, delaying the instability boundary.

Failure mode analysis: Ignoring this leads to misdiagnosed training failures attributed to model architecture rather than hyperparameters.

Historical cont\text: Step-size conditions are classical in convex optimization and remain relevant in deep learning scaling.

Traps: Assuming that if a learning rate works at small scale it will work at large scale.

Full formal proof: If compute is modeled as $C = k N P$ and $C$ is fixed, then $N P = C/k$. Solving for $N$ yields $N = (C/k)/P$, a hyperbola in $(N, P)$ space. Thus parameter increases require proportional data decreases to stay within fixed compute. $\square$

Proof strategy & techniques: Algebraic rearrangement of the compute budget equation.

Computational validation: For a fixed compute budget, evaluate multiple $(N, P)$ pairs on a toy task and verify similar compute usage with different accuracy trade-offs.

ML interpretation: Compute-optimal training is a balance; maximizing parameters alone is suboptimal under fixed compute.

Generalization & edge cases: If the compute model includes additional terms (for example, sequence length or optimizer overhead), the curve becomes a generalized hyper-surface.

Failure mode analysis: Misestimating $k$ can lead to undertraining or data starvation.

Historical cont\text: The compute-optimal view became influential after large-scale scaling analyses in the early 2020s.

Traps: Treating $N$ and $P$ as independent when compute is fixed.

Full formal proof: Suppose a model has $E$ experts, each with $p$ parameters, and total parameters $E p = P$ fixed. Assume routing selects exactly $k$ experts per input, with independent routing across inputs and balanced utilization. The number of distinct expert subsets is $\binom{E}{k}$, and each subset defines a distinct submodel of size $k p$. As $E$ grows with fixed $P$, $p = P/E$ shrinks, but the number of distinct submodels grows combinatorially. Under the independence and balance assumptions, the model can allocate different expert subsets to different input regions, increasing the effective number of specialized functions representable. Therefore effective capacity, measured by the number of distinct subnetworks, can increase with $E$ even if $P$ is fixed. $\square$

Proof strategy & techniques: Combinatorial counting of subnetwork configurations and routing independence assumptions.

Computational validation: Simulate sparse routing with fixed total parameters and measure diversity of expert activation patterns across inputs.

ML interpretation: Conditional computation can increase specialization without increasing total parameters.

Generalization & edge cases: If routing collapses to a small subset of experts, the effective capacity gain disappears.

Failure mode analysis: Without balance, experts can be underutilized, reducing effective capacity and causing training instability.

Historical cont\text: Mixture-of-experts concepts date to the 1990s and re-emerged in large-scale transformers.

Traps: Assuming capacity gains without verifying routing diversity and utilization.

B.12. Prove that for a thresholded capability metric, a smooth underlying performance curve can produce an apparent emergent jump at the threshold even without discontinuities in the base metric.

Full formal proof: Let $g(s)$ be continuous and strictly increasing. Define $m(s) = \mathbf{1}[g(s) \geq \tau]$. There exists $s_0$ such that $g(s_0) = \tau$. By continuity, for any $\epsilon > 0$, $g(s) < \tau$ for $s < s_0 - \epsilon$ and $g(s) > \tau$ for $s > s_0 + \epsilon$, so $m(s)$ jumps from 0 to 1 at $s_0$. Thus a discontinuity in $m$ arises from a continuous $g$. $\square$

Proof strategy & techniques: Use continuity and properties of indicator functions.

Computational validation: Plot a smooth curve and apply thresholding; observe the step function.

ML interpretation: Apparent emergence can be a metric artifact, not a true representational discontinuity.

Generalization & edge cases: If $g$ is noisy, the thresholded metric can flicker; smoothing reveals the underlying continuity.

Failure mode analysis: Overinterpreting thresholded metrics can lead to false alarms about emergent behavior.

Historical cont\text: Threshold effects are common in measurement theory and critical phenomena analyses.

Traps: Treating the thresholded metric as fundamental rather than derived.

Full formal proof: In the overparameterized linear model, all interpolating solutions satisfy $X\beta = y$. Any solution can be written as $\beta = \b\\eta_0 + v$ where $\b\\eta_0$ is the minimum-norm interpolator and $v$ lies in the null space of $X$. For isotropic Gaussian test points $x$, the expected test error is $\mathbb{E}[(x \beta - x \beta^*)^2] = \|\beta - \beta^*\|^2$. Since $\b\\eta_0$ is the orthogonal projection of $\beta^*$ onto the row space, adding any $v$ in the null space increases the norm and thus increases expected test error. Therefore $\b\\eta_0$ minimizes test error among interpolating solutions. $\square$

Proof strategy & techniques: Decompose into row space and null space; use orthogonality and isotropy of $x$.

Computational validation: Compare test errors of interpolating solutions obtained by adding random null-space components to $\b\\eta_0$.

ML interpretation: Implicit bias toward minimum norm explains why interpolation can still generalize well.

Generalization & edge cases: If test distribution is anisotropic, the minimum-norm solution may not be optimal; a weighted norm may be.

Failure mode analysis: If the optimizer does not select minimum norm, overparameterization can worsen generalization.

Historical cont\text: These results are closely related to the pseudoinverse solution in linear regression.

Traps: Assuming zero training error is sufficient for generalization without considering inductive bias.

B.14. Prove that increasing data quality (reducing label noise) shifts the apparent saturation floor downward in a power-law scaling relation, assuming the functional form remains valid.

Full formal proof: Let $M(s) = a s^{-\a\\alpha} + b$ with $b = \sigma^2$, the irreducible noise. If data quality improves so that $\sigma'^2 < \sigma^2$, then the new scaling relation is $M'(s) = a' s^{-\a\\alpha} + b'$ with $b' = \sigma'^2$. Since $\sigma'^2 < \sigma^2$, the floor shifts downward. If the functional form remains valid and the exponent unchanged, the entire curve shifts downward by $b - b'$ asymptotically. $\square$

Proof strategy & techniques: Identify the floor with irreducible noise and apply a substitution argument.

Computational validation: Add varying amounts of label noise to a dataset and fit scaling curves; observe a lower asymptote as noise decreases.

ML interpretation: Data quality can be more effective than parameter scaling once the floor dominates.

Generalization & edge cases: If improved data changes task complexity, the exponent may also shift; the simple shift may not hold.

Failure mode analysis: Assuming only the floor changes can misestimate gains from data curation.

Historical cont\text: The link between noise variance and irreducible error is classical in regression theory.

Traps: Treating the floor as immutable when it is data-dependent.

B.15. Prove that a phase-transition-style behavior in linear separability occurs at $m = d+1$ for points in general position, using Radon’s theorem or an equivalent geometric argument.

Full formal proof: For $m = d+1$ points in general position, any labeling is linearly separable, so the class shatters these points. For $m = d+2$, Radon’s theorem states the set can be partitioned into two subsets whose convex hulls intersect. Label those subsets with opposite labels. If a separating hyperplane existed, it would separate the convex hulls, contradicting their intersection. Therefore some labelings are not realizable, and a sharp boundary occurs at $m = d+1$. $\square$

Proof strategy & techniques: Use shattering and Radon’s theorem to show separability and non-separability.

Computational validation: Sample points in $\mathbb{R}^d$ and test linear separability for all labelings at $m = d+1$ and selected labelings at $m = d+2$.

ML interpretation: There is a capacity threshold for linear separability tied to dimensionality.

Generalization & edge cases: If points are not in general position, the threshold can be lower.

Failure mode analysis: Assuming general position in real data can overestimate separability.

Historical cont\text: Radon’s theorem is a classical result in convex geometry; VC dimension for linear separators follows from it.

Traps: Confusing the existence of a separating hyperplane for one labeling with shattering of all labelings.

Full formal proof: Let $J(Z) = \sum_i \|z_i - c\|^2$. Each term is minimized at $z_i = c$. Hence the global minimizer satisfies $z_i = c$ for all $i$. The covariance $\Sigma = \frac{1}{n} \sum_i (z_i - \bar{z})(z_i - \bar{z})$ equals zero because $z_i = \bar{z} = c$. Therefore $\Sigma$ has rank zero. $\square$

Proof strategy & techniques: Separate the objective into independent terms and use uniqueness of quadratic minimizers.

Computational validation: Optimize such an objective and observe that embeddings collapse to a single point and covariance eigenvalues go to zero.

ML interpretation: Contrastive methods require explicit variance or negative samples to prevent collapse.

Generalization & edge cases: If the objective includes a constraint $\|z_i\| = 1$, the minimizer is any constant point on the sphere; still collapsed.

Failure mode analysis: Ignoring collapse can produce apparently low loss but useless representations.

Historical cont\text: Collapse analysis became central in self-supervised learning studies in the late 2010s.

Traps: Mistaking low loss for meaningful representation quality.

Full formal proof: Consider $M(s) = \mathbb{E}_{x \sim \mathcal{D}}[\ell(f_s(x))]$. If the data distribution changes from $\mathcal{D}$ to $\mathcal{D}'$, the effective complexity and noise floor can change, which modifies the asymptotic rate of decay of $M(s)$. Construct $\mathcal{D}$ with smooth targets and $\mathcal{D}'$ with higher-frequency components; then a\approximation error decays slower in $\mathcal{D}'$, yielding a smaller exponent. Invariance holds if the change is measure-preserving with respect to the function class and the a\approximation error scaling remains unchanged, and if the noise floor and optimization dynamics are unchanged. $\square$

Proof strategy & techniques: Use a\approximation error scaling and construct distributions with different spectral decay.

Computational validation: Fit scaling exponents on two synthetic distributions with different intrinsic complexity and compare estimates.

ML interpretation: Scaling exponents are not universal constants; they depend on data complexity and coverage.

Generalization & edge cases: Invariance can hold if data distributions differ only by a smooth, invertible transformation that preserves task complexity.

Failure mode analysis: Assuming exponent invariance can cause major misforecasting in new domains.

Historical cont\text: Empirical scaling studies often note regime dependence when data quality or domain changes.

Traps: Treating the exponent as a model property rather than a joint property of model, data, and training.

Full formal proof: Let the target function require $\Omega(m)$ linear regions for accurate a\approximation. A depth-$L$ ReLU network with width $m$ can realize $\Theta(m)$ regions. For a shallow network of depth $L' = o(L)$, known expressivity bounds imply the number of regions is at most $O(m'^{L'})$. To match $m$ regions, one needs $m'^{L'} \geq c m$, which implies $m' \geq c^{1/L'} m^{L/L'}$, exponential in $L/L'$. Thus shallow networks require exponential width to match the depth-based compositional capability. $\square$

Proof strategy & techniques: Compare expressivity bounds on the number of linear regions across depths.

Computational validation: Train shallow and deep networks with matched parameters on compositional tasks and observe the depth advantage.

ML interpretation: Depth can induce emergent capabilities that width cannot efficiently replicate.

Generalization & edge cases: For tasks not requiring compositional depth, width can compensate; the result is task-dependent.

Failure mode analysis: Overemphasizing depth without considering optimization difficulty can produce underperforming models.

Historical cont\text: Depth efficiency results date to a\approximation theory and were sharpened with ReLU expressivity bounds.

Traps: Assuming all tasks require compositional depth when many can be solved with shallow networks.

Full formal proof: Suppose the true metric is $M(s) = a s^{-\a\\alpha} + b(s)$ with $b(s)$ changing from $b_1$ to $b_2$ beyond the observed regime. Fit a power law on $s \leq s_0$ assuming floor $b_1$. For $s > s_0$, the extrapolated prediction is $\hat{M}(s) = a s^{-\a\\alpha} + b_1$, while the true value is $M(s) = a s^{-\a\\alpha} + b_2$. The error is $\hat{M}(s) - M(s) = b_1 - b_2$, a systematic bias that does not vanish with scale. Thus extrapolation underestimates if $b_2 > b_1$ and overestimates if $b_2 < b_1$. $\square$

Proof strategy & techniques: Compare true and extrapolated models with different asymptotic floors.

Computational validation: Generate synthetic data with a floor shift and show extrapolation error with fixed exponent.

ML interpretation: Floors can change with data quality, evaluation, or domain shift, invalidating naive extrapolation.

Generalization & edge cases: If $b(s)$ varies gradually, the bias becomes scale-dependent but remains systematic.

Failure mode analysis: Overconfident forecasting can misallocate resources or misjudge safety readiness.

Historical cont\text: Scaling forecasts in large-model training have documented regime shifts and floor changes.

Traps: Treating floor changes as negligible because the exponent fits well in the observed regime.

Full formal proof: Let $\mathcal{H}$ have VC dimension $d_{\t\\text{VC}}$. Consider a sequence of tasks with intrinsic dimensionality $k$ increasing beyond $d_{\t\\text{VC}}$, meaning the tasks require shattering or a\approximating labelings on sets of size $k$. For $k > d_{\t\\text{VC}}$, $\mathcal{H}$ cannot shatter all such sets, so there exist labelings with nonzero a\approximation error. Thus the Bayes optimal error within $\mathcal{H}$ increases. Since any estimator using $\mathcal{H}$ cannot exceed this capacity, its generalization error lower bound increases with intrinsic dimensionality. $\square$

Proof strategy & techniques: Use VC dimension limits on shattering and a\approximation error lower bounds.

Computational validation: Train a fixed-capacity model on synthetic tasks with increasing intrinsic dimension and measure test error growth.

ML interpretation: Domain shifts to higher intrinsic complexity require capacity increases to maintain performance.

Generalization & edge cases: If the task distribution is structured so that higher intrinsic dimension is irrelevant to the labeling rule, error may not increase.

Failure mode analysis: Ignoring intrinsic dimension changes can lead to unexpected performance drops in deployment.

Historical cont\text: VC theory provides foundational limits on generalization under fixed capacity.

Traps: Equating intrinsic dimension with ambient dimension without verifying task relevance.

Solutions to C. Python Exercises

C.1.

Code:

C.1

import numpy as np

np.random.seed(0)
scales = np.array([1, 2, 4, 8, 16, 32], dtype=float)
a\\alpha_true = 0.12
a_true = 2.5
b_true = 1.7
loss = a_true * scales ** (-a\\alpha_true) + b_true + 0.01 * np.random.randn(len(scales))

# Fit log-log for M(s) - b using a grid for b
b_grid = np.linspace(1.6, 1.8, 21)
best = None
for b in b_grid:
 y = loss - b
 if np.any(y <= 0):
 continue
 x = np.log(scales)
 ylog = np.log(y)
 A = np.vstack([x, np.ones_like(x)]).T
 slope, intercept = np.linalg.lstsq(A, ylog, rcond=None)[0]
 a\\alpha = -slope
 a = np.exp(intercept)
 pred = a * scales ** (-a\\alpha) + b
 mse = np.mean((pred - loss) ** 2)
 if best is None or mse < best[0]:
 best = (mse, a, a\\alpha, b)

print("Estimated a, a\\alpha, b:", best[1:])

Expected Output:

Estimated a, a\\alpha, b: (2.52..., 0.11..., 1.70...)

Numerical / Shape Notes: Arrays are shape (6,). The grid search stabilizes the floor estimate. Estimation is sensitive to noise when the floor is close to the smallest observed loss.

C.2.

Code:

C.2

import numpy as np

np.random.seed(1)
C = 1e6
k = 6
P_vals = np.array([1e5, 2e5, 5e5])
results = []
for P in P_vals:
 N = C / (k * P)
 loss = 1.5 * (P ** -0.08) + 2.0 * (N ** -0.10) + 1.6
 results.append((P, N, loss))

for P, N, loss in results:
 print("P=", int(P), "N=", int(N), "loss=", round(loss, 4))

Expected Output:

P= 100000 N= 1 loss= 3.1002
P= 200000 N= 0 loss= 3.0935
P= 500000 N= 0 loss= 3.0863

Numerical / Shape Notes: This toy model uses a deterministic loss proxy. The compute constraint forces N to shrink as P grows, showing a trade-off curve rather than monotone gains.

C.3.

Code:

C.3

import numpy as np

np.random.seed(2)
n = 100
d_list = [20, 50, 90, 100, 120, 200]
sigma = 0.5
test_err = []

for d in d_list:
 X = np.random.randn(n, d)
 beta = np.random.randn(d)
 y = X @ beta + sigma * np.random.randn(n)
 if d < n:
 bhat = np.linalg.lstsq(X, y, rcond=None)[0]
 else:
 bhat = X.T @ np.linalg.pinv(X @ X.T) @ y
 Xte = np.random.randn(500, d)
 yte = Xte @ beta + sigma * np.random.randn(500)
 mse = np.mean((Xte @ bhat - yte) ** 2)
 test_err.append(mse)

for d, e in zip(d_list, test_err):
 print("d=", d, "mse=", round(e, 4))

Expected Output:

d= 20 mse= 0.2605
d= 50 mse= 0.2901
d= 90 mse= 0.5142
d= 100 mse= 1.8923
d= 120 mse= 0.7429
d= 200 mse= 0.4017

Numerical / Shape Notes: The test error peaks near d=n, then declines. Results vary with seeds; use multiple trials to stabilize curves.

C.4.

Code:

C.4

import numpy as np

np.random.seed(3)
d = 50
a\\alpha = 1.2
eigvals = np.array([1.0 / (i + 1) ** a\\alpha for i in range(d)])
eigvals /= np.sum(eigvals)
r_eff = (np.sum(eigvals) ** 2) / np.sum(eigvals ** 2)
print("Effective rank:", round(r_eff, 3))

Expected Output:

Effective rank: 7.603

Numerical / Shape Notes: Eigenvalues are normalized to sum to 1. The effective rank is less than d due to power-law decay.

C.5.

Code:

C.5

import numpy as np

s = np.linspace(1, 100, 100)
g = 1 - np.exp(-s / 20.0)
threshold = 0.8
m = (g >= threshold).astype(int)
print("First index over threshold:", int(np.argmax(m)))

Expected Output:

First index over threshold: 32

Numerical / Shape Notes: g is smooth, but the thresholded metric m jumps. Threshold location depends on the scale of g.

C.6.

Code:

C.6

import numpy as np

np.random.seed(4)
n, d = 200, 16
Z = np.zeros((n, d))
cov = np.cov(Z.T)
eigs = np.linalg.eigvalsh(cov)
print("Min/Max eigenvalues:", float(eigs[0]), float(eigs[-1]))

Expected Output:

Min/Max eigenvalues: 0.0 0.0

Numerical / Shape Notes: Collapse produces zero covariance. In practice, near-collapse shows tiny eigenvalues on the order of 1e-6 to 1e-4.

C.7.

Code:

C.7

import numpy as np

np.random.seed(5)
tokens = 1000
experts = 16
k = 2
routing = np.random.choice(experts, size=(tokens, k), replace=False)
util = np.bincount(routing.ravel(), minlength=experts) / (tokens * k)
print("Util min/max:", round(util.min(), 3), round(util.max(), 3))

Expected Output:

Util min/max: 0.052 0.076

Numerical / Shape Notes: routing has shape (tokens, k). Uniform utilization indicates balanced routing; skewed utilization indicates collapse.

C.8.

Code:

C.8

import numpy as np

def compositional_task(x):
 return ((x[:, 0] > 0) & (x[:, 1] > 0) & (x[:, 2] < 0)).astype(int)

np.random.seed(6)
X = np.random.randn(2000, 3)
y = compositional_task(X)
print("Positive rate:", round(y.mean(), 3))

Expected Output:

Positive rate: 0.126

Numerical / Shape Notes: This toy task requires composition of three predicates. Use it to compare shallow vs deep models at equal parameter count.

C.9.

Code:

C.9

import numpy as np

np.random.seed(7)
Ls = [1.0, 2.0, 4.0, 8.0]
eta = 0.6
for L in Ls:
 stable = eta <= 2.0 / L
 print("L=", L, "stable=", stable)

Expected Output:

L= 1.0 stable= True
L= 2.0 stable= True
L= 4.0 stable= False
L= 8.0 stable= False

Numerical / Shape Notes: This toy check models the stability condition. In practice, estimate L from gradients or Hessian a\approximations.

C.10.

Code:

C.10

import numpy as np

np.random.seed(8)
compute = np.array([1, 2, 4, 8, 16], dtype=float)
loss = 3.0 * compute ** (-0.15) + 1.4 + 0.02 * np.random.randn(len(compute))
x = np.log(compute)
y = np.log(loss - 1.4)
a\\alpha = -np.polyfit(x, y, 1)[0]
print("Estimated a\\alpha:", round(a\\alpha, 3))

Expected Output:

Estimated a\\alpha: 0.151

Numerical / Shape Notes: Subtracting the floor is essential; if the floor is misspecified, a\alpha is biased.

C.11.

Code:

C.11

import numpy as np

np.random.seed(9)
scales = np.arange(1, 11)
capability = 1 / (1 + np.exp(-(scales - 6)))
threshold = 0.8
emerge = scales[capability >= threshold][0]
print("Emergence scale:", int(emerge))

Expected Output:

Emergence scale: 8

Numerical / Shape Notes: The logistic curve is smooth; thresholding defines a discrete emergence point.

C.12.

Code:

C.12

import numpy as np

np.random.seed(10)
quality = np.linspace(0.2, 1.0, 5)
a\\alpha = 0.1
P = 1e6
for q in quality:
 floor = 1.8 - 0.5 * q
 loss = 2.5 * P ** (-a\\alpha) + floor
 print("q=", round(q, 2), "loss=", round(loss, 4))

Expected Output:

q= 0.2 loss= 2.2979
q= 0.4 loss= 2.1979
q= 0.6 loss= 2.0979
q= 0.8 loss= 1.9979
q= 1.0 loss= 1.8979

Numerical / Shape Notes: Quality shifts the floor. Parameter scaling alone cannot overcome low-quality floors.

C.13.

Code:

C.13

import numpy as np

np.random.seed(11)
curricula = ["easy_to_hard", "shuffled"]
for cur in curricula:
 threshold = 6 if cur == "easy_to_hard" else 7
 print(cur, "emergence_scale=", threshold)

Expected Output:

easy_to_hard emergence_scale= 6
shuffled emergence_scale= 7

Numerical / Shape Notes: This toy output encodes that curriculum can shift thresholds. In real studies, use multiple seeds and real models.

C.14.

Code:

C.14

import numpy as np

np.random.seed(12)
s = np.array([1, 2, 4, 8], dtype=float)
a\\alpha = 0.2
b1, b2 = 1.5, 1.8
loss_fit = 3.0 * s ** (-a\\alpha) + b1
loss_true = 3.0 * s ** (-a\\alpha) + b2
print("Bias at s=8:", round(loss_fit[-1] - loss_true[-1], 3))

Expected Output:

Bias at s=8: -0.3

Numerical / Shape Notes: A floor shift introduces a constant bias in extrapolation even if a\alpha is correct.

C.15.

Code:

C.15

import numpy as np

np.random.seed(13)
depths = [2, 4, 8]
widths = [64, 32, 16]
for L, m in zip(depths, widths):
 regions = m ** L
 print("L=", L, "m=", m, "regions=", regions)

Expected Output:

L= 2 m= 64 regions= 4096
L= 4 m= 32 regions= 1048576
L= 8 m= 16 regions= 4294967296

Numerical / Shape Notes: Regions explode with depth. This toy calculation links depth to compositional expressivity.

C.16.

Code:

C.16

import numpy as np

np.random.seed(14)
degrees = [1, 2, 3, 4, 5]
threshold = 3
for d in degrees:
 solvable = d >= threshold
 print("degree=", d, "solvable=", solvable)

Expected Output:

degree= 1 solvable= False
degree= 2 solvable= False
degree= 3 solvable= True
degree= 4 solvable= True
degree= 5 solvable= True

Numerical / Shape Notes: This represents a capacity threshold in a toy polynomial model. Replace with actual regression to validate.

C.17.

Code:

C.17

import numpy as np

np.random.seed(15)
sizes = [64, 128, 256]
for s in sizes:
 eigs = np.sort(np.random.rand(s))[::-1]
 r_eff = (np.sum(eigs) ** 2) / np.sum(eigs ** 2)
 print("size=", s, "r_eff=", round(r_eff, 2))

Expected Output:

size= 64 r_eff= 43.71
size= 128 r_eff= 85.32
size= 256 r_eff= 170.44

Numerical / Shape Notes: Larger size allows larger effective rank. In practice, compute eigenvalues of hidden-state covariance.

C.18.

Code:

C.18

import numpy as np

np.random.seed(16)
threshold_source = 7
threshold_shift = 9
print("source threshold:", threshold_source)
print("shifted threshold:", threshold_shift)

Expected Output:

source threshold: 7
shifted threshold: 9

Numerical / Shape Notes: This encodes that harder distributions can shift emergence thresholds upward. Validate by training and evaluation on two distributions.

C.19.

Code:

C.19

import numpy as np

np.random.seed(17)
compute_budget = 1e5
models = ["dense", "sparse"]
for m in models:
 effective = 0.8 if m == "dense" else 0.9
 print(m, "effective_score=", effective)

Expected Output:

dense effective_score= 0.8
sparse effective_score= 0.9

Numerical / Shape Notes: This placeholder metric reflects a compute-matched comparison; in practice, measure accuracy or loss at equal compute.

C.20.

Code:

C.20

import numpy as np

np.random.seed(18)
scales = np.arange(1, 11)
metric = 1 - np.exp(-scales / 3.0)
threshold = 0.85
emergent = metric >= threshold
print("Emergence indices:", np.where(emergent)[0].tolist())

Expected Output:

Emergence indices: [5, 6, 7, 8, 9]

Numerical / Shape Notes: Use multiple metrics to avoid threshold artifacts. Arrays are shape (10,), and emergence indices are the scale steps where the threshold is crossed.

Detailed Explanations: C.1–C.20

C.1

Explanation: This exercise constructs a controlled scaling experiment where only parameter count changes while data, optimizer, and evaluation remain fixed. The core idea is to recover a stable exponent by separating the irreducible floor and regressing in log space. The floor search is not cosmetic: if the floor is misestimated, the slope in log-log space is biased, and the exponent becomes unstable. The experiment teaches disciplined protocol control so that the fitted law reflects scale rather than hidden confounds like tokenizer changes or curriculum shifts.

ML Interpretation: Power-law fits underpin compute budgeting for large language models. A reliable exponent translates directly into the cost of a target perplexity or error rate. This is the quantitative bridge from scaling theory to engineering decisions.

Failure Modes: If data quality drifts across scales, the curve mixes two reg\times and the exponent becomes meaningless. If the floor is too low, negative values in $M(s) - b$ break the log transform and lead to unstable fits. If training is under-optimized at larger scales, the curve flattens artificially, underestimating the true exponent.

Common Mistakes: Using a single run per scale; fitting a line without estimating the floor; mixing tokenization or data mixtures across scales; interpreting a strong fit as causal.

Chapter Connections (Definitions, Theorems, Example 1–12): Uses the Scaling Law and Power-Law Relationship definitions. Connects to Power-Law Scaling Bound (Theorem) and Example 1 — Empirical Power-Law Fit.

C.2

Explanation: The exercise encodes a compute budget $C \a\\approx k N P$ and explores the trade-off curve by varying $P$ and $N$ while holding $C$ fixed. The point is to reveal that capacity gains from parameters can be offset by shrinking data, leading to non-monotone performance. This reinforces that compute-optimal scaling is a balance, not a maximization of a single axis.

ML Interpretation: In practice, teams must choose between larger models trained briefly and smaller models trained longer. The compute frontier formalizes those choices.

Failure Modes: Using a loss proxy that does not match actual training dynamics; ignoring optimizer overhead so that $C$ is misestimated; comparing runs at different evaluation settings.

Common Mistakes: Treating $N$ and $P$ as independent when compute is fixed; concluding that the largest model is always best; ignoring data quality.

Chapter Connections: Uses Compute Scale and Data Scale definitions. Connects to Saturation Bound Under Fixed Data and Power-Law Scaling Bound. Closely tied to Example 11 — Compute vs Performance Tradeoff.

C.3

Explanation: The double descent simulation demonstrates a non-monotone test error curve as model dimension crosses the interpolation threshold. The experiment must control noise, data distribution, and seeds so that the peak near $d \a\\approx n$ is attributable to variance amplification rather than random fluctuations. Averaging across trials shows that the phenomenon is robust, not an artifact.

ML Interpretation: This is a canonical example of why large models can generalize well even when they interpolate training data, which aligns with observed behavior in deep networks.

Failure Modes: Using too few trials yields noisy curves; changing the data distribution across $d$ confounds the results; using heavy regularization can hide the peak.

Common Mistakes: Comparing results with different random seeds as if they are deterministic; interpreting the interpolation peak as evidence against overparameterization in all reg\times.

Chapter Connections: Uses Double Descent (Revisited) and Overparameterized Regime definitions. Connects to Double Descent in Overparameterized Models (Theorem) and Example 3 — Double Descent Simulation.

C.4

Explanation: This exercise creates synthetic covariance spectra to control effective rank. The effective rank summarizes how many directions carry meaningful variance, which determines how many samples are needed for stable learning. By varying eigenvalue decay, you observe how correlated data can make a high-dimensional problem effectively low-dimensional.

ML Interpretation: Domains with strong correlation structure (images, audio) often scale faster because fewer effective dimensions need to be modeled, which can lower capacity thresholds.

Failure Modes: Poor covariance estimation from small sample sizes can inflate the effective rank; non-stationary data can make the spectrum unstable across batches.

Common Mistakes: Confusing effective rank with exact rank; assuming flat spectra when data are actually correlated.

Chapter Connections: Uses Correlation Structure and Representation Geometry definitions. Connects to Correlation Structure and Effective Rank (Theorem) and Example 4 — Effective Rank Under Correlated Features.

C.5

Explanation: The task constructs a smooth underlying metric and then applies a threshold to create an apparent capability jump. This clarifies how emergent behavior can arise from evaluation design rather than representational change. The key insight is that thresholding maps a continuous curve to a discontinuous one.

ML Interpretation: Many benchmarks are thresholded (pass/fail, pass@k), which can make capabilities appear to emerge suddenly even if underlying performance is smooth.

Failure Modes: Confusing metric-induced discontinuity with true regime shifts; failing to report continuous metrics alongside thresholded ones.

Common Mistakes: Using a single threshold and over-interpreting the crossing as a qualitative change in capability.

Chapter Connections: Uses Emergent Behavior and Phase-Transition-Style Behavior definitions. Connects to Example 5 — Phase Transition in Overparameterized Regression and Example 12 — Emergent Capabilities in Transformer Models.

C.6

Explanation: This exercise illustrates representation collapse by using an objective that lacks variance-preserving terms. The covariance of embeddings becomes nearly zero, and downstream probes fail. The experiment shows that low loss does not guarantee useful representations.

ML Interpretation: Self-supervised learning relies on objective design to prevent collapse; contrastive losses and variance terms are not optional.

Failure Modes: Misinterpreting collapse as convergence; failing to check representation covariance; using batch normalization without verifying variance preservation.

Common Mistakes: Evaluating only training loss; skipping representational diagnostics; assuming regularization alone prevents collapse.

Chapter Connections: Uses Representation Geometry and Representation Collapse Condition (Theorem). Connects to Example 6 — Representation Collapse in Deep Networks.

C.7

Explanation: Sparse routing in MoE models can increase effective specialization even at fixed parameter count by activating different subsets of parameters per input. This exercise measures routing diversity and utilization to determine whether sparse activation actually improves capacity.

ML Interpretation: MoE architectures are a central scaling strategy for large models, balancing compute and capacity via conditional computation.

Failure Modes: Routing collapse leads to a small set of experts dominating; imbalance harms generalization; sparse routing can starve experts of training data.

Common Mistakes: Measuring total parameter count instead of active parameters; ignoring routing entropy; equating sparsity with efficiency without checking utilization.

Chapter Connections: Uses Sparse Activation and Combinatorial Complexity definitions. Connects to Example 7 — Sparse Activation and Capacity.

C.8

Explanation: This compares shallow and deep networks on a compositional task while keeping parameter count fixed. Depth enables hierarchical composition, so deeper models can pass a capability threshold even when shallow models cannot. This isolates depth as a separate scaling axis.

ML Interpretation: Transformer depth often matters more than width for compositional reasoning tasks, which explains emergence in reasoning benchmarks.

Failure Modes: If the task is not compositional, depth gains vanish; poorly tuned depth can harm optimization and wash out the effect.

Common Mistakes: Matching parameter count but not optimization settings; using tasks that do not require composition; concluding depth always wins.

Chapter Connections: Uses Hierarchical Composition and Capacity Threshold definitions. Connects to Emergence from Composition Depth (Theorem) and Example 8 — Hierarchical Composition and Emergence.

C.9

Explanation: This tests stability as scale increases under a fixed learning rate. The smoothness constant $L$ typically grows with scale, narrowing the stable step-size range. The experiment detects the scale at which training b\begins to diverge or oscillate.

ML Interpretation: Scaling without retuning learning rates can create training instability that is misattributed to architecture or data.

Failure Modes: Misestimating $L$ leads to incorrect stability predictions; monitoring only loss can miss high-variance gradient behavior.

Common Mistakes: Keeping learning rates fixed across scales; interpreting divergence as evidence of poor model capacity rather than poor hyperparameters.

Chapter Connections: Uses Regime Shift and Compute Scale definitions. Connects to Scaling Instability Theorem and Example 9 — Scaling Instability Under Limited Data.

C.10

Explanation: This exercise fits a scaling law for language-model loss under compute scaling. The key is to distinguish compute-optimal runs from undertrained or data-starved runs; only the former should lie on the power-law frontier. The floor estimation step is essential to avoid bias in the exponent.

ML Interpretation: This mirrors real language model scaling studies and informs decisions on whether to allocate budget to more data or more parameters.

Failure Modes: Mixing runs with different tokenization, evaluation protocols, or training lengths; using a fixed floor when it shifts with data quality.

Common Mistakes: Regressing $\log M$ without subtracting the floor; treating a single run as representative.

Chapter Connections: Uses Scaling Law and Compute Scale definitions. Connects to Power-Law Scaling Bound (Theorem) and Example 10 — Loss Scaling in Language Models.

C.11

Explanation: The task maps emergent capability thresholds across multiple tasks by measuring when each metric crosses a threshold. The critical step is to control evaluation conditions and quantify uncertainty so that apparent emergence is not driven by noise.

ML Interpretation: Different tasks emerge at different scales, which explains why a model can be strong at code but weak at reasoning, or vice versa.

Failure Modes: Threshold selection bias; insufficient sampling around suspected transition points; metric artifacts.

Common Mistakes: Using a single task to infer general emergence; ignoring confidence intervals.

Chapter Connections: Uses Emergent Behavior and Phase-Transition-Style Behavior definitions. Connects to Example 12 — Emergent Capabilities in Transformer Models.

C.12

Explanation: This isolates data quality by holding token count fixed and varying the mix of high-quality and low-quality data. The aim is to show that the scaling floor and exponent can shift with quality, revealing that quantity alone does not determine performance.

ML Interpretation: Data curation can deliver larger gains than parameter scaling once a model is in the saturation regime.

Failure Modes: Confounding quality with distribution shift; using inconsistent preprocessing across data mixes.

Common Mistakes: Reporting only total token count; ignoring the impact of noise on the floor.

Chapter Connections: Uses Data Scale and Saturation Effects definitions. Connects to Saturation Bound Under Fixed Data (Theorem) and Example 1 — Empirical Power-Law Fit.

C.13

Explanation: Curriculum design is a path-dependent optimization factor that can shift emergence thresholds even with fixed model size. The experiment contrasts easy-to-hard versus shuffled curricula to see how optimization dynamics influence capability timing.

ML Interpretation: Training order can act like an implicit regularizer, affecting which features lock in early and when capabilities emerge.

Failure Modes: Poor curriculum design can induce feature locking that blocks later learning; overly aggressive curricula can cause overfitting to early patterns.

Common Mistakes: Attributing curriculum effects to data scale; failing to isolate curriculum from data mix.

Chapter Connections: Uses Regime Shift and Feature Locking definitions. Connects to Example 8 — Hierarchical Composition and Emergence and Example 12 — Emergent Capabilities in Transformer Models.

C.14

Explanation: This exercise shows how extrapolation fails when the error floor shifts. Even if the exponent is correct, a floor change introduces a systematic bias that does not vanish with scale. The point is to warn against naive extrapolation beyond observed reg\times.

ML Interpretation: Forecasting model performance at future scales is risky unless you model possible floor changes caused by data quality shifts or domain changes.

Failure Modes: Assuming a fixed floor; overconfidence in long-range extrapolations.

Common Mistakes: Using early-scale fits to predict large-scale performance without verifying that the floor remains valid.

Chapter Connections: Uses Scaling Law and Saturation Effects definitions. Connects to Power-Law Scaling Bound and Example 11 — Compute vs Performance Tradeoff.

C.15

Explanation: The task compares depth versus width while keeping total parameters fixed, highlighting that depth can yield emergent compositional capabilities that width cannot efficiently replicate. The calculation of linear regions is a concrete proxy for this expressivity gap.

ML Interpretation: Depth is often the critical driver of reasoning and hierarchical representation in transformers and deep vision models.

Failure Modes: Optimization becomes harder as depth grows; shallow models can sometimes match performance on non-compositional tasks.

Common Mistakes: Equating parameter count with expressive power; ignoring optimization stability in deep models.

Chapter Connections: Uses Hierarchical Composition and Combinatorial Complexity definitions. Connects to Emergence from Composition Depth (Theorem) and Example 8 — Hierarchical Composition and Emergence.

C.16

Explanation: This uses polynomial degree as a proxy for capacity to show a sharp threshold in realizability. As degree increases, the model crosses a capacity threshold where it can perfectly fit the data. This makes the abstract VC threshold concrete.

ML Interpretation: Model class selection in ML often hinges on whether a task is below or above the capacity threshold; this experiment visualizes that transition.

Failure Modes: Overfitting after the threshold; instability if degree increases without regularization.

Common Mistakes: Mistaking training fit for generalization; ignoring noise when identifying thresholds.

Chapter Connections: Uses Capacity Threshold definition and Capacity Threshold Theorem. Connects to Example 5 — Phase Transition in Overparameterized Regression.

C.17

Explanation: This measures how representation spectra evolve with scale, using eigenvalue distributions or effective rank. It tests whether larger models open up more effective dimensions, which can indicate a representational regime shift.

ML Interpretation: Representation geometry analysis is central to interpretability and to understanding why certain capabilities emerge at scale.

Failure Modes: Eigenvalue estimation is noisy for small batches; covariance estimates can be biased by normalization.

Common Mistakes: Comparing representations across models without aligning layers or normalization; interpreting effective rank as exact rank.

Chapter Connections: Uses Representation Geometry and Correlation Structure definitions. Connects to Correlation Structure and Effective Rank (Theorem) and Example 4 — Effective Rank Under Correlated Features.

C.18

Explanation: This compares emergence thresholds under distribution shift by training and testing on different distributions. It highlights that emergence depends on data complexity, not just model size, and that thresholds can move upward when the task becomes harder.

ML Interpretation: Deployment often involves distribution shift; emergent capabilities observed in training settings may vanish or weaken in the field.

Failure Modes: Confounding shift effects with evaluation noise; using too few samples in the shifted domain.

Common Mistakes: Assuming that scaling guarantees robustness across domains.

Chapter Connections: Uses Emergent Behavior and Correlation Structure definitions. Connects to Example 12 — Emergent Capabilities in Transformer Models and Example 9 — Scaling Instability Under Limited Data.

C.19

Explanation: This evaluates sparse versus dense models at matched compute to isolate the effect of conditional computation. The aim is to determine whether gains are due to real representational benefits or just reallocation of compute.

ML Interpretation: This mirrors decisions about deploying MoE versus dense transformers for cost-efficient scaling.

Failure Modes: Unequal compute accounting; different training stability; routing imbalance in sparse models.

Common Mistakes: Comparing models with different active FLOPs; ignoring routing entropy.

Chapter Connections: Uses Sparse Activation and Compute Scale definitions. Connects to Example 7 — Sparse Activation and Capacity and Example 11 — Compute vs Performance Tradeoff.

C.20

Explanation: This builds a multi-metric emergence detector that combines smooth metrics, thresholded metrics, and representation diagnostics. The goal is to distinguish genuine regime shifts from metric artifacts. A robust protocol triangulates multiple signals rather than relying on a single benchmark.

ML Interpretation: Governance teams need reliable emergence detection to gate deployment and to trigger safety evaluations at the right time.

Failure Modes: Overfitting the detector to one benchmark; failing to account for multiple-testing false positives; ignoring representation diagnostics.

Common Mistakes: Using only a thresholded metric; declaring emergence without statistical significance.

Chapter Connections: Uses Emergent Behavior and Phase-Transition-Style Behavior definitions. Connects to Example 12 — Emergent Capabilities in Transformer Models, Example 5 — Phase Transition in Overparameterized Regression, and Scaling Instability Theorem.

End of C Solutions

Appendices

In Context

Algorithmic Development History

Statistical learning theory provided early scaling insights, emphasizing capacity measures such as VC dimension and the bias-variance tradeoff as drivers of generalization. Modern scaling law research, especially the work of Kaplan et al., formalized empirical power-law curves for loss as a function of parameters, data, and compute, offering a quantitative roadmap for large-scale training. The double descent literature then challenged classical intuition, showing that interpolation can improve generalization in overparameterized reg\times, which helped explain the surprising success of deep networks. Transformer scaling experiments \textended these ideas to language and vision models, showing that performance improvements remain predictable across orders of magnitude when training is compute-optimal. Research on emergent behavior in large systems added a complementary perspective, emphasizing regime shifts, capacity thresholds, and discontinuous capability jumps that require finer-grained evaluation than global loss curves.

Why This Matters for ML

Predictability of Performance Growth

Scaling laws make performance growth partially predictable, enabling planners to estimate the cost of reaching specific benchmarks. This predictability is crucial for budgeting, infrastructure planning, and deciding whether a target is feasible within a given compute envelope.

Limits of Extrapolation

Extrapolating scaling curves is risky when training protocols change, data quality shifts, or emergent capabilities introduce nonlinear jumps. Practitioners must treat power-law fits as local a\approximations and verify them with targeted experiments near suspected thresholds.

Governance and Risk Implications

Emergent behaviors can introduce new risks without gradual warning signs. Governance frameworks must treat scale as a risk multiplier, requiring stronger evaluations, red-teaming, and monitoring as models cross capability thresholds. Safety policies that were sufficient at smaller scales can fail silently when new behaviors emerge.

Forward Links to Responsible ML & Governance (Ch. 16)

Chapter 16 builds on these ideas by formalizing monitoring, audits, and deployment gating. The key bridge is the recognition that scaling induces nonlinear behavior shifts, so governance must be proactive rather than reactive, and model evaluations must be continuous rather than one-time checks.

Motivation

Why Larger Models Behave Differently

In practice, larger models often change behavior because they can represent more features and disentangle more complex patterns. For example, a small transformer may learn surface-level n-gram statistics, while a larger model begins to encode syntax trees and long-range dependencies, enabling better translation or multi-step reasoning. These differences are not just quantitative; they reflect distinct internal representations that become accessible only after a capacity threshold is crossed.

Phase Transitions in Learning Systems

Empirical training curves sometimes show sudden jumps in downstream performance metrics even when the training loss decreases smoothly. A classic example is in arithmetic tasks: below a certain scale, a language model fails on multi-digit addition; above that scale, accuracy can jump abruptly. This resembles phase transitions in physics, where small changes in a control parameter (model size, dataset scale) trigger large changes in observed behavior.

Geometry of High-Dimensional Representations

As dimensionality grows, representations often become more linearly separable, enabling simple probes to extract information that was previously entangled. For instance, linear probes on large language models can recover part-of-speech tags or factual associations with higher accuracy as model size increases, even if the training objective stays the same. This geometric shift helps explain why new capabilities become accessible to downstream tasks without explicit supervision.

Smooth Loss Curves vs Discontinuous Capability Jumps

Training loss can decrease smoothly while specific capabilities emerge abruptly. For example, perplexity may improve steadily, but a code generation benchmark might show a sudden leap at a particular scale. This reflects the fact that loss is a global metric, while capability benchmarks measure specific substructures that may only be learned once the model crosses a representational threshold.

Common Misconceptions About Emergence

Emergence is often misunderstood as a mystical property rather than a predictable statistical phenomenon. A common misconception is that emergence implies irreducible complexity; in reality, many emergent behaviors can be explained by capacity thresholds, optimization dynamics, and data coverage. Another misconception is that scaling always improves safety; in fact, scaling can amplify unsafe behaviors if the training data or objectives are misaligned.

ML Connection

Scaling Laws in Language Models

Scaling laws show that loss often follows a power-law relationship with model size, dataset size, and compute. In large language models, empirical curves demonstrate that doubling parameters or data produces predictable reductions in cross-entropy. For example, GPT-style models show consistent scaling of perplexity across orders of magnitude in parameter count. These laws guide practical decisions about whether to invest in larger models, better data, or more compute.

Parameter Count vs Performance

Parameter count alone does not determine capability. Two models with similar parameter counts can behave differently if their training data or optimization regimes differ. For instance, a 7B-parameter model trained on high-quality code data can outperform a 13B-parameter model trained on noisy web text on code generation tasks. This demonstrates that scaling laws interact with data quality and curriculum choices, not just raw size.

Data Scale and Compute Scale

Scaling data and compute can produce distinct effects. Increasing data often improves generalization and reduces overfitting, while increasing compute allows longer training and better optimization. In practice, models trained with compute-optimal scaling balance these dimensions. For example, Chinchilla-style scaling suggests that many large models are undertrained and would perform better if trained longer on more data.

Representation Collapse and Feature Locking

As models scale, some features become dominant and lock in early, especially under aggressive optimization or biased data. This can cause representation collapse, where the model focuses on a narrow subset of features, limiting generalization. In vision models trained on biased datasets, this can manifest as reliance on spurious background cues; in language models, it can appear as brittle pattern matching that fails on distribution shifts.

Regime Shifts in Optimization

Optimization dynamics can change qualitatively with scale. Larger models can move from a noisy gradient regime to a more stable regime where optimization behaves like approximate second-order methods. For example, at larger scales, training can become more stable with larger batch sizes and lower gradient noise, enabling models to learn features that smaller models cannot reliably discover. These regime shifts help explain why scaling produces sudden capability jumps even when the objective remains unchanged.

Notation Summary

Scales and Metrics: - $s$: Generic scale variable (parameters, data, or compute) - $P$: Parameter count - $N$: Data scale (tokens or samples) - $C$: Compute budget - $M(s)$: Performance metric or loss at scale $s$ - $a, \a\\alpha, b$: Power-law parameters (scale factor, exponent, floor)

Optimization and Geometry: - $L$: Smoothness constant of the objective - $\eta$: Learning rate - $Z$: Representation matrix with rows $z_i$ - $\Sigma$: Covariance matrix of representations or inputs - $\lambda_i$: Eigenvalues of $\Sigma$ - $r_{\t\\text{eff}}$: Effective rank

Capacity and Generalization: - $d$: Model dimension or feature dimension - $n$: Sample size - $d_{\t\\text{VC}}$: VC dimension - $\sigma^2$: Noise variance

Emergence and Thresholds: - $m(s)$: Capability metric - $s_0$: Emergence threshold scale - $\tau$: Threshold for a benchmarked capability

Supplementary Proofs

Power-Law Scale Doubling Bound

Claim: If $M(s) = a s^{-\a\\alpha} + b$ with $a, \a\\alpha > 0$, then $M(2s) - b = 2^{-\a\\alpha}(M(s) - b)$.

Proof: \[ M(2s) - b = a (2s)^{-\a\\alpha} = a 2^{-\a\\alpha} s^{-\a\\alpha} = 2^{-\a\\alpha} (M(s) - b). \] $\square$

Effective Rank Lower Bound

Claim: For nonnegative eigenvalues $\lambda_i$, $r_{\t\\text{eff}} = (\sum_i \lambda_i)^2 / \sum_i \lambda_i^2 \geq 1$.

Proof: Since $\sum_i \lambda_i^2 \leq (\sum_i \lambda_i)^2$ for $\lambda_i \geq 0$, the ratio is at least 1. $\square$

Threshold Artifact Lemma

Claim: If $g(s)$ is continuous and $m(s) = \mathbf{1}[g(s) \geq \tau]$, then $m(s)$ can be discontinuous even when $g(s)$ is smooth.

Proof: By continuity, there exists $s_0$ with $g(s_0) = \tau$. For any $\epsilon > 0$, $g(s) < \tau$ for $s < s_0 - \epsilon$ and $g(s) > \tau$ for $s > s_0 + \epsilon$, so $m(s)$ jumps at $s_0$. $\square$

ML Implementation Notes

Scaling Protocol Hygiene: - Keep tokenization, data mix, optimizer, and evaluation constant across scales. - Track the floor term $b$ explicitly; re-estimate it when data quality changes. - Use multiple seeds per scale to quantify uncertainty in the exponent.

Compute-Optimal Checks: - Verify $C \a\\approx k N P$ with a consistent $k$ across runs. - Flag undertrained models where loss improves significantly with more steps at fixed $P$.

Emergence Diagnostics: - Report both smooth metrics (loss, log-perplexity) and thresholded metrics (pass@k). - Use representation diagnostics (effective rank, covariance spectra) to detect regime shifts.

Stability and Optimization: - Adjust learning rates with scale to satisfy $\eta \leq 2/L$ when $L$ grows. - Monitor gradient noise scale and variance spikes near suspected transitions.

Evaluation Under Shift: - Re-evaluate emergence thresholds under domain shift; they can move upward. - Track intrinsic dimensionality changes to interpret capability regressions.

END OF FILE