Chapter 03 — Linear Maps and Matrix Representations

Overview

Purpose of the Chapter

This chapter transforms the abstract notion of “linear map” (a function that preserves linear combinations) into concrete computational objects: matrices. Linear maps are the fundamental operations in machine learning—they encode how data flows through models, how features interact, how dimensions change, and how models transform raw inputs into predictions. Understanding linear maps and their matrix representations is essential because every neural network layer, every kernel method, every dimensionality reduction technique, and every classification boundary originates from linear transformations (or their compositions with nonlinearities). This chapter answers the question: How do we compute with linear maps, and what do their structural properties (rank, kernel, image) reveal about model behavior?

The journey of this chapter is from the abstract (a function \(T: V \to W\) respecting linearity) to the concrete (a matrix \([T]_\mathcal{B} \in \mathbb{R}^{m \times n}\) indexed by basis choices), and then back to interpretation (what does the matrix’s rank, null space, and column space tell us about the data and learning problem?). By the end, you will see that matrices are not arbitrary arrays of numbers—they are carefully structured encoding of geometric transformations, and their properties directly predict model expressivity, optimization behavior, and generalization.

Conceptual Scope

This chapter covers five essential topics: (1) Definition and properties of linear maps: what makes a function “linear,” how to verify linearity, and how linear maps behave under composition and scalar multiplication. (2) Matrix representations: how different basis choices yield different matrix representations of the same linear map, and how to convert between representations via similarity transformations. (3) Kernel and image: the fundamental subspaces associated with a matrix (null space and column space), which partition the domain and codomain, and bound the map’s “injectivity” and “surjectivity” (one-to-oneness and onto-ness). (4) Rank and nullity: quantifying the “loss” of dimension through the map via the rank-nullity theorem, and understanding how rank constrains the effective degrees of freedom in models. (5) Computational aspects: algorithms for computing with linear maps (Gaussian elimination, eigendecomposition, SVD), and numerical stability when matrices are singular or ill-conditioned. This chapter bridges the abstract structure (Chapter 2’s basis and dimension theory) and the computational practice (Chapter 7’s SVD and eigendecompositions, Chapters 12+ on neural networks).

Questions This Chapter Answers

• What is a linear map, and why are they ubiquitous in machine learning? A linear map is a function respecting addition and scalar multiplication, making them simple, composable, and mathematically tractable. Nearly every operation in ML (feature engineering, neural network layers, kernel methods) is linear or a composition of linear and nonlinear parts.

• How do we compute with linear maps? Via matrices: choose bases for domain and codomain, and represent the map as a matrix whose columns are images of basis vectors.

• How do basis choices affect matrix representation? Different bases yield different matrices representing the same geometric transformation. Change-of-basis matrices relate them via similarity: \(A' = P^{-1}AP\) (for square maps). Some bases (eigenvectors, singular vectors) are “special” and simplify computation.

• What does the rank of a matrix tell us about the linear map? Rank is the dimension of the image (output space reachable by the map). Full rank means the map is injective (no information loss); rank deficiency means multiple inputs map to the same output.

• What is the kernel, and why does it matter? The kernel (null space) is the set of inputs mapping to zero. Its dimension (nullity) quantifies how much “input diversity” is lost. By rank-nullity, \(\dim(\text{domain}) = \text{rank} + \text{nullity}\): the domain splits into preserved directions (rank) and annihilated directions (nullity).

• How do linear maps compose, and what happens to rank? Composing maps \(T: V \to W\) and \(S: W \to Z\) yields \(S \circ T: V \to Z\). The rank of the composition is at most the minimum of the two ranks: \(\text{rank}(S \circ T) \leq \min(\text{rank}(S), \text{rank}(T))\). A bottleneck layer (low rank) limits the overall capacity.

• What are eigenvectors and eigenvalues, and why are they special? For a square map \(T: V \to V\), an eigenvector is a direction where \(T\) acts as pure scaling: \(T(\mathbf{v}) = \lambda \mathbf{v}\). Eigenvectors are special because in that (“eigen”)basis, the map is diagonal—the simplest form. For symmetric matrices, all eigenvectors are real and orthogonal.

• When can we diagonalize a matrix, and what does it mean geometrically? A matrix \(A\) is diagonalizable iff there exists a basis of eigenvectors. Diagonalization \(A = PDP^{-1}\) (where \(D\) is diagonal, \(P\) columns are eigenvectors) is a change of basis that simplifies—powers, exponentials, and iterations are trivial in diagonal form.

How This Chapter Fits Into the Full Book

Backward links: This chapter depends critically on Chapter 2 (Basis and Dimension): we use basis changes, dimension, rank, and nullity daily. It also builds on Chapter 1 (Vector Spaces): linear maps are functions on vector spaces, preserving their structure.

Forward links: This chapter is the foundation for everything that follows. Chapter 4 (Determinants and Inverses) uses matrix structure to define determinants (a scaling factor for volumes) and discuss invertibility (when \(\text{rank} = n\) for an \(n \times n\) matrix). Chapter 5 (Eigenvalues and Eigenvectors) dives deep into eigenvectors and diagonalization, showing how eigendecomposition reveals invariant subspaces and hidden structure. Chapter 6 (Orthogonality and the Spectral Theorem) specializes to symmetric matrices and proves they are always diagonalizable via an orthonormal eigenbasis, unifying geometry and algebra. Chapter 7 (SVD and Decompositions) generalizes eigendecomposition to rectangular matrices, showing that any matrix is a composition of three simpler operations: orthogonal, diagonal, orthogonal. Chapter 8 (Least Squares and Regression) uses the rank and image of the design matrix to characterize when regression solutions exist and are unique. Chapters 12+ (Neural Networks, Deep Learning) model each layer as a linear map followed by a nonlinearity; understanding layer ranks and kernels explains representational capacity and training dynamics. Throughout Part 2 (Applications to ML), linear maps and matrices are the lingua franca—understanding their properties is the foundation for diagnosing and debugging models in practice.

Motivation

Why Linear Maps Are the Engine of ML

Machine learning is fundamentally about learning transformations from data: given inputs (raw features, images, text embeddings), learn a transformation that produces useful outputs (predictions, classifications, embeddings). The simplest transformations are linear: \(\hat{\mathbf{y}} = A\mathbf{x} + \mathbf{b}\) (a matrix applied to the input, plus a bias). This linear step is present in nearly every ML model: as the output layer of a neural network (classification head), as the kernel of a kernel method (implicit feature space), as the basis selection in dimensionality reduction (PCA projects via a linear map), and as the core of regression (fit \(\mathbf{w}\) to minimize \(\|A\mathbf{w} - \mathbf{y}\|^2\)).

Why are linear operations so prevalent? Several reasons: (1) Computational efficiency: linear algebra is highly optimized (fast matrix multiplication via BLAS/LAPACK; GPU-accelerated), making linear operations the bottleneck of model inference and training. (2) Composability: composing linear maps yields another linear map (linearity is closed under composition), enabling deep architectures. (3) Interpretability: linear maps have well-understood geometry (images, kernels, eigenvectors) that can be visualized and analyzed. (4) Theoretical tractability: many results in learning theory, optimization, and statistics assume linearity or study how nonlinearity breaks linear assumptions. (5) Practical effectiveness: even simple linear models (logistic regression, linear SVM) are highly competitive on many real-world tasks; adding nonlinearity via kernels or depth often improves performance, but linear models remain a baseline.

The relationship between inputs and outputs in a linear model is transparent: \(A\) encodes how each input dimension contributes to each output dimension. A large entry \(A_{ij}\) means “input dimension \(i\) strongly influences output dimension \(j\).” Zero rows of \(A\) mean those output dimensions are unused. Zero columns mean those input dimensions are ignored. The rank of \(A\) tells you how many “independent directions” of output can be achieved—the effective output dimension. This transparency makes linear maps ideal for building intuition, and it extends to more complex models: neural networks are compositions of linear (weight matrices) and nonlinear (activations) steps, so understanding the linear parts is essential to understanding the whole.

Transformations Between Feature Spaces

A central motif in machine learning is feature space transformation: starting with one representation of the data, apply a (usually learned) transformation to a new representation where the learning problem is simpler. Examples are everywhere:

• Word embeddings in NLP: raw tokens (high-dimensional one-hot vectors) are transformed to dense embeddings (100–300 dimensional vectors where semantics are encoded). This is a linear projection from a huge one-hot space (vocabulary size, often \(> 10^5\)) to a small dense space via a learned embedding matrix (a linear map).

• Convolutional feature maps in vision: raw pixels are convolved (a linear operation in the input, parameterized by convolutional kernels) to produce feature maps. Early layers detect edges (linear combinations of nearby pixels looking for specific patterns), intermediate layers combine these to detect textures, and late layers combine textures into objects. Each layer is a linear operation followed by a nonlinearity.

• Whitening in preprocessing: raw features have correlations and different scales. Whitening applies a linear transformation (via Cholesky or SVD) to produce zero-mean, unit-variance, uncorrelated features. The covariance matrix diagonalizes (becomes identity) under this linear change.

• Kernel methods: data in the original space may not be linearly separable. Implicitly, kernels apply a linear map to a higher-dimensional feature space (via a kernel function) where separation is possible. The effective transformation is linear in the implicit space, even though the original kernel function is nonlinear.

All of these transformations, at their core, are linear maps: they take an input vector/matrix and output a transformed version via matrix multiplication (possibly composed with nonlinearities or special structure like convolution, which is a special linear map). Understanding the rank, kernel, and image of these maps—what they preserve, what they annihilate, how much information they pass through—is critical to debugging and improving models.

Structure Preservation Under Linear Maps

A key property of linear maps is that they preserve linear structure: the image of a linear subspace is a linear subspace. Formally, if \(U \subseteq V\) is a subspace and \(T: V \to W\) is linear, then \(T(U) := \{T(\mathbf{u}) : \mathbf{u} \in U\}\) is a subspace of \(W\). This means that linear maps respect relationships in data: if inputs lie on a low-dimensional manifold, their images lie on a transformed low-dimensional manifold (possibly higher-dimensional, but still low relative to the output space dimension). This structure preservation is why dimensionality reduction via linear maps (PCA) works: data concentration in a low-dimensional subspace is preserved under the linear projection.

Conversely, linear maps can also destroy structure: if the image of \(T\) is lower-dimensional than the domain (rank-deficient), information is compressed—multiple distinct inputs may map to the same output. The kernel \(\ker(T) = \{T(\mathbf{v}) : \mathbf{v} \in V, T(\mathbf{v}) = \mathbf{0}\}\) quantifies this: all vectors in the kernel map to zero, so they are “invisible” to downstream processing. In neural networks, a low-rank hidden layer is a bottleneck—subsequent layers cannot distinguish inputs that differ in the null space of that layer’s weight matrix.

Understanding structure preservation and destruction is essential for debugging models: if a layer has rank \(r < n\) (where \(n\) is the layer width), that layer is compressing—it is a dimensionality reduction step, intentional or not. If unintentional, it may be a sign of poor initialization, dead neurons (ReLU outputs always zero), or feature collapse. If intentional (as in autoencoders), the bottleneck rank controls compression ratio. Either way, computing rank reveals the true information flow through the network, independent of the nominal layer width.

Matrices as Encodings of Transformations

A matrix is an encoding of a linear map: given bases \(\mathcal{B}\) for the domain and \(\mathcal{C}\) for the codomain, the matrix \(A = [T]_{\mathcal{B}, \mathcal{C}}\) encodes \(T\) by specifying, in columns, where basis vectors map:

\[A = [T(\mathbf{b}_1) \mid T(\mathbf{b}_2) \mid \cdots \mid T(\mathbf{b}_n)]_\mathcal{C}\]

where \([\cdot]_\mathcal{C}\) means “express in coordinates with respect to basis \(\mathcal{C}\).” This encoding is powerful because:

1. Computation: to apply \(T\) to a vector \(\mathbf{v}\) with coordinates \([\mathbf{v}]_\mathcal{B} = \mathbf{c} \in \mathbb{R}^n\), we simply multiply: \([T(\mathbf{v})]_\mathcal{C} = A\mathbf{c}\). Matrix-vector multiplication is highly optimized.

2. Composition: composing two maps \(S \circ T\) corresponds to matrix multiplication: if \(B = [T]_{\mathcal{B}, \mathcal{C}}\) and \(C = [S]_{\mathcal{C}, \mathcal{D}}\), then \([S \circ T]_{\mathcal{B}, \mathcal{D}} = CB\).

3. Inversion: if \(T\) is invertible (bijective), then \(A\) is invertible (square, non-singular), and \(T^{-1}\) corresponds to \(A^{-1}\).

4. Decomposition: any matrix can be decomposed into simpler matrices (via eigendecomposition, SVD, QR, LU), each revealing structure (eigenvectors reveal invariant subspaces, SVD reveals principal directions).

The flexibility of this encoding—different bases yield different matrices representing the same geometric transformation—is both powerful and subtle. The same linear map has infinitely many matrix representations (one for each pair of basis choices), and these matrices are related by similarity transformations: \(A' = P^{-1}AP\) (for change of domain basis from \(\mathcal{B}\) to \(\mathcal{B}'\) with change-of-basis matrix \(P\)). Some representations are specially useful: eigenbases diagonalize (reducing to scaling in each direction), orthonormal bases preserve geometry (distances and angles), and singular vector bases decomposes the map into orthogonal-diagonal-orthogonal (revealing rank and conditioning). Much of linear algebra is finding the “right” basis—the one where the map’s structure is most apparent.

Common Misconceptions About Linearity

Misconception 1: “Linear means the graph is a straight line.” False. A linear map \(T(\mathbf{x}) = A\mathbf{x} + \mathbf{b}\) (affine, not purely linear unless \(\mathbf{b} = \mathbf{0}\)) can have an arbitrary graph (a hyperplane); the term “linear” refers to the algebraic property (\(T(\alpha \mathbf{x} + \beta \mathbf{y}) = \alpha T(\mathbf{x}) + \beta T(\mathbf{y})\)), not the graph’s appearance. In machine learning, “linear model” often means “linear in parameters”—the model is a weighted sum of (possibly nonlinear) features.

Misconception 2: “A linear map is always invertible.” False. A linear map is invertible iff it is bijective (one-to-one and onto), which for finite-dimensional spaces requires the domain and codomain have the same dimension, and the map must have trivial kernel (injective) and full image (surjective). A tall matrix (\(m > n\)) is automatically non-surjective (cannot reach all of \(\mathbb{R}^m\)); a wide matrix (\(m < n\)) is automatically non-injective (kernel is non-trivial). Only square, full-rank matrices are invertible.

Misconception 3: “Linearity implies simplicity.” Partially true. Linear maps are simpler than general nonlinear maps—they have closed-form solutions to many problems (e.g., solving \(A\mathbf{x} = \mathbf{b}\) via Gaussian elimination), and their properties are well-characterized (rank, kernel, image). However, even linear problems can be computationally challenging: if the matrix is ill-conditioned (sensitive to perturbations), numerical solutions are unstable. Large-scale linear systems (\(A\) is \(10^9 \times 10^9\)) require iterative solvers and preconditioners, not direct methods.

Misconception 4: “A linear layer preserves all information.” False. A linear layer can compress information if its rank is less than the input dimension. A \(1000 \times 10000\) weight matrix (1000 outputs, 10000 inputs, assuming rank 1000) compresses 10000-dimensional input to a 1000-dimensional output—information is lost. If the next layer tries to reverse this, it cannot—the lost information is permanently gone.

Misconception 5: “Linear maps don’t interact—each output is independently determined by inputs.” False. The off-diagonal entries of a weight matrix \(A\) encode interactions: \(A_{ij} \neq 0\) means “input dimension \(i\) influences output dimension \(j\).” A fully connected layer couples all inputs and outputs; a sparse matrix couples only a subset.

ML Connection

Linear Layers in Neural Networks

Every artificial neural network consists of alternating linear and nonlinear operations. A standard feedforward layer is:

\[\mathbf{h}^{(\ell)} = \sigma(W^{(\ell)} \mathbf{h}^{(\ell-1)} + \mathbf{b}^{(\ell)})\]

where \(W^{(\ell)} \in \mathbb{R}^{n_\ell \times n_{\ell-1}}\) is the weight matrix (a linear map), \(\mathbf{b}^{(\ell)} \in \mathbb{R}^{n_\ell}\) is the bias (a translation), and \(\sigma\) is a nonlinearity (ReLU, tanh, sigmoid, or other activation function). The linear part \(W^{(\ell)} \mathbf{h}^{(\ell-1)} + \mathbf{b}^{(\ell)}\) is a linear map from \(n_{\ell-1}\) dimensions to \(n_\ell\) dimensions, followed by the nonlinearity \(\sigma\) applied element-wise.

Why does this architecture work? The nonlinearity \(\sigma\) breaks the linearity: if we removed \(\sigma\) and just stacked linear layers, the composition would be a single linear map (matrix multiplication is associative), and no amount of depth would increase expressivity. The nonlinearity enables depth to increase expressivity exponentially: \(d\) linear layers with \(n\)-dimensional hidden layer, separated by nonlinearities, can represent substantially more functions than a single linear layer, and the gap widdens with depth. However, the linear parts are still the workhorse—they do the “feature mixing,” combining input dimensions into hidden dimensions, and ultimately mixing hidden dimensions into outputs.

Understanding weight matrices as linear maps: Consider a simple feedforward network for image classification: input layer (784 dimensions for \(28 \times 28\) MNIST images), hidden layer (128 dimensions), output layer (10 dimensions for 10 digit classes). The first weight matrix is \(W^{(1)} \in \mathbb{R}^{128 \times 784}\). As a linear map, it transforms raw pixels (sparse, 784-dimensional, highly structured because nearby pixels are correlated) to 128-dimensional “hidden” features. This is not a dimensionality reduction—we increased from 784 to 128 is a reduction, not increase. Wait, actually \(128 < 784\), so this is dimensionality reduction. The matrix has rank at most 128, so it compresses 784-dimensional input through a 128-dimensional bottleneck. The second layer \(W^{(2)} \in \mathbb{R}^{10 \times 128}\) further reduces to 10 dimensions. The composition \(W^{(2)} W^{(1)} \in \mathbb{R}^{10 \times 784}\) has rank at most \(\min(\text{rank}(W^{(2)}), \text{rank}(W^{(1)})) \leq 10\). This means the final predictions (before the softmax nonlinearity) lie in a 10-dimensional subspace of the original 784-dimensional pixel space. This makes sense: regardless of input image, the network can only produce 10 distinct outputs (one per digit), so a 10-dimensional subspace is sufficient (and necessary, if the network is trained to distinguish all 10 classes).

Concrete ML example: sentiment classification with embeddings. In NLP, a sentiment classifier takes text as input. The first step is embedding: convert each word (represented as an integer token, e.g., word 1000) to a dense vector (e.g., 100 dimensions). The embedding matrix \(E \in \mathbb{R}^{V \times d}\) (vocabulary size \(V = 100000\), embedding dimension \(d = 100\)) is a lookup table—for token \(i\), we get the \(i\)-th row \(E_i \in \mathbb{R}^{100}\). This embedding is a linear map in a generalized sense: it maps discrete tokens to an embedding space. Next, a sequential model (RNN, Transformer) processes the sequence of embeddings, using linear layers (interspersed with nonlinearities) to mix information across time steps. The final linear layer projects the last hidden state (a 100-dimensional vector after several linear transformations) to 2 dimensions (positive vs. negative sentiment), producing logits \(\mathbf{z} = W^{\text{final}} \mathbf{h}^{\text{last}} + \mathbf{b}\). The rank of \(W^{\text{final}}\) is at most 2, encoding that only 2 independent directions of sentiment are possible. The model learns to represent sequences from the embedding and sequential processing in a way that aligns with these 2 output dimensions.

Feature Mixing and Weight Matrices

A weight matrix’s role is feature mixing: combining input features into output features in a learnable way. The entry \(W_{ij}\) quantifies how much input feature \(j\) contributes to output feature \(i\). Large magnitude means strong influence; zero means no influence; negative values mean suppressing that feature.

Concrete ML example: image classification with convolutions. A convolutional layer applies a learned filter (convolution kernel) to the input image, producing feature maps. Mathematically, convolution is a linear operation when we view it as a large, highly structured matrix: the matrix has \(H_{\text{out}} \times W_{\text{out}} \times C_{\text{out}}\) rows (one per output feature map location and channel) and \(H_{\text{in}} \times W_{\text{in}} \times C_{\text{in}}\) columns (one per input location and channel). The matrix is sparse (most entries are zero—only weights near the receptive field are nonzero) and has a special structure (Toeplitz-like, repeating the same convolution kernel at each position). This sparsity and structure makes convolutions efficient (far fewer parameters than a fully connected layer) and more expressive (the convolution kernel learns local patterns that repeat across the image, like “edge detector” or “corner detector”). Although presented as convolution, it is fundamentally a linear map \(\mathbf{z} = W \mathbf{x}\), where \(W\) is the (implicit) convolution matrix and \(\mathbf{x}\) is the flattened input image. Understanding it as a linear map with special structure helps explain why convolutional networks are so effective: they exploit the spatial structure of images (locality, translation equivariance) by restricting the weight matrix to be sparse and Toeplitz.

Feature normalization and whitening. In practice, input features often have different scales and correlations. Whitening (Chapter 2, Appendix C) applies a linear transformation: \(Z = XL^{-\top}\) (where \(L L^\top = \text{Cov}(X)\) via Cholesky decomposition). This linear map decorrelates features and equalizes scales, making the loss landscape more isotropic—gradient descent takes equal-sized steps in all directions. The weight matrix of whitening is \(L^{-\top}\), a full-rank transformation from the original space to the whitened space. Batch normalization in neural networks applies a similar (but data-dependent, mini-batch wise) linear transformation at each layer, with a learnable diagonal scaling and shift afterward: \(\mathbf{h}' = \gamma \odot \frac{\mathbf{h} - \boldsymbol{\mu}}{\boldsymbol{\sigma}} + \boldsymbol{\beta}\) (where \(\odot\) is element-wise multiplication). The normalization part is a linear map (affine, after subtracting mean and dividing by standard deviation); the learnable scaling/shift \((\gamma, \boldsymbol{\beta})\) adds expressivity. The effect is that the learned weight matrices operate in a “cleaner” coordinate system, improving training stability and convergence speed.

Kernel and Image in Model Expressivity

The image of a linear map \(T: V \to W\), denoted \(\text{im}(T)\) or \(T(V)\), is the set of all reachable outputs: \(\{T(\mathbf{v}) : \mathbf{v} \in V\}\). The image is a subspace of \(W\), and its dimension is the rank of \(T\). In a neural network, the image of the weight matrix (the linear part of a layer) determines the maximum expressivity in the next layer: if a layer’s weight matrix has rank \(r\), then all outputs lie in an \(r\)-dimensional subspace. No matter what the next layer does (nonlinearity, additional weights), it can only operate on the \(r\)-dimensional image of the previous layer.

The kernel of a linear map \(T: V \to W\), denoted \(\ker(T)\) or \(\text{Null}(T)\), is the set of inputs mapping to zero: \(\{\mathbf{v} \in V : T(\mathbf{v}) = \mathbf{0}\}\). The kernel is a subspace of \(V\), and its dimension is the nullity of \(T\). By the rank-nullity theorem, \(\dim(V) = \text{rank}(T) + \text{nullity}(T)\). Geometrically, the kernel represents “directions that are invisible to the map”—any two inputs differing by a vector in the kernel map to the same output.

Concrete ML example: redundant features in regression. Suppose a linear regression model for predicting house prices has features: square footage, number of rooms, and “rooms-per-1000-sq-ft” (a derived feature). If the third feature is a linear combination of the first two (derived from them), it adds no new information. The design matrix \(X\) (with 3 columns, one per feature) has rank 2, not 3. The null space is 1-dimensional: spanning the vector that represents “the redundancy” (typically something like the gradient of the linear relationship). The kernel of the regression operator (the normal equations \(X^\top X \mathbf{w} = X^\top \mathbf{y}\)) is non-trivial: infinitely many weight vectors \(\mathbf{w}\) produce identical predictions. Interpretability breaks: which weight is the “true” effect of the third feature? There is no answer—any weight for the third feature can be offset by adjusting the others. This is multicollinearity, a classic problem in statistics and ML. The remedy is regularization (ridge, LASSO) or feature selection (dropping the redundant feature), both of which reduce the dimensionality or impose structure to make the solution unique.

Concrete ML example: kernel methods and implicit feature spaces. In support vector machines (SVMs) with a non-linear kernel (e.g., Gaussian RBF kernel), the data are implicitly mapped to a high-dimensional (possibly infinite-dimensional) feature space where they become linearly separable. The kernel function \(K(\mathbf{x}, \mathbf{x}') = \langle \phi(\mathbf{x}), \phi(\mathbf{x}') \rangle\) computes dot products in this implicit space without explicitly constructing \(\phi(\mathbf{x})\). However, the machinery is still linear algebra in disguise: the SVM decision boundary is \(\sum_i \alpha_i K(\mathbf{x}_i, \mathbf{x}) + b = 0\), which is a linear combination of kernel functions (a hyperplane in the implicit feature space). The rank of the Gram matrix \(K\) (with entries \(K_{ij} = K(\mathbf{x}_i, \mathbf{x}_j)\)) determines the effective dimensionality of the problem: if the Gram matrix has rank \(r\), there are only \(r\) independent directions in the feature space (relative to the data points), and the support vector machine can have at most \(r\) support vectors. This limits the model’s complexity, which acts as implicit regularization.

Rank and Capacity in Linear Models

Rank as a measure of model capacity. The rank of a weight matrix directly bounds the model’s capacity—how complex a function it can represent. A weight matrix of rank \(r\) has an image of dimension \(r\), so all outputs lie in an \(r\)-dimensional subspace of the output space. If the output space is \(m\)-dimensional, a rank-\(r\) matrix can only represent functions into an \(r\)-dimensional subspace; it cannot reach all of \(\mathbb{R}^m\).

In a linear regression setting with design matrix \(X \in \mathbb{R}^{n \times d}\), the effective degrees of freedom is \(\text{rank}(X)\), not the number of features \(d\). If \(\text{rank}(X) = r < d\) (multicollinearity), the model has only \(r\) independent parameters, even though there are \(d\) parameter dimensions. The excess \(d - r\) parameters are “free”—they can adjust without changing predictions. This non-identifiability is problematic: the solution to the least-squares problem is non-unique; inference on individual parameters becomes undecidable (which weight is the “true” effect of a feature?). The solution (regularization, feature selection) amounts to imposing structure on the \(d - r\) free directions.

Concrete ML example: bottleneck layers in autoencoders. An autoencoder learns a compressed representation of data by forcing it through a bottleneck layer with dimension much smaller than the input. For example, an autoencoder for MNIST images (784 pixels) might have a hidden layer of dimension 32 (a \(784 \to 32\) bottleneck). The weight matrix from the input layer to the bottleneck is \(W_{\text{enc}} \in \mathbb{R}^{32 \times 784}\). The rank of this matrix is at most 32 (it maps \(\mathbb{R}^{784}\) to \(\mathbb{R}^{32}\)), so the bottleneck is a dimensionality reduction step. The 784-dimensional image diversity is compressed to a 32-dimensional code. The decoder’s weight matrix \(W_{\text{dec}} \in \mathbb{R}^{784 \times 32}\) maps back, recovering (approximately) the original image. The capacity of the autoencoder is limited by the bottleneck: the code dimension 32 is the effective dimensionality of the learned representation. If the true intrinsic dimension of natural images is higher—say, 50—then a 32-dimensional bottleneck will incur approximation error (bias), permanently losing information. Conversely, if the true intrinsic dimension is 20, a 32-dimensional bottleneck has excess capacity but generalizes better (lower variance, as it cannot overfit every data point in high-dimensional directions).

Composition of Linear Maps in Deep Architectures

Deep neural networks stack many layers, each applying a linear transformation followed by a nonlinearity. Understanding how ranks compose explains representational capacity and identifies bottlenecks.

Mathematically, composing two linear maps \(T_1: \mathbb{R}^d \to \mathbb{R}^h\) and \(T_2: \mathbb{R}^h \to \mathbb{R}^m\) yields a composite map \(T_2 \circ T_1: \mathbb{R}^d \to \mathbb{R}^m\). If \(T_1\) is represented by matrix \(A \in \mathbb{R}^{h \times d}\) and \(T_2\) by matrix \(B \in \mathbb{R}^{m \times h}\), then the composition is \(BA \in \mathbb{R}^{m \times d}\). The rank of the composition satisfies:

\[\text{rank}(BA) \leq \min(\text{rank}(B), \text{rank}(A))\]

This is a fundamental inequality: the rank of a product is at most the minimum of the ranks. If either factor has low rank, the product’s rank is limited. In a deep network, if any intermediate layer (the weight matrix, ignoring the nonlinearity) has low rank, all subsequent computations are constrained to an even lower-rank subspace. This is why low-rank hidden layers are bottlenecks: they compress information, and downstream layers cannot recover what was lost.

Concrete ML example: representational capacity in deep networks for vision. A convolutional neural network for image classification stacks convolution, pooling, and normalization layers. Each convolution layer applies a learned (hopefully full-rank) transformation; pooling (max pooling) discards information, reducing spatial dimension but not feature channel dimension (though effectively reducing information content). After several layers, spatial dimensions shrink from, say, \(224 \times 224\) (ImageNet images) to \(1 \times 1\) (after 5 pooling steps with stride 2 each, \(224 / 2^5 \approx 7 \to 1\) with additional steps). At this point, the “image” is reduced to a set of feature vectors (one per output channel, e.g., 2048 channels in ResNet50). These vectors are then fed to a fully connected classifier (linear layers with softmax), which predicts one of 1000 object categories. The compressive path (convolution + pooling reducing spatial dimensions, though possibly increasing channel count) acts as feature extraction. A fully connected layer would require \(224^2 \times C\) parameters (where \(C\) is the number of channels), causing parameter explosion. Instead, convolutions are local (small receptive fields), reducing parameters and exploiting spatial locality. The overall pipeline is: raw image → learned representation (via convolutions) → final classification (via fully connected layers). The rank of the learned representation (the bottleneck at the outputs of the final convolutional layer) determines how much of the image diversity can be captured—if the representation dimension is too small relative to the data complexity, the classifier cannot achieve low training error, no matter how expressive the final fully connected layer.

Definitions

Linear Map

Formal Definition: A function \(T: V \to W\) between vector spaces \(V\) and \(W\) (over the same field, e.g., \(\mathbb{R}\)) is a linear map (or linear transformation) if it preserves addition and scalar multiplication: for all \(\mathbf{u}, \mathbf{v} \in V\) and all scalars \(c \in \mathbb{R}\), \(T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})\) and \(T(c\mathbf{v}) = c T(\mathbf{v})\).

Explicit Assumptions: \(V\) and \(W\) are vector spaces over the same field (throughout this chapter, \(\mathbb{R}\)); operations like addition and scalar multiplication are defined on both spaces.

Notation Discipline: Write \(T: V \to W\) to denote a linear map from \(V\) to \(W\). Write \(T(\mathbf{v})\) or \(T \mathbf{v}\) for the image of a vector. Composition of \(T: V \to W\) and \(S: W \to Z\) is \(S \circ T\) or \(ST\), meaning \((S \circ T)(\mathbf{v}) = S(T(\mathbf{v}))\).

Usage and Interpretation: Linear maps are composable (unlike generic nonlinear functions), satisfying associativity: \((R \circ S) \circ T = R \circ (S \circ T)\). They are invertible iff bijective (one-to-one and onto), and many problems (solving \(T(\mathbf{x}) = \mathbf{b}\), eigenvalue equations \(T(\mathbf{v}) = \lambda \mathbf{v}\)) admit closed-form solutions. The set of all linear maps from \(V\) to \(W\) forms a vector space (operations: pointwise addition, scalar multiplication).

Valid Example: \(T: \mathbb{R}^2 \to \mathbb{R}^2\) defined by \(T(x, y) = (2x + y, x - y)\). Check linearity: \(T(c(x, y)) = T(cx, cy) = (2cx + cy, cx - cy) = c(2x + y, x - y) = cT(x, y)\) ✓. And \(T((x_1, y_1) + (x_2, y_2)) = T(x_1 + x_2, y_1 + y_2) = (2(x_1+x_2) + (y_1+y_2), (x_1+x_2) - (y_1+y_2)) = (2x_1 + y_1, x_1 - y_1) + (2x_2 + y_2, x_2 - y_2) = T(x_1, y_1) + T(x_2, y_2)\) ✓.

Failure Case: \(T(x, y) = (x^2, y)\) is NOT linear: \(T(2(1, 1)) = T(2, 2) = (4, 2)\) but \(2T(1, 1) = 2(1, 1) = (2, 2)\). The nonlinearity in the first coordinate breaks linearity.

Explicit ML Relevance: Every neural network layer (before the nonlinearity) is a linear map: \(\mathbf{h} = W\mathbf{x} + \mathbf{b}\) (ignoring the nonlinearity and bias, the core \(W\mathbf{x}\) is linear). Regression, classification, dimensionality reduction, and kernel methods all rely on linear maps as building blocks. Understanding linear map properties (invertibility, rank, kernel) directly informs model capacity and behavior.

Domain and Codomain

Formal Definition: For a linear map \(T: V \to W\), the domain is \(V\) and the codomain is \(W\). (Do not confuse codomain with the range or image, which is a subset of the codomain containing only the actually-attained output values.)

Explicit Assumptions: The domain \(V\) is the input space (all possible inputs to \(T\)); the codomain \(W\) is the output space (all possible output positions, not just those actually reached by \(T\)).

Notation Discipline: Write \(T: V \to W\) to indicate domain \(V\) and codomain \(W\). For an \(m \times n\) matrix, the domain is \(\mathbb{R}^n\) (column vectors of size \(n\)) and the codomain is \(\mathbb{R}^m\) (row vectors of size \(m\)).

Usage and Interpretation: The codomain is part of the definition of the linear map. Two maps with the same input–output rule but different codomains are technically different: \(T_1: \mathbb{R}^2 \to \mathbb{R}^2\) and \(T_2: \mathbb{R}^2 \to \mathbb{R}^3\) defined by \(T_1(x, y) = (x, y, 0)\) (if we ignore the third coordinate) differ in surjectivity (onto-ness), even though the rule is similar. This matters for defining invertibility and discussing ranks.

Valid Example: \(T: \mathbb{R}^3 \to \mathbb{R}^2\) defined by \(T(x, y, z) = (x + y, z)\). Domain: \(\mathbb{R}^3\). Codomain: \(\mathbb{R}^2\). The range (image of all inputs) is all of \(\mathbb{R}^2\) (for any \((a, b) \in \mathbb{R}^2\), choose \((x, y, z) = (a, 0, b)\) to get \(T(a, 0, b) = (a, b)\)), so the map is surjective.

Failure Case: Confusing “codomain” with “range.” The codomain is fixed by the definition; the range depends on which inputs are actually fed to the map. Example: \(T: \mathbb{R}^2 \to \mathbb{R}^2\) defined by \(T(x, y) = (x, 0)\) has codomain \(\mathbb{R}^2\) but range (image) is the \(x\)-axis (1-dimensional), a proper subset of the codomain.

Explicit ML Relevance: In neural networks, the codomain dimension of a layer is the layer’s output dimension (parameter \(n_\ell\) in “dense layer with \(n_\ell\) units”). The domain dimension is the input feature count. A layers with domain \(\mathbb{R}^{784}\) and codomain \(\mathbb{R}^{128}\) compresses inputs (if rank is \(< 128\)). The distinction between codomain and range reveals that the layer’s output may not fill the entire codomain space—information density may be low.

Kernel

Formal Definition: For a linear map \(T: V \to W\), the kernel (or null space), denoted \(\ker(T)\) or \(\text{Null}(T)\), is the set of all vectors in the domain that map to zero: \(\ker(T) := \{ \mathbf{v} \in V : T(\mathbf{v}) = \mathbf{0} \}\).

Explicit Assumptions: The zero vector \(\mathbf{0}_W \in W\) exists (every vector space has a zero). \(T(\mathbf{0}_V) = \mathbf{0}_W\) always (linearity), so the kernel is non-empty.

Notation Discipline: Write \(\ker(T)\) for the kernel of \(T: V \to W\). For a matrix \(A \in \mathbb{R}^{m \times n}\), the kernel is \(\ker(A) = \{ \mathbf{x} \in \mathbb{R}^n : A\mathbf{x} = \mathbf{0} \}\), also called the null space.

Usage and Interpretation: The kernel measures “how much the map compresses.” A trivial kernel (\(\ker(T) = \{\mathbf{0}\}\)) means the map is injective (one-to-one); different inputs map to different outputs. A non-trivial kernel means multiple distinct inputs produce the same output—information is “lost.” The dimension of the kernel is the nullity, quantifying degrees of freedom lost.

Valid Example: \(T: \mathbb{R}^3 \to \mathbb{R}^2\) defined by \(T(x, y, z) = (x, y)\). Kernel: solve \((x, y) = (0, 0)\), giving \(x = y = 0\) and \(z\) free. So \(\ker(T) = \{ (0, 0, z) : z \in \mathbb{R} \} = \text{span}((0, 0, 1))\), a 1-dimensional subspace (the \(z\)-axis). Nullity = 1.

Failure Case: Assuming the kernel is always trivial. Many real matrices have non-trivial kernels (e.g., any matrix with fewer rows than columns: an \(m \times n\) matrix with \(m < n\) has a non-trivial null space by rank-nullity: \(\text{nullity} = n - \text{rank} \geq n - m > 0\)).

Explicit ML Relevance: In regression with a rank-deficient design matrix, the null space of that matrix parameterizes non-unique solutions. If \(X \mathbf{w} = \mathbf{y}\) and \(\mathbf{v} \in \ker(X)\), then \(X(\mathbf{w} + \mathbf{v}) = \mathbf{y}\) as well. This non-identifiability is problematic: which weights are the “true” effects? Regularization (ridge, LASSO) selects a unique solution by imposing structure on the kernel directions.

Image

Formal Definition: For a linear map \(T: V \to W\), the image (or range), denoted \(\text{im}(T)\) or \(T(V)\), is the set of all vectors in the codomain that are actually attained as outputs: \(\text{im}(T) := \{ T(\mathbf{v}) : \mathbf{v} \in V \} = \{ \mathbf{w} \in W : \mathbf{w} = T(\mathbf{v}) \text{ for some } \mathbf{v} \in V \}\).

Explicit Assumptions: The image is a subset of the codomain (image \(\subseteq\) codomain), but may be proper (smaller, if the map is not surjective).

Notation Discipline: Write \(\text{im}(T)\) or \(T(V)\) for the image of \(T: V \to W\). For a matrix \(A \in \mathbb{R}^{m \times n}\), the image is \(\text{im}(A) = \{ A\mathbf{x} : \mathbf{x} \in \mathbb{R}^n \} = \text{span}(\text{columns of } A) = \text{Col}(A)\).

Usage and Interpretation: The image captures the “reachable outputs.” If the image is low-dimensional, the map compresses diversity: many distinct inputs collapse to a small output space. The dimension of the image is the rank, quantifying the “effective output dimension.” A surjective map has image equal to the entire codomain.

Valid Example: \(T: \mathbb{R}^2 \to \mathbb{R}^3\) defined by \(T(x, y) = (x, y, x+y)\). Image: all outputs are of the form \((x, y, x+y) = x(1, 0, 1) + y(0, 1, 1)\). So \(\text{im}(T) = \text{span}((1, 0, 1), (0, 1, 1))\), a 2-dimensional subspace of \(\mathbb{R}^3\). Rank = 2. The map is not surjective: not all 3-dimensional vectors can be reached (e.g., \((0, 0, 1)\) is not in the image, as that would require \(x = y = 0\) and \(x + y = 1\), a contradiction).

Failure Case: Assuming the image is always full-dimensional (rank = codomain dimension). Many maps have image strictly smaller than the codomain (rank \(<\) codomain dimension), indicating compression or degeneracy.

Explicit ML Relevance: In neural networks, the image of a weight matrix determines the maximum capacity of that layer: if the weight matrix \(W^{(\ell)}\) has rank \(r\), all outputs of that layer lie in an \(r\)-dimensional subspace of the \(n_\ell\)-dimensional output space. Subsequent layers cannot appeal to the \(n_\ell - r\) “lost” dimensions. This is why low-rank hidden layers are bottlenecks. In autoencoders, the image of the encoder determines the dimension of the learned representation.

Rank

Formal Definition: The rank of a linear map \(T: V \to W\) is the dimension of its image: \(\text{rank}(T) := \dim(\text{im}(T))\). For a matrix \(A \in \mathbb{R}^{m \times n}\), the rank is the dimension of its column space: \(\text{rank}(A) := \dim(\text{Col}(A))\).

Explicit Assumptions: \(V\) and \(W\) are finite-dimensional (rank is well-defined and finite). The dimension of the image is at most \(\min(\dim(V), \dim(W))\) (a map from \(n\)-dimensional space to \(m\)-dimensional space has rank \(\leq \min(m, n)\)).

Notation Discipline: Write \(\text{rank}(T)\) for the rank of \(T: V \to W\), or \(\text{rank}(A)\) for a matrix. Sometimes denoted \(r(T)\) or \(\text{rk}(A)\).

Usage and Interpretation: Rank quantifies the “information passing through” the map. Full rank means the map is injective (for wide or square matrices) or surjective (for tall or square matrices), or both (for square, invertible matrices). Rank deficiency means the map loses information or fails to reach the full codomain.

Valid Example: \(A = \begin{pmatrix} 1 & 2 \\ 0 & 0 \end{pmatrix} \in \mathbb{R}^{2 \times 2}\). Column space: \(\text{span}((1, 0)^\top)\) (just the first column; the second is a multiple of the first, providing no new direction). Rank = 1. The matrix is singular (not invertible).

Failure Case: Confusing “rank” with “number of nonzero entries” or “number of nonzero rows/columns.” Rank is the dimension of the column space, not a count. A matrix with many nonzero entries can have low rank (e.g., a matrix of all 1s has rank 1).

Explicit ML Relevance: Rank directly measures model capacity. A weight matrix of rank \(r\) has effective degrees of freedom \(r\)—even if it has \(mn\) parameters, its output diversity is limited to \(r\) independent directions. Regularization techniques (ridge regression, dropout) often implicitly reduce rank by preventing the network from learning full-rank weight matrices. Rank reduction is a form of implicit compression.

Nullity

Formal Definition: The nullity of a linear map \(T: V \to W\) is the dimension of its kernel: \(\text{nullity}(T) := \dim(\ker(T))\). For a matrix \(A \in \mathbb{R}^{m \times n}\), the nullity is the dimension of its null space: \(\text{nullity}(A) := \dim(\text{Null}(A)) = n - \text{rank}(A)\) (by rank-nullity).

Explicit Assumptions: \(V\) is finite-dimensional. Nullity is non-negative and at most \(\dim(V)\) (the null space is a subspace of the domain).

Notation Discipline: Write \(\text{nullity}(T)\) for the nullity of \(T: V \to W\). Sometimes denoted \(\text{null}(T)\) or \(\text{defect}(T)\).

Usage and Interpretation: Nullity quantifies the “information lost” by the map. A trivial nullity (= 0) means no information is lost; the map is injective. A large nullity means the map collapses many directions.

Valid Example: For the matrix \(A = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \end{pmatrix} \in \mathbb{R}^{2 \times 3}\), rank = 2 (the first two columns are independent). By rank-nullity, nullity = \(3 - 2 = 1\). The null space is \(\{ (x, y, z) : x + z = 0, y + z = 0 \} = \{ (-z, -z, z) \} = \text{span}((−1, −1, 1)^\top)\), a 1-dimensional space ✓.

Failure Case: Assuming nullity is always zero. For any non-square or rank-deficient matrix, nullity is positive.

Explicit ML Relevance: In underdetermined regression (\(d > n\), more parameters than data points), the nullity of the design matrix is positive, indicating infinitely many solutions. The solution set is an affine subspace of dimension equal to nullity. Regularization breaks this symmetry by imposing additional structure (e.g., smallest-norm solution via ridge regression).

Matrix Representation

Formal Definition: Let \(T: V \to W\) be a linear map, and let \(\mathcal{B} = \{ \mathbf{b}_1, \ldots, \mathbf{b}_n \}\) be a basis for \(V\) and \(\mathcal{C} = \{ \mathbf{c}_1, \ldots, \mathbf{c}_m \}\) be a basis for \(W\). The matrix representation of \(T\) with respect to bases \(\mathcal{B}\) and \(\mathcal{C}\), denoted \([T]_{\mathcal{B}, \mathcal{C}}\) or \(M_{\mathcal{B}, \mathcal{C}}(T)\), is the \(m \times n\) matrix whose \(j\)-th column is the coordinate vector of \(T(\mathbf{b}_j)\) with respect to basis \(\mathcal{C}\): \[ [T]_{\mathcal{B}, \mathcal{C}} = \left[ [T(\mathbf{b}_1)]_\mathcal{C} \mid [T(\mathbf{b}_2)]_\mathcal{C} \mid \cdots \mid [T(\mathbf{b}_n)]_\mathcal{C} \right] \]

Explicit Assumptions: Both \(V\) and \(W\) are finite-dimensional with chosen bases \(\mathcal{B}\) and \(\mathcal{C}\). Different basis choices yield different matrix representations.

Notation Discipline: Write \([T]_{\mathcal{B}, \mathcal{C}}\) for a matrix of \(T\) with respect to specific bases. If \(\mathcal{B} = \mathcal{C}\), write \([T]_\mathcal{B}\) (a square matrix for an endomorphism). The standard basis of \(\mathbb{R}^n\) is sometimes implicit, so a matrix \(A \in \mathbb{R}^{m \times n}\) is understood to represent some map between \(\mathbb{R}^n\) and \(\mathbb{R}^m\).

Usage and Interpretation: The matrix representation allows computation: if \(\mathbf{v} = \sum_j c_j \mathbf{b}_j\) with coordinate vector \([\mathbf{v}]_\mathcal{B} = (c_1, \ldots, c_n)^\top\), then \([T(\mathbf{v})]_\mathcal{C} = [T]_{\mathcal{B}, \mathcal{C}} \cdot [\mathbf{v}]_\mathcal{B}\) (matrix-vector multiplication). The same linear map has infinitely many matrix representations (one per choice of bases); these representations are related by similarity/change-of-basis transformations.

Valid Example: \(T: \mathbb{R}^2 \to \mathbb{R}^2\), \(T(x, y) = (2x + y, x - y)\). Standard basis \(\mathcal{B} = \mathcal{C} = \{ (1, 0)^\top, (0, 1)^\top \}\). Then \(T((1, 0)^\top) = (2, 1)^\top = 2(1, 0)^\top + 1(0, 1)^\top\), so the first column is \((2, 1)^\top\). And \(T((0, 1)^\top) = (1, -1)^\top\). Matrix: \([T]_{\mathcal{B}, \mathcal{C}} = \begin{pmatrix} 2 & 1 \\ 1 & -1 \end{pmatrix}\) ✓.

Failure Case: Forgetting to specify bases. The matrix representation depends on the bases chosen. Different bases yield different numerical matrices representing the same geometric transformation.

Explicit ML Relevance: Every neural network layer weight matrix is a representation of a linear map. The choice of basis (feature coordinates, hidden representations) determines the numerical values of the weights. If we preprocess features (change from raw pixels to PCA coordinates), the weight matrix must be recalibrated to account for the new feature basis.

Change of Basis for Linear Maps

Formal Definition: Let \(T: V \to V\) be an endomorphism (a map from a space to itself). If \(\mathcal{B}\) and \(\mathcal{B}'\) are two bases for \(V\), then the matrix representations \(A = [T]_\mathcal{B}\) and \(A' = [T]_{\mathcal{B}'}\) are related by: \[ A' = P^{-1} A P \] where \(P = [I]_{\mathcal{B}', \mathcal{B}}\) is the change-of-basis matrix from \(\mathcal{B}'\) to \(\mathcal{B}\). This relationship is called a similarity transformation.

Explicit Assumptions: Both bases are for the same space \(V\); the linear map \(T: V \to V\) is an endomorphism. The change-of-basis matrix \(P\) is always invertible (since it represents the identity map between different coordinate systems).

Notation Discipline: Write \(P_{\mathcal{B}' \to \mathcal{B}}\) or \([I]_{\mathcal{B}', \mathcal{B}}\) for the change-of-basis matrix from \(\mathcal{B}'\) to \(\mathcal{B}\). The relationship \(A' = P^{-1} A P\) is the similarity transformation.

Usage and Interpretation: Similarity transformations preserve the essential structure of the linear map—eigenvalues, rank, determinant, and trace are all invariant under similarity. But the numerical matrix entries can change dramatically depending on the basis. The goal is to find a basis where the matrix is “simple” (e.g., diagonal, upper-triangular)—this is the essence of matrix diagonalization and other canonical forms.

Valid Example: \(T: \mathbb{R}^2 \to \mathbb{R}^2\), \(T(x, y) = (2x, y)\) (scaling \(x\) by 2, leaving \(y\) unchanged). In the standard basis, \(A = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix}\) (already diagonal). In the basis \(\mathcal{B}' = \{ (1, 1)^\top, (1, -1)^\top \}\), computing the new matrix requires the change-of-basis matrix. This is more involved; the key point is that the similarity transformation relates the two representations.

Failure Case: Confusing the change-of-basis matrix with the linear map itself. \(P\) is not the map \(T\); it is a representation of the identity in two different bases, used to convert between coordinate systems.

Explicit ML Relevance: In deep learning, changing the feature basis (e.g., via preprocessing) requires retraining or recalibrating weights. Whitening and normalization are foundation-level changes-of-basis that simplify subsequent learning. Understanding that different bases yield different weights (even for the same geometric transformation) helps explain why feature engineering and preprocessing are so important.

Composition of Linear Maps

Formal Definition: Let \(T: V \to W\) and \(S: W \to Z\) be linear maps. Their composition is the map \(S \circ T: V \to Z\) defined by \((S \circ T)(\mathbf{v}) = S(T(\mathbf{v}))\). If \(T\) is represented by matrix \(B\) (with respect to bases for \(V\) and \(W\)) and \(S\) by matrix \(C\) (with bases for \(W\) and \(Z\)), then \(S \circ T\) is represented by the matrix product \(CB\).

Explicit Assumptions: The codomain of \(T\) equals the domain of \(S\) (both are \(W\)), enabling composition. Associativity holds: \((R \circ S) \circ T = R \circ (S \circ T)\).

Notation Discipline: Write \(S \circ T\) for the composition (sometimes just \(ST\)). As matrices, the composition is \(CB\) (note: matrix multiplication is opposite to the order of function composition in notation, so extra care is needed).

Usage and Interpretation: Composition is how we build complex transformations from simpler ones. Neural networks are compositions of linear layers (weights) and nonlinearities. The rank of the composition is at most the minimum of the two ranks: \(\text{rank}(S \circ T) \leq \min(\text{rank}(S), \text{rank}(T))\). A bottleneck (low-rank) layer limits the overall capacity.

Valid Example: \(T: \mathbb{R}^2 \to \mathbb{R}^2\), \(T(x, y) = (x + y, x - y)\) (matrix \(B = \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\)), and \(S: \mathbb{R}^2 \to \mathbb{R}^2\), \(S(u, v) = (2u, v)\) (matrix \(C = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix}\)). Composition: \(S \circ T\) has matrix \(CB = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 2 & 2 \\ 1 & -1 \end{pmatrix}\). Rank of \(B\) is 2, rank of \(C\) is 2, so rank of composition is 2 (full rank in this case).

Failure Case: Forgetting that the order matters in matrix multiplication (non-commutative): \(CB \neq BC\) in general. This corresponds to composition order: \(S \circ T \neq T \circ S\) (different operations).

Explicit ML Relevance: Deep neural networks are compositions: a 3-layer network computes \(\sigma_3(W_3 \sigma_2(W_2 \sigma_1(W_1 \mathbf{x})))\) where \(W_i\) are weight matrices and \(\sigma_i\) are nonlinearities. Understanding composition helps explain why depth increases expressivity (compositions of nonlinear maps can represent more functions than single maps), and why low-rank layers create bottlenecks (they constrain all subsequent computations).

Identity Map

Formal Definition: The identity map on a vector space \(V\), denoted \(I_V\) or just \(I\), is the map \(I_V: V \to V\) defined by \(I_V(\mathbf{v}) = \mathbf{v}\) for all \(\mathbf{v} \in V\). It is the unique linear map such that \(T \circ I_V = T\) and \(I_V \circ T = T\) for any linear map \(T\) to/from \(V\). The matrix representation of \(I_V\) (in any basis) is the identity matrix \(I \in \mathbb{R}^{n \times n}\) (all 1s on the diagonal, 0s elsewhere).

Explicit Assumptions: Every vector space has a unique identity map. It is always invertible (its own inverse) and has full rank.

Notation Discipline: Write \(I_V\) for the identity map, or just \(I\) when the space is clear. The matrix is \(I_n\) (for \(\mathbb{R}^n\)), or just \(I\).

Usage and Interpretation: The identity is the baseline: any invertible map can be related to the identity via the equation \(T \circ T^{-1} = I\). It is the neutral element for function composition.

Valid Example: \(I: \mathbb{R}^2 \to \mathbb{R}^2\), \(I(x, y) = (x, y)\). Matrix: \(I = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}\). ✓

Failure Case: Confusing the identity with other “simple” transformations like scal by a constant. \(cI\) (scaling all dimensions by \(c\)) is not the identity unless \(c = 1\).

Explicit ML Relevance: The identity is the reference point for analyzing transformations. In neural networks, a “skip connection” (adding the input to the output: \(\mathbf{h}' = \mathbf{h} + W\mathbf{x}\)) includes an implicit identity term, which helps preserve information and stabilize training (as in ResNets).

Invertible Linear Map

Formal Definition: A linear map \(T: V \to W\) is invertible if there exists a linear map \(T^{-1}: W \to V\) such that \(T^{-1} \circ T = I_V\) and \(T \circ T^{-1} = I_W\) (composing with the inverse yields the identity in both directions). Equivalently, \(T\) is bijective (one-to-one and onto). For finite-dimensional spaces of equal dimension, \(T\) is invertible iff it is injective (ker\(T = \{\mathbf{0}\}\)) iff it is surjective (im\(T = W\)).

Explicit Assumptions: If \(T: V \to W\) with \(\dim(V) \neq \dim(W)\), then \(T\) cannot be invertible (cannot have a two-sided inverse). If \(\dim(V) = \dim(W) = n\) (equal, finite dimensions), \(T\) is invertible iff it is bijective iff rank\(T = n\).

Notation Discipline: Write \(T^{-1}\) for the inverse map (if it exists). For matrices, \(A^{-1}\) is the inverse matrix, defined for square, non-singular matrices.

Usage and Interpretation: Invertible maps are the “good” maps—no information is lost, and they can be undone. In solving linear systems \(T(\mathbf{x}) = \mathbf{b}\), if \(T\) is invertible, the solution is unique: \(\mathbf{x} = T^{-1}(\mathbf{b})\). Non-invertible maps complicate solutions: infinitely many solutions (if \(\mathbf{b}\) is in the image but the kernel is non-trivial) or no solution (if \(\mathbf{b}\) is outside the image).

Valid Example: \(T: \mathbb{R}^2 \to \mathbb{R}^2\), \(T(x, y) = (x + y, x - y)\). Kernel: solve \((x + y, x - y) = (0, 0)\), yielding \(x = y = 0\). Ker\(T = \{\mathbf{0}\}\), so injective. Image: any \((a, b) \in \mathbb{R}^2\) can be written as \((x+y, x-y)\) with \(x = (a+b)/2, y = (a-b)/2\), so surjective. Thus \(T\) is invertible. \(T^{-1}(a, b) = ((a+b)/2, (a-b)/2)\). ✓

Failure Case: Assuming a matrix is invertible without checking. Only square, full-rank matrices are invertible. A \(3 \times 3\) matrix with rank 2 is singular.

Explicit ML Relevance: In neural networks, non-invertible layers (e.g., dimensionality reduction) are bottlenecks: information is lost and cannot be recovered. Invertible neural networks (e.g., normalizing flows) maintain a bijection between input and output, preserving information and enabling likelihood-based training. The invertibility of weight matrices affects the conditioning of the optimization landscape—ill-conditioned (nearly singular) matrices slow training.

Isomorphism

Formal Definition: A linear map \(T: V \to W\) is an isomorphism if it is a bijective (one-to-one and onto) linear map. Equivalently, \(T\) is invertible. Two spaces \(V\) and \(W\) are isomorphic, written \(V \cong W\), if there exists an isomorphism between them. A fundamental theorem: two finite-dimensional vector spaces are isomorphic iff they have the same dimension.

Explicit Assumptions: \(V\) and \(W\) are both vector spaces (over the same field). If finite-dimensional, isomorphism iff equal dimension.

Notation Discipline: Write \(T: V \xrightarrow{\cong} W\) or \(V \cong W\) to denote an isomorphism. The symbol \(\cong\) means “isomorphic to” or “same structure.”

Usage and Interpretation: Isomorphic spaces are “the same” from a linear algebra perspective—any property of one translates to the other via the isomorphism. All \(n\)-dimensional vector spaces over \(\mathbb{R}\) are isomorphic to \(\mathbb{R}^n\), so the concrete \(\mathbb{R}^n\) captures the full structure.

Valid Example: \(\mathcal{P}_2\) (polynomials of degree \(\leq 2\)) and \(\mathbb{R}^3\) are isomorphic. Isomorphism: \(\phi: \mathcal{P}_2 \to \mathbb{R}^3\), \(\phi(a_0 + a_1 x + a_2 x^2) = (a_0, a_1, a_2)^\top\) (coordinate vector w.r.t. the standard basis \(\{1, x, x^2\}\)). Both spaces are 3-dimensional, and this map is a bijective linear map (isomorphism).

Failure Case: Assuming isomorphic spaces are identical. They have the same linear structure but may be concretely different (polynomials vs. vectors). The isomorphism is a bridge, not an equality.

Explicit ML Relevance: Learned representations in neural networks are supposed to be isomorphic to the true underlying data manifold (or a simpler structure). If the learned representation is poor, the isomorphism breaks, and the model cannot effectively capture the data distribution. Representation learning aims to find a good isomorphism from data space to a learned feature space where the task (e.g., classification) is easy.

Injective Linear Map

Formal Definition: A linear map \(T: V \to W\) is injective (or one-to-one) if different inputs map to different outputs: \(T(\mathbf{u}) = T(\mathbf{v}) \implies \mathbf{u} = \mathbf{v}\). Equivalently, \(\ker(T) = \{\mathbf{0}\}\) (the kernel is trivial).

Explicit Assumptions: Injectivity is a property of the map itself; it does not depend on the codomain dimension (as long as dimensions are finite).

Notation Discipline: Write “T is injective” or “\(T\) is one-to-one.” The kernel criterion: \(T\) is injective iff ker\(T = \{\mathbf{0}\}\).

Usage and Interpretation: Injective maps preserve diversity: distinct inputs yield distinct outputs, so no information is lost through the map. For a matrix \(A\), injectivity means the null space is trivial (rank = number of columns, when \(A\) is tall).

Valid Example: \(T: \mathbb{R}^2 \to \mathbb{R}^3\), \(T(x, y) = (x, y, x+y)\). Kernel: \(T(x, y) = (0, 0, 0)\) implies \(x = y = x + y = 0\), so \(x = y = 0\). Ker\(T = \{\mathbf{0}\}\), so injective. ✓

Failure Case: Assuming injective implies surjective. A tall matrix (\(m > n\)) is injective (if full rank) but not surjective (cannot reach all of \(\mathbb{R}^m\)). Injectivity and surjectivity are independent for non-square maps.

Explicit ML Relevance: In neural networks, an injective encoding preserves input diversity—if two input samples differ, their encodings differ. This is desirable for representation learning. However, most neural networks are not injective—neurons can saturate (ReLU outputs zero for negative inputs), causing different inputs to map to the same hidden state.

Surjective Linear Map

Formal Definition: A linear map \(T: V \to W\) is surjective (or onto) if every element of the codomain is attained as an output: for all \(\mathbf{w} \in W\), there exists \(\mathbf{v} \in V\) such that \(T(\mathbf{v}) = \mathbf{w}\). Equivalently, im\(T = W\) (the image is the entire codomain), or rank\(T = \dim(W)\).

Explicit Assumptions: Surjectivity requires the codomain dimension \(\leq\) domain dimension (for finite dimensions): if \(\dim(W) > \dim(V)\), surjectivity is impossible.

Notation Discipline: Write “T is surjective” or “\(T\) is onto.” The image criterion: \(T\) is surjective iff im\(T = W\).

Usage and Interpretation: Surjective maps “cover” the codomain—all output values can be reached. For a matrix \(A \in \mathbb{R}^{m \times n}\), surjectivity means rank = \(m\) (full row rank).

Valid Example: \(T: \mathbb{R}^3 \to \mathbb{R}^2\), \(T(x, y, z) = (x, y)\) (projection onto the first two coordinates). Image: all of \(\mathbb{R}^2\) (for any \((a, b) \in \mathbb{R}^2\), choose \((a, b, 0)\) to get \(T(a, b, 0) = (a, b)\)). Surjective. ✓

Failure Case: Assuming surjective implies injective. A wide matrix (\(m < n\)) is surjective (if full row rank) but not injective (kernel is non-trivial).

Explicit ML Relevance: A surjective output layer (e.g., in classification) should theoretically produce all possible outputs. However, in practice, neural networks often produce outputs in a narrow range (saturation, dead neurons), reducing effective surjectivity.

Endomorphism

Formal Definition: A linear map \(T: V \to V\) (from a space to itself) is an endomorphism. An endomorphism that is also an isomorphism (bijective) is called an automorphism.

Explicit Assumptions: Endomorphisms map a space to itself. The set of all endomorphisms of \(V\) forms an algebra (closed under addition, composition, and scalar multiplication).

Notation Discipline: Endomorphisms form the set \(\text{End}(V)\) or \(\text{Hom}(V, V)\). Automorphisms form the group \(\text{Aut}(V)\) or \(\text{GL}(V)\).

Usage and Interpretation: Endomorphisms include many important structures: projections, rotations, diagonalizable maps with eigenvalues. They are central to studying dynamical systems (iterating \(T\) to compute \(T^n\), the \(n\)-th power) and spectral theory.

Valid Example: \(T: \mathbb{R}^2 \to \mathbb{R}^2\), \(T(x, y) = (2x, y)\). Endomorphism (same space on both sides). Matrix: \(\begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix}\). Diagonalizable (main diagonal entries are eigenvalues), and the standard basis vectors are eigenvectors. Invertible (rank 2 = dimension), so also an automorphism.

Failure Case: Confusing endomorphism with other structures. Not every endomorphism is diagonalizable or invertible.

Explicit ML Relevance: Neural network layers are endomorphisms of hidden representation spaces: \(\mathbf{h}^{(\ell)} = \sigma(W^{(\ell)} \mathbf{h}^{(\ell-1)}\)) maps from one hidden space to another (or back to itself in recurrent networks). Understanding endomorphisms helps analyze how information flows through hidden layers.

Automorphism

Formal Definition: A linear map \(T: V \to V\) is an automorphism if it is an endomorphism and an isomorphism (bijective). Equivalently, \(T\) is invertible.

Explicit Assumptions: Automorphisms are bijective endomorphisms. They form a group under composition: the general linear group \(\text{GL}(V)\).

Notation Discipline: Automorphisms of \(V\) form the group \(\text{Aut}(V)\) or \(\text{GL}(V)\). For \(\mathbb{R}^n\), denoted \(\text{GL}(n, \mathbb{R})\) or \(\text{GL}_n(\mathbb{R})\) (the set of all invertible \(n \times n\) matrices).

Usage and Interpretation: Automorphisms are structure-preserving transformations that can be reversed. Basis changes are automorphisms (the change-of-basis matrix is invertible). Diagonalizable invertible endomorphisms are common examples.

Valid Example: \(T: \mathbb{R}^2 \to \mathbb{R}^2\), \(T(x, y) = (x+y, x-y)\). Invertible (rank 2 = dimension), so an automorphism. \(T^{-1}(a, b) = ((a+b)/2, (a-b)/2)\).

Failure Case: Confusing automorphisms with arbitrary invertible maps. All automorphisms are invertible, but not all invertible maps are automorphisms (those must be endomorphisms too).

Explicit ML Relevance: Automorphisms represent reversible transformations in data space. Normalizing flows (invertible neural networks) use automorphisms to transform complex data distributions into simple ones (e.g., Gaussian) while maintaining the ability to compute likelihoods exactly.

Projection Map

Formal Definition: A linear map \(P: V \to V\) is a projection (or idempotent) if \(P \circ P = P\) (meaning \(P(P(\mathbf{v})) = P(\mathbf{v})\) for all \(\mathbf{v}\)). Geometrically, a projection typically maps a space onto a subspace, and applying it twice has no additional effect (the image is already in the target subspace).

Explicit Assumptions: Projections are endomorphisms (domain = codomain). An orthogonal projection is one that is also self-adjoint (in inner product spaces): \(P^* = P\) (the adjoint, or matrix transpose, equals the matrix itself).

Notation Discipline: Write \(P_U\) for the projection onto subspace \(U\). The idempotent condition: \(P^2 = P\).

Usage and Interpretation: Projections decompose a space into a “target” subspace (the image of \(P\)) and an “annihilated” part (the kernel of \(P\)). They are useful for dimensionality reduction (project onto a low-dimensional subspace) and for orthogonal decomposition (decompose vectors into components along and perpendicular to a subspace).

Valid Example: \(P: \mathbb{R}^3 \to \mathbb{R}^3\), \(P(x, y, z) = (x, y, 0)\) (orthogonal projection onto the \(xy\)-plane). Check: \(P(P(x, y, z)) = P(x, y, 0) = (x, y, 0) = P(x, y, z)\) ✓. Idempotent. Matrix: \(\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}\). Image: \(xy\)-plane (2-dimensional). Kernel: \(z\)-axis (1-dimensional).

Failure Case: Confusing projection with arbitrary linear maps. Not every linear map is a projection; projections must satisfy \(P^2 = P\).

Explicit ML Relevance: PCA projection is a projection onto a low-dimensional subspace spanned by principal components. Attention mechanisms in transformers can be viewed as applying learned (soft) projections. Projections are fundamental to dimensionality reduction and feature extraction.

Operator Norm (Preview)

Formal Definition: The operator norm (or subordinate norm) of a linear map \(T: V \to W\) (with norms \(\|\cdot\|_V\) and \(\|\cdot\|_W\) on domain and codomain) is defined as \(\|T\| := \max_{\|\mathbf{v}\|_V = 1} \|T(\mathbf{v})\|_W\), the maximum factor by which \(T\) can scale a unit vector. For Euclidean norms and a matrix \(A \in \mathbb{R}^{m \times n}\), the operator norm is the largest singular value: \(\|A\|_2 = \sigma_{\max}(A)\).

Explicit Assumptions: Norms are defined on both domain and codomain. The operator norm is finite for linear maps between finite-dimensional spaces.

Notation Discipline: Write \(\|T\|\) or \(\|A\|_{\text{op}}\) or \(\|A\|_2\) (for spectral norm / largest singular value).

Usage and Interpretation: The operator norm measures how much a map can stretch vectors. A contraction has operator norm \(< 1\) (shrinks all vectors); a norm \(> 1\) can expand certain vectors. This is useful in linear system analysis (convergence of iterative methods) and optimization (step size selection in gradient descent).

Valid Example: \(T(x, y) = (2x, y)\) in \(\mathbb{R}^2\). Operator norm: a unit vector \((\cos\theta, \sin\theta)\) maps to \((2\cos\theta, \sin\theta)\), with norm \(\sqrt{4\cos^2\theta + \sin^2\theta} = \sqrt{3\cos^2\theta + 1}\). Maximized when \(\cos^2\theta = 1\) (at \(\theta = 0\)), giving norm \(\sqrt{3 + 0} = 2 = \sigma_{\max}\). ✓

Failure Case: Confusing operator norm with Frobenius norm or other matrix norms. Different norms capture different aspects (spectral norm focuses on worst-case stretching, Frobenius norm is a “total” norm of all entries).

Explicit ML Relevance: In neural networks, controlling the operator norm of weight matrices (via spectral normalization) stabilizes training and improves generalization by preventing exploding/vanishing gradients. The largest singular value determines the gain of each layer.

Matrix Rank

Formal Definition: The rank of a matrix \(A \in \mathbb{R}^{m \times n}\), denoted \(\text{rank}(A)\), is the dimension of its column space (equivalently, the dimension of its row space, which are always equal): \(\text{rank}(A) = \dim(\text{Col}(A)) = \dim(\text{Row}(A))\). Equivalently, it is the number of linearly independent rows or columns, which equals the number of nonzero singular values in the SVD.

Explicit Assumptions: Rank is defined for any matrix (any dimensions). Rank \(\leq \min(m, n)\) (at most the smaller dimension).

Notation Discipline: Write \(\text{rank}(A)\) or sometimes \(r(A)\).

Usage and Interpretation: Rank quantifies linear independence and information content. Full rank (rank = \(\min(m, n)\)) means maximum information; rank-deficiency (rank \(< \min(m, n)\)) means linear dependence and information loss.

Valid Example: \(A = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \end{pmatrix} \in \mathbb{R}^{2 \times 3}\). Columns: \((1, 2)^\top, (2, 4)^\top, (3, 6)^\top\). Second column = \(2 \times\) first; third column = \(3 \times\) first. Only the first column is independent. Rank = 1 (degenerate).

Failure Case: Confusing rank with the number of nonzero entries or the number of rows/columns. Rank is the dimension of the column (or row) space, not counting entries.

Explicit ML Relevance: In neural networks, a layer’s weight matrix rank determines the effective dimension of the transformation. Low-rank layers compress information; full-rank layers preserve information (if injective/surjective).

Matrix as Linear Transformation

Formal Definition: Any matrix \(A \in \mathbb{R}^{m \times n}\) can be viewed as a linear map \(T_A: \mathbb{R}^n \to \mathbb{R}^m\) defined by \(T_A(\mathbf{x}) = A\mathbf{x}\) (matrix-vector multiplication). This map is linear by properties of matrix multiplication (distributive over addition and scalar multiplication). Conversely, every linear map between finite-dimensional Euclidean spaces can be represented as a matrix (in the standard basis).

Explicit Assumptions: \(A\) has compatible dimensions: \(m\) rows and \(n\) columns. The input vector \(\mathbf{x}\) is a column vector in \(\mathbb{R}^n\).

Notation Discipline: Write \(T_A\) or sometimes just \(A\) to denote the map. The kernel of \(A\) is \(\text{Null}(A) = \{\mathbf{x} : A\mathbf{x} = \mathbf{0}\}\); the image is \(\text{Col}(A) = \{A\mathbf{x} : \mathbf{x} \in \mathbb{R}^n\}\).

Usage and Interpretation: This identification—matrices \(\leftrightarrow\) linear maps—is the bridge between abstract linear algebra (functions, spaces) and computation (arrays, matrix operations). It makes linear algebra practical and algorithmic.

Valid Example: \(A = \begin{pmatrix} 1 & 2 \\ 0 & 3 \end{pmatrix} \in \mathbb{R}^{2 \times 2}\) represents the linear map \(T_A: \mathbb{R}^2 \to \mathbb{R}^2\), \(T_A(x, y)^\top = (x + 2y, 3y)^\top\). Kernel: \((1 + 2y, 3y) = (0, 0)\) implies \(y = 0, x = 0\), so ker\(A = \{\mathbf{0}\}\) (injective). Image: apply to standard basis vectors—\(T_A(1, 0)^\top = (1, 0)^\top, T_A(0, 1)^\top = (2, 3)^\top\). These are independent, so image is all of \(\mathbb{R}^2\) (surjective). Full rank, invertible.

Failure Case: Forgetting that the identification requires a basis. Different bases yield different matrices representing the same geometric transformation.

Explicit ML Relevance: Every neural network weight matrix is a matrix-as-linear-transformation. Understanding how matrices encode maps—via ranks, kernels, images—directly translates to understanding model structure and behavior. This perspective is essential for diagnosing and designing neural networks.

Theorems

Kernel Is a Subspace

Formal Statement: Let \(T: V \to W\) be a linear map. Then \(\ker(T) \subseteq V\) is a vector subspace of \(V\).

Full Formal Proof:

We verify three conditions for a subspace: (1) contains zero, (2) closed under addition, (3) closed under scalar multiplication.

(1) Zero vector in kernel: \(T(\mathbf{0}_V) = \mathbf{0}_W\) (by linearity: \(T(\mathbf{0}) = T(0 \cdot \mathbf{v}) = 0 \cdot T(\mathbf{v}) = \mathbf{0}\)). Thus \(\mathbf{0}_V \in \ker(T)\). ✓

(2) Closed under addition: If \(\mathbf{u}, \mathbf{v} \in \ker(T)\), then \(T(\mathbf{u}) = \mathbf{0}\) and \(T(\mathbf{v}) = \mathbf{0}\). We have \(T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v}) = \mathbf{0} + \mathbf{0} = \mathbf{0}\) (by linearity). Thus \(\mathbf{u} + \mathbf{v} \in \ker(T)\). ✓

(3) Closed under scalar multiplication: If \(\mathbf{v} \in \ker(T)\) and \(c \in \mathbb{R}\), then \(T(c\mathbf{v}) = cT(\mathbf{v}) = c \cdot \mathbf{0} = \mathbf{0}\) (by linearity). Thus \(c\mathbf{v} \in \ker(T)\). ✓

All three conditions hold, so \(\ker(T)\) is a subspace. \(\square\)

Interpretation: The kernel is a “closed” structure under the vector space operations. This justifies calling its dimension the “nullity”—it is a legitimate subspace, not a scattered set.

Explicit ML Relevance: In regression with a rank-deficient design matrix, the kernel (null space of the design matrix) is a subspace—the set of all weight adjustments that don’t change predictions. Understanding it as a subspace is key to parameterizing the solution family.

Image Is a Subspace

Formal Statement: Let \(T: V \to W\) be a linear map. Then \(\text{im}(T) \subseteq W\) is a vector subspace of \(W\).

Full Formal Proof:

(1) Zero vector in image: \(T(\mathbf{0}_V) = \mathbf{0}_W \in \text{im}(T)\). ✓

(2) Closed under addition: If \(\mathbf{w}_1, \mathbf{w}_2 \in \text{im}(T)\), then \(\mathbf{w}_1 = T(\mathbf{v}_1)\) and \(\mathbf{w}_2 = T(\mathbf{v}_2)\) for some \(\mathbf{v}_1, \mathbf{v}_2 \in V\). Then \(\mathbf{w}_1 + \mathbf{w}_2 = T(\mathbf{v}_1) + T(\mathbf{v}_2) = T(\mathbf{v}_1 + \mathbf{v}_2) \in \text{im}(T)\) (by linearity). ✓

(3) Closed under scalar multiplication: If \(\mathbf{w} \in \text{im}(T)\) and \(c \in \mathbb{R}\), then \(\mathbf{w} = T(\mathbf{v})\) for some \(\mathbf{v} \in V\). Then \(c\mathbf{w} = c T(\mathbf{v}) = T(c\mathbf{v}) \in \text{im}(T)\). ✓

All conditions hold, so \(\text{im}(T)\) is a subspace. \(\square\)

Interpretation: The image is the set of “reachable outputs”—it has subspace structure, justifying the definition of rank as its dimension.

Explicit ML Relevance: In neural networks, the image of a weight matrix is the subspace of activations reachable by that layer. If the image is low-dimensional, the layer is a bottleneck—subsequent layers can only work with those low-dimensional outputs.

Rank-Nullity Theorem

Formal Statement: Let \(T: V \to W\) be a linear map, and assume \(V\) is finite-dimensional. Then: \[ \dim(V) = \dim(\ker(T)) + \dim(\text{im}(T)) = \text{nullity}(T) + \text{rank}(T) \]

Full Formal Proof:

Let \(k = \dim(\ker(T))\) (nullity) and \(r = \dim(\text{im}(T))\) (rank). We must show \(\dim(V) = k + r\).

Step 1: Choose a basis \(\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}\) for \(\ker(T)\) (the trivial basis \(\emptyset\) if \(k = 0\)).

Step 2: Extend this to a basis \(\{\mathbf{v}_1, \ldots, \mathbf{v}_k, \mathbf{u}_1, \ldots, \mathbf{u}_r\}\) for \(V\) (by the Basis Extension Theorem, Chapter 2). This extended basis has \(k + r\) vectors.

Step 3: We claim \(\{T(\mathbf{u}_1), \ldots, T(\mathbf{u}_r)\}\) is a basis for \(\text{im}(T)\).

Spanning: Any \(\mathbf{w} \in \text{im}(T)\) satisfies \(\mathbf{w} = T(\mathbf{v})\) for some \(\mathbf{v} \in V\). Write: \[ \mathbf{v} = \sum_{i=1}^k \alpha_i \mathbf{v}_i + \sum_{j=1}^r \beta_j \mathbf{u}_j \] Then: \[ \mathbf{w} = T(\mathbf{v}) = \sum_{i=1}^k \alpha_i T(\mathbf{v}_i) + \sum_{j=1}^r \beta_j T(\mathbf{u}_j) = \sum_{j=1}^r \beta_j T(\mathbf{u}_j) \] (since \(\mathbf{v}_i \in \ker(T)\) implies \(T(\mathbf{v}_i) = \mathbf{0}\)). Thus \(\{T(\mathbf{u}_1), \ldots, T(\mathbf{u}_r)\}\) spans \(\text{im}(T)\). ✓

Linear independence: If \(\sum_{j=1}^r \gamma_j T(\mathbf{u}_j) = \mathbf{0}\), then: \[ T\left( \sum_{j=1}^r \gamma_j \mathbf{u}_j \right) = \mathbf{0} \] So \(\sum_{j=1}^r \gamma_j \mathbf{u}_j \in \ker(T)\). But the kernel has basis \(\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}\), so: \[ \sum_{j=1}^r \gamma_j \mathbf{u}_j = \sum_{i=1}^k \alpha_i \mathbf{v}_i \] Rearranging: \[ \sum_{i=1}^k \alpha_i \mathbf{v}_i + \sum_{j=1}^r (-\gamma_j) \mathbf{u}_j = \mathbf{0} \] By linear independence of the full basis \(\{\mathbf{v}_1, \ldots, \mathbf{v}_k, \mathbf{u}_1, \ldots, \mathbf{u}_r\}\) for \(V\), all coefficients are zero, including \(\gamma_j = 0\) for all \(j\). Thus \(\{T(\mathbf{u}_1), \ldots, T(\mathbf{u}_r)\}\) is linearly independent. ✓

Conclusion: \(\{T(\mathbf{u}_1), \ldots, T(\mathbf{u}_r)\}\) is a basis for \(\text{im}(T)\) with \(r\) vectors, confirming \(\dim(\text{im}(T)) = r\). The extended basis for \(V\) has \(k + r\) vectors, so \(\dim(V) = k + r = \text{nullity}(T) + \text{rank}(T)\). \(\square\)

Interpretation: This theorem partitions the domain into two components: the kernel (annihilated directions) and a “complementary” subspace whose image spans the image space. It is fundamental—rank and nullity are not independent; together they equal the domain dimension.

Explicit ML Relevance: In underdetermined regression (\(d > n\) features vs. samples), rank-nullity implies the solution set is an affine space of dimension \(d - \text{rank}(X)\). Understanding this nullity dimension is key to characterizing the solution family and justifying regularization.

Matrix Representation Theorem

Formal Statement: Let \(T: V \to W\) be a linear map, let \(\mathcal{B} = \{\mathbf{b}_1, \ldots, \mathbf{b}_n\}\) be a basis for \(V\), and let \(\mathcal{C} = \{\mathbf{c}_1, \ldots, \mathbf{c}_m\}\) be a basis for \(W\). Then there exists a unique \(m \times n\) matrix \(A = [T]_{\mathcal{B}, \mathcal{C}}\) such that for any \(\mathbf{v} \in V\): \[ [T(\mathbf{v})]_\mathcal{C} = A [\mathbf{v}]_\mathcal{B} \] where \([\cdot]_\mathcal{B}\) and \([\cdot]_\mathcal{C}\) denote coordinate vectors with respect to the respective bases. The columns of \(A\) are: \[ A = [T]_{\mathcal{B}, \mathcal{C}} = \left[ [T(\mathbf{b}_1)]_\mathcal{C} \mid \cdots \mid [T(\mathbf{b}_n)]_\mathcal{C} \right] \]

Full Formal Proof:

Construction: Define \(A\) as stated (columns are images of basis vectors).

Correctness: For any \(\mathbf{v} = \sum_j c_j \mathbf{b}_j\) with \([\mathbf{v}]_\mathcal{B} = (c_1, \ldots, c_n)^\top\), we have: \[ T(\mathbf{v}) = T\left(\sum_j c_j \mathbf{b}_j\right) = \sum_j c_j T(\mathbf{b}_j) \] (by linearity). Expressing each \(T(\mathbf{b}_j)\) in the \(\mathcal{C}\) basis: \(T(\mathbf{b}_j) = \sum_i a_{ij} \mathbf{c}_i\) where \([T(\mathbf{b}_j)]_\mathcal{C} = (a_{1j}, \ldots, a_{mj})^\top\) (the \(j\)-th column of \(A\)). Then: \[ T(\mathbf{v}) = \sum_j c_j \sum_i a_{ij} \mathbf{c}_i = \sum_i \left( \sum_j a_{ij} c_j \right) \mathbf{c}_i \] So: \[ [T(\mathbf{v})]_\mathcal{C} = \left( \sum_j a_{1j} c_j, \ldots, \sum_j a_{mj} c_j \right)^\top = A [\mathbf{v}]_\mathcal{B} \] (by matrix-vector multiplication). ✓

Uniqueness: If \(A'\) also satisfies this property, evaluating at basis vectors \(\mathbf{b}_j\) shows \(A' [:, j] = [T(\mathbf{b}_j)]_\mathcal{C} = A[:, j]\) for each column, so \(A' = A\). ✓

Interpretation: This theorem justifies the use of matrices to represent linear maps. It makes the abstract map concrete and computational.

Explicit ML Relevance: Every neural network weight matrix is a matrix representation of a linear map (with respect to some choice of feature basis). Understanding this relationship clarifies how changing the basis (preprocessing, normalization) changes the weights.

Composition Corresponds to Matrix Multiplication

Formal Statement: Let \(T: V \to W\) and \(S: W \to Z\) be linear maps with matrix representations \(B = [T]_{\mathcal{B}, \mathcal{C}}\) (with respect to bases \(\mathcal{B}\) for \(V\), \(\mathcal{C}\) for \(W\)) and \(C = [S]_{\mathcal{C}, \mathcal{D}}\) (with respect to bases \(\mathcal{C}\) for \(W\), \(\mathcal{D}\) for \(Z\)). Then the composition \(S \circ T: V \to Z\) has matrix representation: \[ [S \circ T]_{\mathcal{B}, \mathcal{D}} = C B \] where \(CB\) is the standard matrix product.

Full Formal Proof:

By the Matrix Representation Theorem, for any \(\mathbf{v} \in V\): \[ [T(\mathbf{v})]_\mathcal{C} = B [\mathbf{v}]_\mathcal{B} \] \[ [S(T(\mathbf{v}))]_\mathcal{D} = C [T(\mathbf{v})]_\mathcal{C} = C (B [\mathbf{v}]_\mathcal{B}) = (CB) [\mathbf{v}]_\mathcal{B} \] (by associativity of matrix multiplication). Since \((S \circ T)(\mathbf{v}) = S(T(\mathbf{v}))\), we have: \[ [(S \circ T)(\mathbf{v})]_\mathcal{D} = (CB) [\mathbf{v}]_\mathcal{B} \] By the Matrix Representation Theorem’s uniqueness, \([S \circ T]_{\mathcal{B}, \mathcal{D}} = CB\). \(\square\)

Interpretation: Function composition (abstract) translates to matrix multiplication (concrete). This is the bridge enabling computational algebra.

Explicit ML Relevance: In neural networks, cascading layers corresponds to multiplying weight matrices: the composition of layer 1 and layer 2 is the product of their weights (ignoring nonlinearities and biases). Understanding rank composition (\(\text{rank}(CB) \leq \min(\text{rank}(C), \text{rank}(B))\)) explains why bottleneck layers limit downstream capacity.

Invertibility Characterization Theorem

Formal Statement: Let \(V\) and \(W\) be finite-dimensional vector spaces with \(\dim(V) = \dim(W) = n\), and let \(T: V \to W\) be a linear map. The following are equivalent: 1. \(T\) is invertible (bijective). 2. \(\ker(T) = \{\mathbf{0}\}\) (injective). 3. \(\text{im}(T) = W\) (surjective). 4. The matrix representation \(A = [T]_{\mathcal{B}, \mathcal{C}}\) is a square, invertible \(n \times n\) matrix.

Full Formal Proof:

We prove the equivalences in a cycle.

(1) \(\Rightarrow\) (2): If \(T\) is bijective, then \(T\) is injective, so \(\ker(T) = \{\mathbf{0}\}\). ✓

(2) \(\Rightarrow\) (1): Assume \(\ker(T) = \{\mathbf{0}\}\). By rank-nullity, \(\text{rank}(T) = \dim(V) - 0 = n = \dim(W)\), so \(\text{im}(T)\) is \(n\)-dimensional and \(\text{im}(T) \subseteq W\) has \(\dim(W) = n\). Thus \(\text{im}(T) = W\) (surjective). Combined with injectivity, \(T\) is bijective. ✓

(1) \(\Rightarrow\) (3): A bijection is surjective. ✓

(3) \(\Rightarrow\) (1): If \(\text{im}(T) = W\), then \(T\) is surjective. By rank-nullity, \(\text{rank}(T) = n\) implies \(\text{nullity}(T) = 0\), so \(\ker(T) = \{\mathbf{0}\}\) (injective). Combined with surjectivity, \(T\) is bijective. ✓

(1) \(\Rightarrow\) (4): If \(T\) is invertible, then there exists \(T^{-1}: W \to V\). Both have matrix representations: \(A = [T]_{\mathcal{B}, \mathcal{C}} \in \mathbb{R}^{n \times n}\) and \(A^{-1} = [T^{-1}]_{\mathcal{C}, \mathcal{B}} \in \mathbb{R}^{n \times n}\). Since \(T^{-1} \circ T = I_V\) and \(T \circ T^{-1} = I_W\), by Composition Theorem, \(A^{-1} A = I\) and \(A A^{-1} = I\) (both identity matrices). Thus \(A\) is invertible. ✓

(4) \(\Rightarrow\) (1): If \(A\) is invertible, let \(A' = A^{-1}\). Define a linear map \(T': W \to V\) by \([T']_{\mathcal{C}, \mathcal{B}} = A'\). By the Composition Theorem, \(T' \circ T\) has matrix representation \(A' A = I\), so \(T' \circ T = I_V\). Similarly, \(T \circ T' = I_W\). Thus \(T' = T^{-1}\), and \(T\) is invertible. ✓

All equivalences hold. \(\square\)

Interpretation: For finite-dimensional spaces of equal dimension, injectivity, surjectivity, and invertibility are equivalent (not true in infinite dimensions!). This simplifies reasoning about maps.

Explicit ML Relevance: A square weight matrix is invertible iff it has trivial kernel (no information loss) iff its rank is full. Singular weight matrices (rank-deficient) are non-invertible and indicate bottlenecks or redundancy.

Rank Characterization Theorem

Formal Statement: Let \(A \in \mathbb{R}^{m \times n}\) be a matrix. The following quantities are all equal: 1. The dimension of the column space: \(\dim(\text{Col}(A))\). 2. The dimension of the row space: \(\dim(\text{Row}(A))\). 3. The number of linearly independent columns. 4. The number of linearly independent rows. 5. The number of pivot columns in row echelon form. 6. The number of nonzero singular values in the SVD.

All are denoted \(\text{rank}(A)\).

Full Formal Proof (sketch, as this unifies multiple methods):

Columns & rows equal: Row operations preserve the row space. Reducing \(A\) to row echelon form \(R\) preserves the row space: \(\text{Row}(A) = \text{Row}(R)\). The nonzero rows of \(R\) are a basis for the row space. The number of nonzero rows in \(R\) equals the number of pivot columns (one per nonzero row in RREF). These pivot columns are linearly independent (by echelon structure), and the non-pivot columns are linear combinations of pivots. Thus, the number of linearly independent columns equals the number of pivot columns, which equals the number of nonzero rows in \(R\), which equals the dimension of the row space. Similarly, \(\dim(\text{Col}(A)) = \dim(\text{Row}(A))\). ✓

SVD characterization: In the SVD \(A = U \Sigma V^\top\), \(\Sigma\) is diagonal with singular values \(\sigma_1 \geq \cdots \geq \sigma_{\min(m,n)} \geq 0\). The rank is the number of nonzero singular values. The left singular vectors (columns of \(U\)) span the column space; those corresponding to nonzero singular values span the image, which has dimension equal to the number of nonzero singular values. ✓

All characterizations identify the same quantity—the rank. \(\square\)

Interpretation: Rank can be computed multiple ways (column space, row space, echelon form, SVD), each offering different insights.

Explicit ML Relevance: Computing rank via different methods (row reduction, SVD, singular values) provides computational flexibility and numerical stability insights. SVD is numerically more stable for ill-conditioned matrices.

Dimension of Image Theorem

Formal Statement: Let \(T: V \to W\) be a linear map with \(V\) finite-dimensional. Then: \[ \dim(\text{im}(T)) = \text{rank}(T) = \dim(V) - \text{nullity}(T) \]

(This is restating rank-nullity; listed separately as it emphasizes the image dimension.)

Full Formal Proof:

By the Rank-Nullity Theorem, \(\dim(V) = \text{nullity}(T) + \text{rank}(T)\), so: \[ \text{rank}(T) = \dim(V) - \text{nullity}(T) \]

By definition of rank, \(\text{rank}(T) = \dim(\text{im}(T))\). Thus: \[ \dim(\text{im}(T)) = \dim(V) - \text{nullity}(T) \] \(\square\)

Interpretation: The image dimension (rank) is what remains after accounting for the kernel (nullity). This is a restatement, but it emphasizes the image dimension.

Explicit ML Relevance: In neural network layers, the image dimension (rank of weights) bounds the effective output dimensionality. Low-rank layers output low-dimensional features.

Change-of-Basis Similarity Theorem (Preview)

Formal Statement: Let \(T: V \to V\) be an endomorphism, and let \(\mathcal{B}\) and \(\mathcal{B}'\) be two bases for \(V\). Let \(A = [T]_\mathcal{B}\) be the matrix representation of \(T\) in basis \(\mathcal{B}\), and let \(A' = [T]_{\mathcal{B}'}\) be the representation in the new basis. Then: \[ A' = P^{-1} A P \] where \(P = [I]_{\mathcal{B}', \mathcal{B}}\) is the change-of-basis matrix from \(\mathcal{B}'\) to \(\mathcal{B}\).

Full Formal Proof:

For any \(\mathbf{v} \in V\): \[ [\mathbf{v}]_\mathcal{B} = P [\mathbf{v}]_{\mathcal{B}'} \] (by definition of change-of-basis matrix). Applying \(T\): \[ [T(\mathbf{v})]_\mathcal{B} = A [\mathbf{v}]_\mathcal{B} = A (P [\mathbf{v}]_{\mathcal{B}'}) = (AP) [\mathbf{v}]_{\mathcal{B}'} \]

Expressing the result in the \(\mathcal{B}'\) basis: \[ [T(\mathbf{v})]_{\mathcal{B}'} = P^{-1} [T(\mathbf{v})]_\mathcal{B} = P^{-1} (AP) [\mathbf{v}]_{\mathcal{B}'} \]

But also, by definition of \(A'\): \[ [T(\mathbf{v})]_{\mathcal{B}'} = A' [\mathbf{v}]_{\mathcal{B}'} \]

Comparing, \(A' = P^{-1} A P\). \(\square\)

Interpretation: Similarity transformations preserve essential properties of the linear map (eigenvalues, determinant, trace, rank) while changing the numerical matrix entries. The goal of finding “nice” bases is to find an \(A'\) with special structure (diagonal, upper-triangular, etc.).

Explicit ML Relevance: This theorem is the mathematical foundation for diagonalization (Chapter 5, detailing when we can find a basis where \(A'\) is diagonal) and other canonical forms. It’s key to understanding why different feature bases lead to different weights but the same learned function.

Injective–Surjective Equivalence in Finite Dimensions

Formal Statement: Let \(V\) and \(W\) be finite-dimensional vector spaces with \(\dim(V) = \dim(W) = n\), and let \(T: V \to W\) be a linear map. Then: \[ T \text{ is injective} \quad \Leftrightarrow \quad T \text{ is surjective} \]

Both conditions are equivalent to \(T\) being bijective (invertible).

Full Formal Proof:

Suppose \(T\) is injective. Then \(\ker(T) = \{\mathbf{0}\}\), so \(\text{nullity}(T) = 0\). By rank-nullity: \[ \text{rank}(T) = \dim(V) - 0 = n = \dim(W) \]

So \(\dim(\text{im}(T)) = n = \dim(W)\), and since \(\text{im}(T) \subseteq W\) (a subspace), we have \(\text{im}(T) = W\). Thus \(T\) is surjective. ✓

Conversely, if \(T\) is surjective, then \(\text{im}(T) = W\), so \(\text{rank}(T) = n = \dim(V)\). By rank-nullity: \[ \text{nullity}(T) = n - n = 0 \]

So \(\ker(T) = \{\mathbf{0}\}\), and \(T\) is injective. ✓

Thus injectivity and surjectivity are equivalent, and both imply bijectivity (invertibility). \(\square\)

Interpretation: In finite dimensions of equal size, checking one property (injectivity via kernel) is sufficient to guarantee invertibility. This does not hold in infinite dimensions.

Explicit ML Relevance: For square weight matrices in neural networks, checking rank (= dimension) determines invertibility. A rank-deficient \(n \times n\) matrix is singular and cannot be learned as an invertible transformation.

Worked Examples

Example 1: Matrix as Linear Map on ℝⁿ

Setup: Consider the real vector space \(\mathbb{R}^2\) and the matrix \(A = \begin{pmatrix} 2 & 1 \\ 0 & 3 \end{pmatrix}\). We want to understand \(A\) as a linear map \(T_A: \mathbb{R}^2 \to \mathbb{R}^2\) defined by \(T_A(\mathbf{x}) = A\mathbf{x}\). This is a concrete instantiation of the definition: a matrix is a linear transformation acting on vectors.

Reasoning: The matrix \(A\) acts on a vector \(\mathbf{v} = (v_1, v_2)^\top\) by multiplication: \[ A\mathbf{v} = \begin{pmatrix} 2 & 1 \\ 0 & 3 \end{pmatrix} \begin{pmatrix} v_1 \\ v_2 \end{pmatrix} = \begin{pmatrix} 2v_1 + v_2 \\ 3v_2 \end{pmatrix} \]

This transformation does two things: scales the first coordinate by 2 and adds a contribution from the second coordinate (the “1” term), while scaling the second coordinate by 3 and ignoring the first. The matrix encodes how the standard basis vectors transform: \(T_A(1, 0)^\top = (2, 0)^\top\) (first column) and \(T_A(0, 1)^\top = (1, 3)^\top\) (second column). These are exactly the columns of \(A\), confirming the relationship: matrix columns = images of basis vectors.

The transformation is linear. Verify: \(T_A(c\mathbf{v}) = A(c\mathbf{v}) = c(A\mathbf{v}) = cT_A(\mathbf{v})\) and \(T_A(\mathbf{u} + \mathbf{v}) = A(\mathbf{u} + \mathbf{v}) = A\mathbf{u} + A\mathbf{v} = T_A(\mathbf{u}) + T_A(\mathbf{v})\) (by properties of matrix multiplication). ✓

Interpretation: The matrix \(A\) compresses and rotates the plane. The image of the unit circle under \(A\) is an ellipse (determined by the singular values). The first column \((2, 0)^\top\) stretches the standard basis vector \(\mathbf{e}_1\) by a factor of 2; the second column \((1, 3)^\top\) moves \(\mathbf{e}_2\) up (adding 1 to the first coordinate) and stretches by 3. The rank of \(A\) is 2 (both columns are linearly independent), so the transformation is invertible and preserves all directions (no compression). The kernel is trivial: \(A\mathbf{x} = \mathbf{0}\) implies \(2x_1 + x_2 = 0\) and \(3x_2 = 0\), so \(x_2 = 0\) and \(x_1 = 0\).

Common Misconceptions: A frequent error is to view the matrix as a “static object” rather than a functional transformation. Students sometimes confuse the matrix entries with the transformation geometry—the numerical values (2, 1, 0, 3) are coordinates of the transformation in the standard basis, not inherent to the map. If we change the basis, the numerical values change even though the geometric effect is identical. Another misconception is to assume the matrix “preserves direction”—it doesn’t (unless it’s a rotation or scaling). The vector \((1, 0)^\top\) maps to \((2, 0)^\top\) (same direction), but \((0, 1)^\top\) maps to \((1, 3)^\top\) (different direction). The linearity property (preservation of linear combinations) does not mean individual vectors are scaled uniformly.

What-If Scenarios: What if we change the matrix to \(A' = \begin{pmatrix} 2 & 1 \\ 0 & 0 \end{pmatrix}\) (the last row becomes zero)? The image becomes the span of \((2, 0)^\top\) and \((1, 0)^\top\), which is just the \(x\)-axis (1-dimensional). Rank drops to 1. The second dimension is annihilated (all \(y\)-component of inputs is lost). The kernel is now non-trivial: any vector with \(v_2 \neq 0\) and \(v_1 = -v_2/2\) maps to zero (e.g., \((-1, 2)^\top \to (0, 0)^\top\)). This illustrates how rank deficiency creates information loss.

What if \(A = \begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix}\) (diagonal)? The transformation is a scaling: the first coordinate is scaled by 2, the second by 3. This is much simpler geometrically—the basis vectors remain eigenvectors (directions unchanged), only stretched. Why is the diagonal case preferred? Because it reveals the “principal” scaling directions. The change-of-basis theorem says that every invertible matrix can be put into a simpler diagonal form (if diagonalizable); finding that basis is a primary goal (Chapter 5, on eigenvalues).

Explicit ML Relevance: In neural networks, every weight matrix in a fully connected layer is a matrix-as-linear-map. Given an input vector \(\mathbf{x}\) of features, the layer computes \(\mathbf{h} = W\mathbf{x} + \mathbf{b}\) where \(W\) is the weight matrix and \(\mathbf{b}\) is the bias (ignoring the bias and nonlinearity for now, the core \(W\mathbf{x}\) is a linear transformation). The columns of \(W\) tell us how each input feature affects the entire output vector—a dense interconnection. If rank\(W < \min(m, n)\), the layer is a bottleneck (compresses information). Understanding the matrix as a transformation is key to diagnosing when layers are too narrow, when there’s redundancy, or when regularization is needed.

Example 2: Computing Kernel of a Matrix

Setup: Consider the matrix \(A = \begin{pmatrix} 1 & 2 & 1 \\ 1 & 2 & 1 \\ 0 & 0 & 0 \end{pmatrix} \in \mathbb{R}^{3 \times 3}\). We want to find the kernel (null space) of \(A\), which is the set of all vectors \(\mathbf{x} \in \mathbb{R}^3\) such that \(A\mathbf{x} = \mathbf{0}\).

Reasoning: Solving \(A\mathbf{x} = \mathbf{0}\) means finding all column vectors \(\mathbf{x} = (x_1, x_2, x_3)^\top\) such that: \[ \begin{pmatrix} 1 & 2 & 1 \\ 1 & 2 & 1 \\ 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix} \]

The equations are: \(x_1 + 2x_2 + x_3 = 0\) (row 1), \(x_1 + 2x_2 + x_3 = 0\) (row 2), \(0 = 0\) (row 3). The first two equations are identical, so they give only one constraint: \(x_1 + 2x_2 + x_3 = 0\), or \(x_1 = -2x_2 - x_3\). The variables \(x_2\) and \(x_3\) are free. So: \[ \mathbf{x} = \begin{pmatrix} -2x_2 - x_3 \\ x_2 \\ x_3 \end{pmatrix} = x_2 \begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix} + x_3 \begin{pmatrix} -1 \\ 0 \\ 1 \end{pmatrix} \]

The kernel is the span of two vectors: \(\ker(A) = \text{span}\left( \begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix}, \begin{pmatrix} -1 \\ 0 \\ 1 \end{pmatrix} \right)\). The dimension is 2 (the nullity). By rank-nullity, rank\(A\) = 3 - 2 = 1 (confirmed: only one independent row in \(A\)).

Interpretation: The kernel is a 2-dimensional subspace (a plane through the origin) in \(\mathbb{R}^3\). Any vector in this plane maps to zero under \(A\). This represents information loss: two distinct inputs can produce the same output (the zero vector) if they differ by a vector in the kernel. The rank-deficiency is geometric: the matrix “flattens” 2 out of 3 dimensions. The non-kernel subspace (orthogonal complement, if we use inner products) is 1-dimensional—the image is indeed rank 1, just the span of the (repeated) first row.

Common Misconceptions: Students often assume the kernel is always trivial (only the zero vector). This is false for rank-deficient matrices. Another error is confusing the kernel (which is independent of the codomain) with the “missing” part of the codomain (the orthogonal complement of the image). The kernel is in the domain; the complement of the image is in the codomain. A third mistake is parametrizing the kernel incorrectly. The free variables \(x_2, x_3\) must span the kernel space proportionally; setting them as the full-rank basis vectors (as above) ensures linear independence.

What-If Scenarios: What if \(A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}\) (the identity)? The kernel is trivial: \(\mathbf{x} = \mathbf{0}\) is the only solution. The map is injective. What if \(A = \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}\)? The rows are identical, so rank is 1. The kernel is 1-dimensional: any vector \((a, -a)^\top\) satisfies \(a - a = 0\). The kernel forms a 1-dimensional line. What if \(A = \begin{pmatrix} 0 & 0 \\ 0 & 0 \end{pmatrix}\)? The kernel is the entire domain (\(\mathbb{R}^n\))—every vector maps to zero (complete information loss). These extremes illustrate the spectrum: from trivial kernel (full-rank, injective) to full-domain kernel (zero-rank, constant map).

Explicit ML Relevance: In regression with a design matrix \(X \in \mathbb{R}^{n \times d}\) (n samples, d features), the kernel of \(X\) parameterizes non-unique solutions to \(X\mathbf{w} = \mathbf{y}\). If the kernel is non-trivial (rank-deficient design matrix), infinitely many weight vectors \(\mathbf{w}\) fit the data equally well—they differ by vectors in the kernel. Regularization (ridge regression, LASSO) selects a specific solution by penalizing the magnitude or structure of \(\mathbf{w}\), breaking the degeneracy. Understanding the kernel is crucial to diagnosing and resolving multicollinearity (when features are linearly dependent, the kernel contains directions of redundant feature combinations).

Example 3: Image and Column Space

Setup: Given the matrix \(B = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 5 \\ 0 & 0 & -1 \end{pmatrix} \in \mathbb{R}^{3 \times 3}\), we want to find the image (range) of the linear map \(T_B: \mathbb{R}^3 \to \mathbb{R}^3\), i.e., the set of all outputs \(B\mathbf{x}\) for \(\mathbf{x} \in \mathbb{R}^3\).

Reasoning: The image is the column space of \(B\), the span of the column vectors: \[ \text{im}(B) = \text{Col}(B) = \text{span}\left( \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}, \begin{pmatrix} 2 \\ 4 \\ 0 \end{pmatrix}, \begin{pmatrix} 3 \\ 5 \\ -1 \end{pmatrix} \right) \]

The second column is \(2 \times\) the first, so it adds no new direction. The first and third columns are linearly independent (no scalar multiple relationship; the first has a zero in position 3, the third does not). So a basis for the image is: \[ \text{im}(B) = \text{span}\left( \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}, \begin{pmatrix} 3 \\ 5 \\ -1 \end{pmatrix} \right) \]

Rank\(B\) = 2. Any output vector \(\mathbf{y} = B\mathbf{x}\) is a linear combination of these two columns. The image is a 2-dimensional subspace (a plane) in \(\mathbb{R}^3\), not the entire space. For instance, the vector \(\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}\) is NOT in the image (it would require a linear combination of columns to have a nonzero \(y_1\) unless the coefficients balance exactly, which doesn’t happen here—the first column’s first entry is 1, the third column’s first entry is 3; any combination gives \(1 \cdot c_1 + 3 \cdot c_3\) for some \(c_1, c_3\)). Let’s verify: if \(\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix} = c_1 \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix} + c_3 \begin{pmatrix} 3 \\ 5 \\ -1 \end{pmatrix}\), then \(c_1 + 3c_3 = 0, 2c_1 + 5c_3 = 0, -c_3 = 1\). From the third equation, \(c_3 = -1\), so \(c_1 = 3\) and \(2(3) + 5(-1) = 6 - 5 = 1 \neq 0\). Contradiction—vector \(\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}\) is outside the image.

Interpretation: The image is the reachable output space. Vectors outside the image cannot be produced by the linear map (no matter what input we choose). The surjectivity of a map is determined by whether the image fills the entire codomain. Here, the map is not surjective (image is 2-dimensional, codomain is 3-dimensional). The “missing” direction forms the orthogonal complement of the image (the left null space of \(B\), or equivalently, the null space of \(B^\top\)).

Common Misconceptions: A widespread error is assuming every linear map is surjective—it’s not. Especially for non-square matrices, the image is often strictly smaller than the codomain. Another mistake is confusing column space with the matrix itself; they’re related but different concepts. The column space is determined by which columns are linearly independent, not by the matrix’s numerical entries. A third misconception involves computing a span: students sometimes list all columns as “spanning the image” without recognizing linear dependence (the second column is redundant here).

What-If Scenarios: What if columns were independent? Suppose \(B' = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 5 & 5 \\ 0 & 0 & 1 \end{pmatrix}\) (change the second and third entries of column 2). Then all three columns might be independent, rank = 3, and the image is the entire \(\mathbb{R}^3\) (surjective). What if we add a fourth column \(\begin{pmatrix} 1 \\ 3 \\ 0 \end{pmatrix}\)? The image doesn’t change—it’s still rank 2 if the first three columns remain linearly dependent in the same way. The image dimension is bounded by \(\min(\text{number of columns}, \text{number of rows})\).

What if the matrix is tall (more rows than columns, e.g., 4 rows, 2 columns)? The image is at most 2-dimensional; it cannot fill a 4-dimensional codomain. Adding rows (more constraints) can only decrease or maintain the image dimension. This is why regression with many more samples than features often has no solution (the data \(\mathbf{y}\) is outside the image of the design matrix).

Explicit ML Relevance: In autoencoders, the encoder maps high-dimensional data \(\mathbf{x} \in \mathbb{R}^d\) to a bottleneck representation \(\mathbf{z} = f_{\text{enc}}(\mathbf{x}) \in \mathbb{R}^k\) where \(k < d\). The image of the encoder weight matrix is at most \(k\)-dimensional (if the weights have rank \(k\)). The decoder then tries to reconstruct \(\hat{\mathbf{x}} = f_{\text{dec}}(\mathbf{z})\) from this compressed representation. Information outside the image of the encoder cannot be preserved. Understanding the rank and image of weight matrices is essential for designing bottleneck architectures and assessing whether the learned representation has sufficient capacity.

Example 4: Rank-Nullity in Linear Regression

Setup: In linear regression, we have a design matrix \(X \in \mathbb{R}^{n \times d}\) (n samples, d features) and a target vector \(\mathbf{y} \in \mathbb{R}^n\) (n responses). The regression problem is to find \(\mathbf{w} \in \mathbb{R}^d\) such that \(X\mathbf{w} = \mathbf{y}\) (or as close as possible if exact solution doesn’t exist). We examine rank-nullity in this context.

Reasoning: The linear system \(X\mathbf{w} = \mathbf{y}\) has a solution iff \(\mathbf{y}\) is in the image of \(X\) (as a linear map from \(\mathbb{R}^d\) to \(\mathbb{R}^n\)). Assuming \(\mathbf{y}\) is in the image (or we’re finding the least-squares approximation, which always exists), the solution set is an affine subspace of dimension equal to the nullity of \(X\).

Let \(r = \text{rank}(X)\). Then \(\text{nullity}(X) = d - r\). If \(r = d\) (full column rank), the nullity is 0, and the solution \(\mathbf{w}\) is unique (if it exists). If \(r < d\) (rank-deficient), the nullity is \(d - r > 0\), and the solution set is an affine subspace of dimension \(d - r\) (infinitely many solutions forming a parallel translate of the kernel).

Example: Suppose \(X = \begin{pmatrix} 1 & 2 \\ 1 & 2 \\ 2 & 4 \end{pmatrix} \in \mathbb{R}^{3 \times 2}\) (3 samples, 2 features). All columns are scalar multiples, so rank = 1. Nullity = \(2 - 1 = 1\). The kernel is 1-dimensional: any \((w_1, w_2)^\top\) with \(w_1 + 2w_2 = 0\) maps to zero. So if \(\mathbf{w}_0 = (-2, 1)^\top\) is one solution to \(X\mathbf{w} = \mathbf{y}\), the complete solution set is \(\{ (-2, 1)^\top + t(-2, 1)^\top : t \in \mathbb{R} \} = \{ (-2-2t, 1+t)^\top \}\) (a 1-dimensional line). Multiple weights fit the same data perfectly, creating non-identifiability.

Interpretation: Rank-nullity in regression reveals the structure of the solution set. Full rank (\(r = d\)) means a unique solution (or no solution if \(\mathbf{y}\) is outside the image). Rank-deficiency means infinitely many solutions, forming a subspace of dimension equal to the nullity. This non-uniqueness is problematic: which solution is “correct”? Regularization (ridge regression, LASSO, elastic net) selects a specific solution by penalizing the norm or the sparsity of \(\mathbf{w}\), effectively choosing a point in the solution subspace based on prior assumptions about the weights (small norm, sparsity, etc.).

Common Misconceptions: A typical error is assuming regression always has a unique solution. This is only true for full-rank design matrices. Another is treating the nullity and rank as unrelated; they’re complementary (sum to the domain dimension). A third misconception involves over-interpreting the weights in rank-deficient settings: if the kernel is non-trivial, the weights are not uniquely identifiable from the data alone, so interpreting them as “feature importance” is misleading without additional assumptions.

What-If Scenarios: What if we add a regularization term? Ridge regression solves \(\min_\mathbf{w} \|X\mathbf{w} - \mathbf{y}\|^2 + \lambda \|\mathbf{w}\|^2\). Even for rank-deficient \(X\), the regularization term breaks the degeneracy, yielding a unique solution (the minimum-norm solution to \(X\mathbf{w} = \mathbf{y}\) when \(\lambda \to 0\)). What if we have more features than samples (\(d > n\))? The rank of \(X\) is at most \(n\), so \(\text{nullity}(X) \geq d - n > 0\). The kernel is always non-trivial (unless the rows are all full-rank, which requires very specific data). The solution set is always infinite-dimensional (before regularization).

Explicit ML Relevance: In modern machine learning, high-dimensional data (\(d \gg n\)) is common. Rank-nullity ensures the kernel is non-trivial, so unregularized regression is ill-posed. Ridge regression or early stopping in gradient descent effectively regularizes the solution. Understanding rank and nullity guides the choice of regularization strength: stronger regularization pushes toward a smaller (lower-norm) solution in the solution subspace. This transparency is valuable for model interpretation and debugging overfitting.

Example 5: Change of Basis for a Linear Map

Setup: Consider the linear map \(T: \mathbb{R}^2 \to \mathbb{R}^2\) defined by \(T(x, y) = (2x, y)\) (scaling the first coordinate by 2). In the standard basis \(\mathcal{E} = \{ (1, 0)^\top, (0, 1)^\top \}\), the matrix representation is: \[ A = [T]_\mathcal{E} = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \]

Now, consider the basis \(\mathcal{B} = \{ (1, 1)^\top, (1, -1)^\top \}\). We want to find the matrix representation of \(T\) in basis \(\mathcal{B}\).

Reasoning: The change-of-basis formula is \(A' = P^{-1} A P\), where \(P = [I]_{\mathcal{B}, \mathcal{E}}\) is the change-of-basis matrix from basis \(\mathcal{B}\) to basis \(\mathcal{E}\). The matrix \(P\) has columns equal to the coordinates of the \(\mathcal{B}\) basis vectors expressed in the standard basis (already in standard basis, so): \[ P = \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} \]

Compute \(P^{-1}\): \[ \det(P) = 1(-1) - (1)(1) = -2, \quad P^{-1} = \frac{1}{-2} \begin{pmatrix} -1 & -1 \\ -1 & 1 \end{pmatrix} = \begin{pmatrix} 1/2 & 1/2 \\ 1/2 & -1/2 \end{pmatrix} \]

Then: \[ A' = P^{-1} A P = \begin{pmatrix} 1/2 & 1/2 \\ 1/2 & -1/2 \end{pmatrix} \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} \]

First, \(AP = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 2 & 2 \\ 1 & -1 \end{pmatrix}\).

Then, \(A' = \begin{pmatrix} 1/2 & 1/2 \\ 1/2 & -1/2 \end{pmatrix} \begin{pmatrix} 2 & 2 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 1/2 \cdot 2 + 1/2 \cdot 1 & 1/2 \cdot 2 + 1/2 \cdot (-1) \\ 1/2 \cdot 2 - 1/2 \cdot 1 & 1/2 \cdot 2 - 1/2 \cdot (-1) \end{pmatrix} = \begin{pmatrix} 3/2 & 1/2 \\ 1/2 & 3/2 \end{pmatrix}\).

The matrix in the \(\mathcal{B}\) basis is \(A' = \begin{pmatrix} 3/2 & 1/2 \\ 1/2 & 3/2 \end{pmatrix}\), which looks different numerically but represents the same geometric transformation.

Interpretation: The matrix entries change, but the transformation is identical. The matrix \(A'\) in basis \(\mathcal{B}\) is no longer diagonal (unlike \(A\) in the standard basis), but it is symmetric (a hint that the transformation has special structure in certain bases, related to eigenvalues and eigenvectors—Chapter 5). The invariant properties (rank = 2, determinant = 2, trace = 3/2 + 3/2 = 3) are preserved. This illustrates the power and limitation of matrices: they depend on basis choice, so the same map can look simple (diagonal \(A\)) or complex (\(A'\)) depending on the basis. The goal of diagonalization is to find a basis where \(A'\) is diagonal, revealing the simplest form.

Common Misconceptions: Students often think the change-of-basis matrix \(P\) should be the basis vectors as rows, not columns. The formula \(A' = P^{-1} A P\) is easy to misremember (forward vs. inverse). A common error is computing \(P\) as the basis vectors themselves without recognizing that \(P\) columns should be the coordinates of \(\mathcal{B}\) vectors in the original basis. Another mistake is applying the formula for a non-endomorphism (a map from one space to another with different bases); the formula is specific to endomorphisms.

What-If Scenarios: What if we choose \(\mathcal{B}\) such that the basis vectors are eigenvectors of \(A\)? Then \(A'\) would be diagonal. For \(A = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix}\), eigenvalues are 2 and 1 with eigenvectors \((1, 0)^\top\) and \((0, 1)^\top\)—these are already the standard basis vectors, so \(A\) is already diagonal! If we choose a non-eigenvector basis, \(A'\) is more complex. This is why finding eigenvectors (Chapter 5) is such a major goal: it leads to diagonal representations.

What if \(A\) has repeated eigenvalues or complex structure? Then no basis over \(\mathbb{R}\) will diagonalize \(A\) (Jordan normal form, Chapter 5, handles this). The change-of-basis idea still applies, but the “simple” form is more subtle.

Explicit ML Relevance: In neural networks, the choice of basis corresponds to the choice of input features. Normalizing features (centering, scaling) is a change of basis. Whitening (decorrelating features via PCA) is another change of basis. The weight matrix \(W\) depends on the feature basis: if we preprocess features (change basis), the weights must be recomputed. Transfer learning leverages this: a model trained on one feature space can be adapted to another by retraining the weights—the change-of-basis effect is handled by gradient descent. Understanding basis changes helps explain why feature engineering and preprocessing matter.

Example 6: Composition of Linear Layers

Setup: Consider two linear maps: \(T: \mathbb{R}^3 \to \mathbb{R}^2\), \(T(\mathbf{x}) = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix} \mathbf{x}\) (matrix \(B = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}\)), and \(S: \mathbb{R}^2 \to \mathbb{R}^2\), \(S(\mathbf{y}) = \begin{pmatrix} 2 & 1 \\ 0 & 1 \end{pmatrix} \mathbf{y}\) (matrix \(C = \begin{pmatrix} 2 & 1 \\ 0 & 1 \end{pmatrix}\)). We compose them: \((S \circ T)(\mathbf{x}) = S(T(\mathbf{x}))\), which is a map from \(\mathbb{R}^3\) to \(\mathbb{R}^2\).

Reasoning: The matrix representation of the composition is the product \(CB\) (the order reverses in notation but matrix multiplication is function composition): \[ CB = \begin{pmatrix} 2 & 1 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix} = \begin{pmatrix} 2 \cdot 1 + 1 \cdot 0 & 2 \cdot 0 + 1 \cdot 1 & 2 \cdot 1 + 1 \cdot 0 \\ 0 \cdot 1 + 1 \cdot 0 & 0 \cdot 0 + 1 \cdot 1 & 0 \cdot 1 + 1 \cdot 0 \end{pmatrix} = \begin{pmatrix} 2 & 1 & 2 \\ 0 & 1 & 0 \end{pmatrix} \]

So \((S \circ T)(\mathbf{x}) = \begin{pmatrix} 2 & 1 & 2 \\ 0 & 1 & 0 \end{pmatrix} \mathbf{x}\). Let’s verify: for \(\mathbf{x} = (1, 2, 3)^\top\), \(T(\mathbf{x}) = \begin{pmatrix} 1 + 3 \\ 2 + 0 \end{pmatrix} = \begin{pmatrix} 4 \\ 2 \end{pmatrix}\). Then \(S(\begin{pmatrix} 4 \\ 2 \end{pmatrix}) = \begin{pmatrix} 2 \cdot 4 + 1 \cdot 2 \\ 0 \cdot 4 + 1 \cdot 2 \end{pmatrix} = \begin{pmatrix} 10 \\ 2 \end{pmatrix}\). Direct computation: \((CB) \mathbf{x} = \begin{pmatrix} 2 & 1 & 2 \\ 0 & 1 & 0 \end{pmatrix} \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix} = \begin{pmatrix} 2 + 2 + 6 \\ 0 + 2 + 0 \end{pmatrix} = \begin{pmatrix} 10 \\ 2 \end{pmatrix}\) ✓.

The composition has rank at most \(\min(\text{rank}(C), \text{rank}(B)) = \min(2, 2) = 2\). Here, both \(C\) and \(B\) are rank 2, so \(CB\) can also be rank 2 (or less, if multiplication causes alignment). Computing rank\(CB\): the columns of \(CB\) are \((2, 0)^\top\) and \((1, 1)^\top\), which are linearly independent, so rank = 2. This means no information is lost in the composition (the first-stage compression \(T: \mathbb{R}^3 \to \mathbb{R}^2\) reduces \(\mathbb{R}^3\) to a 2D space, and the second stage \(S: \mathbb{R}^2 \to \mathbb{R}^2\) preserves those 2 dimensions).

Interpretation: Composition is how we layer transformations. In deep networks, each hidden layer is a composition of previous layers: layer 2 acts on the output of layer 1, etc. The composition formula shows that depth enables rich expressivity (composition of nonlinear maps), but each linear layer can compress (if rank-deficient). A “bottleneck” layer (low rank) reduces dimensionality, potentially losing information that cannot be recovered by subsequent layers. Understanding rank composition is key to network architecture design.

Common Misconceptions: A frequent error is thinking the order of composition doesn’t matter (non-commutativity). \(S \circ T \neq T \circ S\) in general—they have different dimensions (one maps from \(V\) to \(Z\), the other would need compatible spaces). Another mistake is computing \(CB\) backward (using \(BC\) instead of \(CB\) in the order \(S \circ T\)). The mnemonic: “compose right-to-left” (apply \(T\) first, then \(S\)), but “multiply left-to-right” (matrix product is written with factors left-to-right), so the matrix for \(S \circ T\) is \(C \cdot B\).

What-If Scenarios: What if \(T\) is rank-deficient? Suppose \(B = \begin{pmatrix} 1 & 1 \\ 0 & 0 \\ 0 & 0 \end{pmatrix}\) (rank 1, compresses \(\mathbb{R}^2 \to \mathbb{R}^3\)). Then \(CB = \begin{pmatrix} 2 & 1 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 0 & 0 \\ 0 & 0 \end{pmatrix}\) is undefined (incompatible dimensions). OK, let me fix: suppose \(B = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 0 & 0 \end{pmatrix} \in \mathbb{R}^{2 \times 3}\) (rank 1, from \(\mathbb{R}^3\) to \(\mathbb{R}^2\)). The composition \(CB = \begin{pmatrix} 2 & 1 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 & 1 \\ 0 & 0 & 0 \end{pmatrix} = \begin{pmatrix} 2 & 2 & 2 \\ 0 & 0 & 0 \end{pmatrix}\) (rank 1). The low rank in \(B\) “propagates”: the full composition is rank 1 (completely flat in the second dimension). Information is compressed, creating a bottleneck. A later layer cannot recover the lost dimension.

Explicit ML Relevance: In neural networks, the output of layer \(\ell\) becomes the input to layer \(\ell+1\). Compositions of layers determine the overall function learnable by the network. If a hidden layer has low rank (bottleneck), all subsequent layers receive compressed information—they cannot access information lost in that bottleneck. This is why bottleneck architectures (autoencoders, efficient networks) are designed carefully: the bottleneck dimension is chosen to balance compression (useful for regularization, interpretability) with preservation of essential information. Understanding rank composition guides the choice of layer widths (too narrow = bottleneck, too wide = unnecessary capacity).

Example 7: Invertibility and Determinant (Preview)

Setup: Consider the matrix \(A = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \in \mathbb{R}^{2 \times 2}\). We want to determine if this linear map \(T_A: \mathbb{R}^2 \to \mathbb{R}^2\) is invertible.

Reasoning: For a square matrix, invertibility is equivalent to having nonzero determinant (and full rank). Compute: \[ \det(A) = 1 \cdot 4 - 2 \cdot 3 = 4 - 6 = -2 \neq 0 \]

Since the determinant is nonzero, \(A\) is invertible. To find \(A^{-1}\): \[ A^{-1} = \frac{1}{\det(A)} \begin{pmatrix} d & -b \\ -c & a \end{pmatrix} = \frac{1}{-2} \begin{pmatrix} 4 & -2 \\ -3 & 1 \end{pmatrix} = \begin{pmatrix} -2 & 1 \\ 3/2 & -1/2 \end{pmatrix} \]

Verify: \(A A^{-1} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \begin{pmatrix} -2 & 1 \\ 3/2 & -1/2 \end{pmatrix} = \begin{pmatrix} 1(-2) + 2(3/2) & 1(1) + 2(-1/2) \\ 3(-2) + 4(3/2) & 3(1) + 4(-1/2) \end{pmatrix} = \begin{pmatrix} -2+3 & 1-1 \\ -6+6 & 3-2 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}\) ✓.

The linear system \(A\mathbf{x} = \mathbf{b}\) has a unique solution \(\mathbf{x} = A^{-1}\mathbf{b}\) for any \(\mathbf{b}\).

Interpretation: A square matrix with nonzero determinant is invertible: it is a bijection from its domain to codomain. The inverse is unique and computable. Non-zero determinant (and equivalently, full rank = rank 2 for a \(2 \times 2\) matrix) ensures no information is lost: the transformation is reversible. Geometrically, the determinant measures the volume scaling factor: \(|\det(A)| = 2\) here, so the transformation scales area by a factor of 2 (and the negative sign indicates a reflection/orientation reversal). (Determinant is explored fully in Chapter 4.)

Common Misconceptions: Students often confuse “invertible” with “positive determinant.” Non-zero determinant (regardless of sign) suffices for invertibility. A reflection has negative determinant but is still invertible. Another error: assuming non-square matrices can’t be invertible (true—they can’t have two-sided inverses, but left or right inverses can exist, Chapter 4). A third misconception: computation errors in the \(2 \times 2\) formula (confusing the signs or order of the adjugate matrix).

What-If Scenarios: What if \(\det(A) = 0\)? Then \(A\) is singular (not invertible). Example: \(B = \begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}\) (second row = 2 × first row). \(\det(B) = 1(4) - 2(2) = 0\). The columns are linearly dependent (rank 1), and the map is not surjective (image is 1-dimensional). The equation \(B\mathbf{x} = \mathbf{b}\) has no solution unless \(\mathbf{b}\) is in the image (a 1D subspace). If \(\mathbf{b}\) is in the image, infinitely many solutions exist (differing by kernel vectors).

What if \(\det(A) \to 0\) (approaching singular)? The inverse \(A^{-1}\) has very large entries: small changes in \(\mathbf{b}\) lead to large changes in \(\mathbf{x}\). This is an ill-conditioned system (Chapter 13, numerical linear algebra)—solving \(A\mathbf{x} = \mathbf{b}\) numerically is unstable. Regularization helps.

Explicit ML Relevance: In neural networks, training involves solving gradient descent-related linear systems (e.g., Newton’s method, second-order optimization). If the Hessian (curvature matrix) is singular or near-singular, optimization is problematic—large gradient changes yield erratic weight updates. Regularization ensures away from singularity. Understanding determinant and invertibility guides the choice of optimization algorithms and learning rates. Low-rank weight matrices (rank-deficient) are also problematic: they indicate redundancy or bottlenecks, motivating pruning or architectural changes.

Example 8: Projection as Linear Map

Setup: In \(\mathbb{R}^3\), consider the projection of a vector onto the \(xy\)-plane: \(P(x, y, z) = (x, y, 0)\). This is a linear map \(P: \mathbb{R}^3 \to \mathbb{R}^3\). We verify that it is a projection (idempotent: \(P^2 = P\)) and analyze its kernel and image.

Reasoning: The matrix representation is: \[ P = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix} \]

Verify idempotence: \(P^2 = P \cdot P = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix} = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix} = P\) ✓. Once projected, applying the projection again doesn’t change anything.

The kernel is the set of vectors mapped to zero: \(P\mathbf{x} = \mathbf{0}\) means \((x, y, 0) = (0, 0, 0)\), so \(x = y = 0\). The kernel is \(\{\( (0, 0, z) : z \in \mathbb{R} \}\) = \(z\)-axis (1-dimensional).

The image is the set of outputs: \(P(\mathbb{R}^3) = \{(x, y, 0) : x, y \in \mathbb{R}\}\) = \(xy\)-plane (2-dimensional).

Rank-nullity: rank = 2, nullity = 1, domain dimension = 3. ✓

Interpretation: A projection decomposes the space into a “target” (image, here the \(xy\)-plane) and a “discarded” direction (kernel, here the \(z\)-axis). Any vector \((x, y, z) \in \mathbb{R}^3\) can be uniquely written as \((x, y, 0) + (0, 0, z)\) where the first term is in the image and the second is in the kernel. The projection preserves the image component and annihilates the kernel component. This is orthogonal decomposition (\(xy\)-plane \(\perp z\)-axis).

Common Misconceptions: Students sometimes think a projection is injective (it’s not—the kernel is non-trivial). Another error: confusing projection onto a subspace with projection onto a vector (the latter is a specific case—the subspace is 1-dimensional). A third mistake: assuming all idempotent matrices are projections—they are, but understanding the geometric meaning (decomposition) requires more care than just checking \(P^2 = P\).

What-If Scenarios: What if we project onto a different subspace, e.g., the line \(y = x\) in the \(xy\)-plane? The projection matrix is more complex (involves eigenvectors and eigenvalues, Chapter 5). But the structure is similar: idempotent, image = the target line, kernel = orthogonal complement.

What if the projection is not orthogonal (the kernel is not perpendicular to the image)? The math still works: \(P^2 = P\) still holds, decomposition still works. But the geometry is less intuitive (non-orthogonal decomposition). Orthogonal projections have special numerical properties and are preferred in practice.

Explicit ML Relevance: In machine learning, dimensionality reduction via PCA is an orthogonal projection onto a low-dimensional subspace (spanned by principal components). The projection discards variance in directions perpendicular to the components (kernel), retaining the most important variance (image). Understanding projection as decomposition is key to interpreting dimensionality reduction: what information is kept (image) vs. discarded (kernel). In neural networks, a fully connected hidden layer followed by a linear readout is roughly a dimensionality reduction followed by a change of basis—similar in spirit to projection.

Example 9: Feature Transformation in Neural Networks

Setup: In a simple neural network for image classification, the first hidden layer transforms raw pixel values \(\mathbf{x} \in \mathbb{R}^{784}\) (28×28 pixels, MNIST) to hidden features \(\mathbf{h} \in \mathbb{R}^{128}\) via \(\mathbf{h} = \sigma(W\mathbf{x} + \mathbf{b})\), where \(W \in \mathbb{R}^{128 \times 784}\) is the weight matrix, \(\mathbf{b} \in \mathbb{R}^{128}\) is the bias, and \(\sigma\) is a nonlinearity (ReLU, etc.). Ignoring the bias and nonlinearity, the core is a linear map \(T: \mathbb{R}^{784} \to \mathbb{R}^{128}\) defined by \(T(\mathbf{x}) = W\mathbf{x}\).

Reasoning: The matrix \(W\) has rank at most \(\min(128, 784) = 128\). If the rank is exactly 128 (full row rank), the image is the entire \(\mathbb{R}^{128}\) (surjective)—the layer can potentially output any 128-dimensional feature vector, maximizing expressivity. If the rank is less than 128 (e.g., rank = 100), the layer outputs a 100-dimensional subspace of \(\mathbb{R}^{128}\)—28 dimensions are “dead” (never activated, loss of capacity). The kernel has dimension \(784 - \text{rank}(W)\). If rank = 100, the kernel is 684-dimensional—many input pixels differ only in those kernel directions, yielding identical hidden features.

Interpretation: The weight matrix \(W\) is a learned linear map that compresses the domain (\(\mathbb{R}^{784}\)) to the hidden space (\(\mathbb{R}^{128}\)). The compression is quantified by the rank (information loss) and the kernel (directions projecting to zero). After compression, the nonlinearity \(\sigma\) injects nonlinearity (makes the overall layer nonlinear). Without the nonlinearity, the layer would be a linear projection—limited to separating linearly separable classes. The nonlinearity enables complex decision boundaries. The rank of \(W\) determines whether the hidden layer has sufficient capacity to represent diverse features. A low-rank \(W\) is a bottleneck; a full-rank \(W\) (if it exists—rank \(\leq 128\)) provides maximum capacity for that layer width.

Common Misconceptions: A frequent error is ignoring the nonlinearity and treating the layer as purely linear, missing the fact that the overall layer is nonlinear. Another mistake: assuming the layer “learns” a diagonal (simple scaling) matrix—in practice, \(W\) is typically full-rank (or nearly so) and dense (many nonzero entries), capturing complex mixing of input features. A third misconception: thinking the rank automatically equals the layer width (128)—depending on initialization and training, the learned \(W\) might have lower rank, especially if regularization is applied.

What-If Scenarios: What if the hidden layer is too narrow (width < 784)? Compression is forced (rank < 784 at least). If the hidden dimension is 10, at most 10 independent directions can be captured from the input space. Sometimes this is useful (forced features that are more compact), but it can also lose important information. This trade-off is key to architecture design.

What if the hidden layer has width > 784 (e.g., 1000)? The matrix \(W\) can have rank up to 784 (limited by the input dimension). The layer can represent up to 784 independent directions—potentially more expressive. But without strong constraints (regularization, dropout), the layer might overfit, learning spurious features. Regularization (weight decay, L1/L2 penalties) biases toward lower-rank, sparser solutions.

Explicit ML Relevance: The rank of a neural network’s weight matrix directly impacts its representational capacity. Recent work on neural network compression (pruning, low-rank factorization) exploits this: replacing a full-rank \(128 \times 784\) matrix with a low-rank factorization (e.g., \(128 \times 16\) and \(16 \times 784\)) reduces parameters from \(128 \cdot 784 = 100,352\) to \((128 \cdot 16) + (16 \cdot 784) = 2,048 + 12,544 = 14,592\), a ~7× reduction, while often maintaining accuracy. Understanding rank-capacity relationships is central to efficient deep learning.

Example 10: Rank Deficiency and Model Collapse

Setup: In an autoencoder with an encoder \(f_{\text{enc}}: \mathbb{R}^{784} \to \mathbb{R}^{16}\) (compression) and decoder \(f_{\text{dec}}: \mathbb{R}^{16} \to \mathbb{R}^{784}\) (reconstruction), the combination \(f_{\text{dec}} \circ f_{\text{enc}}\) is a map \(\mathbb{R}^{784} \to \mathbb{R}^{784}\) (returns to input space). If both are linear (no nonlinearities), the composition is \(f_{\text{dec}} \circ f_{\text{enc}} = W_{\text{dec}} W_{\text{enc}}\) where \(W_{\text{enc}} \in \mathbb{R}^{16 \times 784}\) and \(W_{\text{dec}} \in \mathbb{R}^{784 \times 16}\).

Reasoning: The product \(W_{\text{dec}} W_{\text{enc}} \in \mathbb{R}^{784 \times 784}\) has rank at most \(\min(16, 16, 784) = 16\) (limited by the bottleneck dimension). So the overall map from input to reconstruction is rank-16, meaning only 16 independent “directions” in the input space are preserved; the other \(784 - 16 = 768\) directions are annihilated (lost). The autoencoder cannot reconstruct inputs perfectly—it can only reconstruct projections onto a 16-dimensional subspace. Any information not captured in the bottleneck is inaccessible to the decoder.

Example: Suppose the encoder preserves the principal components (the 16 directions of highest variance in the data), and the decoder inverts this. Then the reconstruction captures the 16-component approximation, discarding low-variance components. This can be useful (denoising) or harmful (loss of fine details), depending on what the discarded variance contains.

Interpretation: Rank deficiency in a composed map indicates information bottleneck. The composition through a low-dimensional intermediate creates unavoidable information loss. This is the “reconstruction loss” in autoencoders: the data cannot be perfectly recovered from the 16-dimensional representation. The best the autoencoder can do is approximate each input as a linear combination of learned basis vectors. If regularization or sparsity is applied, further information loss occurs, concentrating on the most “important” features (highest variance, most compressible). This is model collapse mitigation: preventing the model from trivially memorizing the identity (which would happen if the bottleneck were full-rank, allowing perfect reconstruction).

Common Misconceptions: Students sometimes assume autoencoders always achieve lossy compression (losing information is good for regularization). But if the bottleneck is too narrow, the information loss can be too severe, even for the training data—the model fails to learn anything useful. There’s a trade-off: wider bottleneck = more capacity to preserve information (less collapse), narrower bottleneck = more compression (stronger regularization, but risk of insufficient capacity). Another misconception: confusing representation collapse (all bottleneck features are the same, captured by \(W_{\text{dec}}\)) with information collapse (high-dimensional input cannot be represented in low dimensions). Both are problems, but the sources differ.

What-If Scenarios: What if the encoder rank is full (\(W_{\text{enc}}\) has rank 16 relative to a theoretically possible rank in a non-embedding context; assume it’s full for its dimensions) and \(W_{\text{dec}}\) is full? Then \(W_{\text{dec}} W_{\text{enc}}\) has rank 16. The reconstruction error (loss of 768-dimensional information) is inevitable from the bottleneck. The autoencoder cannot be perfect for general data—it must learn which 16 directions are most important (usually the principal components via PCA intuition). What if we remove regularization and let the model train freely? Gradient descent might push \(W_{\text{dec}} W_{\text{enc}}\) toward full rank (by making the bottleneck effective in passing all information), achieving perfect reconstruction at the cost of overfitting (memorizing the training data). Regularization prevents this (Chapter 13).

Explicit ML Relevance: Autoencoders and variational autoencoders (VAEs) are foundational, and understanding rank deficiency in their bottlenecks is crucial. The choice of bottleneck dimension (16, 32, 128, etc.) directly impacts reconstruction quality and generalization. Too small → information loss and poor reconstruction; too large → overfitting risk without regularization. Modern architectures (e.g., β-VAE, Chapter 17) use regularization to control the informational capacity of the bottleneck, preventing collapse. Understanding rank-deficiency helps explain why such regularization is necessary.

Example 11: Linear Map Between Function Spaces

Setup: Consider the vector space \(V = \mathcal{P}_2\) (polynomials of degree \(\leq 2\)) and the space \(W = \mathbb{R}^3\). Define a linear map \(T: \mathcal{P}_2 \to \mathbb{R}^3\) by evaluation at three points: \(T(p) = (p(0), p(1), p(2))^\top\). We examine the properties of this map (kernel, image, rank).

Reasoning: A polynomial \(p(x) = a_0 + a_1 x + a_2 x^2 \in \mathcal{P}_2\) is represented by coordinates \((a_0, a_1, a_2)^\top\) in the standard basis \(\{1, x, x^2\}\). The map evaluates \(p\) at \(x = 0, 1, 2\): \[ T(a_0 + a_1 x + a_2 x^2) = (a_0, a_0 + a_1 + a_2, a_0 + 2a_1 + 4a_2)^\top \]

The matrix representation is: \[ A = \begin{pmatrix} 1 & 0 & 0 \\ 1 & 1 & 1 \\ 1 & 2 & 4 \end{pmatrix} \] (columns correspond to evaluations of \(1, x, x^2\) at the three points).

The kernel is \(\ker(T) = \{p \in \mathcal{P}_2 : p(0) = p(1) = p(2) = 0\}\). A polynomial of degree \(\leq 2\) with three roots can only be the zero polynomial (a nonzero degree-2 polynomial has at most 2 roots). So \(\ker(T) = \{0\}\), and the map is injective.

The image is the set of all vectors \((p(0), p(1), p(2))^\top\) for polynomials \(p \in \mathcal{P}_2\). By the rank-nullity theorem, rank\(T\) = \(\dim(\mathcal{P}_2) - \text{nullity}(T) = 3 - 0 = 3\). So the image is 3-dimensional, hence equals all of \(\mathbb{R}^3\). The map is surjective. Since it’s both injective and surjective, it’s bijective and represents an isomorphism \(\mathcal{P}_2 \cong \mathbb{R}^3\).

Interpretation: This example shows that abstract vector spaces (like polynomials) are isomorphic to concrete spaces (like \(\mathbb{R}^n\)). The evaluation map captures this isomorphism: each polynomial is uniquely determined by its values at three points (Lagrange interpolation). This is a bridge between abstract linear algebra (polynomials form a vector space) and matrix computations (coordinate vectors and matrix multiplications). The evaluation points (0, 1, 2) define a basis for the representation—choosing different points yield different matrices representing the same geometric map.

Common Misconceptions: Students sometimes think abstract spaces (polynomials, function spaces) are fundamentally different from \(\mathbb{R}^n\)—they seem harder to visualize. But linear algebra shows they’re structurally identical (isomorphic). The only difference is the representation: polynomials are elements (functions), but coordinates are vectors. Another error: forgetting that polynomial evaluation is linear (\(T(p + q) = T(p) + T(q)\), \(T(cp) = cT(p)\)). A third misconception: assuming the matrix representation depends on the choice of evaluation points—it does, but the abstract map is the same (similarity transformation relates the different representations).

What-If Scenarios: What if we evaluate at only two points? \(T: \mathcal{P}_2 \to \mathbb{R}^2\), \(T(p) = (p(0), p(1))^\top\). The kernel is now non-trivial: any polynomial with \(p(0) = p(1) = 0\) is in the kernel. There are infinitely many (e.g., \(p(x) = x(x-1) \in \mathcal{P}_2\)). The kernel dimension \(\geq 1\), so rank \(\leq 2\). Actually, rank = 2 (the map is surjective), so the kernel is exactly 1-dimensional (by rank-nullity: \(3 - 2 = 1\)). The kernel is spanned by a polynomial with roots 0 and 1 (e.g., \(x(x - 1)\)). What if we add a fourth evaluation point (but keep polynomials of degree \(\leq 2\))? Now \(T: \mathcal{P}_2 \to \mathbb{R}^4\), with a \(4 \times 3\) matrix—rank at most 3, so the map is injective but not surjective. The image is a 3-dimensional subspace (hyperplane) in \(\mathbb{R}^4\).

Explicit ML Relevance: In neural networks with infinite-dimensional inputs (e.g., function regression, operator learning), the inputs are approximated by finite-dimensional representations (e.g., polynomial basis, Fourier basis, neural network basis). The evaluation or projection map becomes an isomorphism between the infinite-dimensional space and the finite-dimensional approximation, enabling matrix-based computations. Understanding function spaces and linear maps is crucial for advanced topics like neural operators (Chapter 18+), where networks learn maps between function spaces.

Example 12: Deep Linear Network as Composed Linear Maps

Setup: Consider a deep linear neural network with three layers: - Input: \(\mathbf{x} \in \mathbb{R}^{10}\) - Hidden 1: \(\mathbf{h}_1 = W_1 \mathbf{x}\), where \(W_1 \in \mathbb{R}^{8 \times 10}\) (rank 8, full row rank) - Hidden 2: \(\mathbf{h}_2 = W_2 \mathbf{h}_1\), where \(W_2 \in \mathbb{R}^{5 \times 8}\) (rank 5, full row rank) - Output: \(\mathbf{y} = W_3 \mathbf{h}_2\), where \(W_3 \in \mathbb{R}^{3 \times 5}\) (rank 3, full row rank)

The overall map is \(f(\mathbf{x}) = W_3 W_2 W_1 \mathbf{x}\): three composed linear maps.

Reasoning: Compute the composition: \[ A = W_3 W_2 W_1 \in \mathbb{R}^{3 \times 10} \]

The rank is bounded by \(\min(3, 5, 8, 10) = 3\). We check the actual rank by analyzing the intermediate dimensions. \(W_1\) maps from \(\mathbb{R}^{10}\) to \(\mathbb{R}^8\), compressing by 2 dimensions (image is 8-dimensional). \(W_2\) maps from \(\mathbb{R}^8\) to \(\mathbb{R}^5\) (full row rank, so the entire output of \(W_1\) is “processed”, image is 5-dimensional). \(W_3\) maps from \(\mathbb{R}^5\) to \(\mathbb{R}^3\) (full row rank, image is 3-dimensional). The composition: \(W_1\) outputs an 8-dimensional representation. \(W_2\) reduces this to 5 dimensions. \(W_3\) further reduces to 3 dimensions. The final output is 3-dimensional.

The rank of \(A = W_3 W_2 W_1\) is 3 (since all compositions maintain full rank at each step, there’s no rank collapse). The map \(f: \mathbb{R}^{10} \to \mathbb{R}^3\) is a linear dimensionality reduction, projecting the 10-dimensional input onto a 3-dimensional output space.

The kernel of \(A\) has dimension \(10 - 3 = 7\) (by rank-nullity). Vectors in the kernel map to zero output. These represent “unimportant” directions (relative to the network’s learned structure)—information lost through the compression stages.

Interpretation: A deep linear network composes multiple linear maps, each potentially compressing dimensionality. The overall map is determined by the bottleneck (narrowest intermediate layer): the 5-dimensional hidden layer in layer 2 is the tightest constraint. Information cannot flow back up (the decoder-like inverse doesn’t exist for non-square layers). The kernel (7-dimensional) represents information discarded. Understanding this decomposition shows that even linear depth adds structure: with one layer \(W_3 W_1 \in \mathbb{R}^{3 \times 10}\), the result is identical (rank 3). So depth doesn’t add expressivity for linear maps (a single layer suffices). This is why nonlinearities are essential: they break linearity, making depth matter. (This is explored further in Chapter 5 on eigenvalues—repeated matrix multiplication has spectral consequences, but for different goals.)

Common Misconceptions: A frequent error is thinking deep linear networks are more expressive than shallow ones—they’re not. Without nonlinearities, composing linear maps yields a linear map, expressible as a single layer. Depth matters only with nonlinearities. Another misconception: confusing the bottleneck dimension (smallest intermediate layer) with the output dimension. The bottleneck constrains all downstream information. A third error: forgetting that even though ranks are full at each stage, the overall rank is bounded by the input and output dimensions (\(\leq \min(3, 10) = 3\)).

What-If Scenarios: What if one of the matrices has less than full row rank? Example: \(W_2 \in \mathbb{R}^{5 \times 8}\) with rank 4 (not full). Then \(W_2\) outputs a 4-dimensional subspace (not all of \(\mathbb{R}^5\)). Subsequent layers \(W_3\) can only process the image of \(W_2\) (4 dimensions). The overall rank drops: \(\text{rank}(W_3 W_2 W_1) \leq \text{rank}(W_2) = 4\). The network has a capacity limit (at most a 4-dimensional output, even though \(W_3\) could theoretically output 3 dimensions). This highlights the principle: any rank-deficient layer creates a bottleneck.

What if we add nonlinearities \(\sigma\) (ReLU, tanh) between layers? The composition \(W_3 \sigma(W_2 \sigma(W_1 \mathbf{x}))\) is no longer linear. Depth now adds expressivity—the network can represent non-polynomial (piecewise linear, bounded, etc.) functions that a single-layer network cannot. This is the power of deep networks: depth + nonlinearity = complex function class. (Explored in Chapters 6–8.)

Explicit ML Relevance: Appreciating linear networks (even though they’re not practically used due to lack of nonlinearity) is pedagogically valuable: they teach how composition works, how rank propagates, and why depth is useless without nonlinearity. In contemporary deep learning, concepts like skip connections, bottleneck architectures, and rank-deficiency constraints (e.g., in knowledge distillation, pruning) all rely on understanding composed linear maps. When debugging neural networks—why is a layer not learning, why is training diverging—often the culprit is rank deficiency or ill-conditioned matrix compositions. Linear algebra provides the diagnostic tools.

Summary

Key Ideas Consolidated

This chapter has developed a comprehensive understanding of linear maps and their matrix representations, establishing the bridge between abstract linear algebra and concrete computational practice. The central ideas are:

Linear maps as structure-preserving functions: A linear map \(T: V \to W\) preserves the fundamental operations of vector spaces—addition and scalar multiplication. This property makes linear maps both theoretically elegant (composition, invertibility, diagonalization) and computationally tractable (matrix algorithms, eigenvalue methods). Unlike arbitrary nonlinear functions, linear maps form a mathematical structure (vector space, algebra) enabling systematic study.

Matrix representation as encoding transformation: Every finite-dimensional linear map can be represented as a matrix (with respect to chosen bases). The matrix columns are the images of basis vectors, directly encoding how the transformation affects the fundamental directions of the domain. Different bases yield different numerical matrices, but the geometric transformation is invariant—a similarity transformation relates the representations. This duality (abstract map ↔︎ concrete matrix) is the power of linear algebra: abstract reasoning guides algorithm design, and matrix computations make the methods practical.

Kernel and image as decomposition: Every linear map decomposes the domain into a kernel (annihilated directions) and a complementary subspace (whose image is the image of the map). The rank-nullity theorem formalizes this decomposition: \(\dim(V) = \text{rank}(T) + \text{nullity}(T)\). This partition reveals how information flows through the map: the rank quantifies information passing through (output diversity), and the nullity quantifies information lost (compression). For neural networks, this is critical: low-rank layers are bottlenecks.

Composition and rank propagation: Linear maps compose (one map’s output becomes another’s input), and the rank of a composition is bounded by the minimum of the composing maps’ ranks. In deep networks, this constraint is fundamental: any low-rank hidden layer limits the effective dimensionality of all subsequent layers. Understanding rank propagation guides architecture design (how wide should each layer be?) and regularization (encouraging sparsity and low-rank for efficiency).

Invertibility and uniqueness: For finite-dimensional spaces of equal dimension, invertibility (bijection) is equivalent to injectivity (trivial kernel) or surjectivity (entire image = codomain). A nonzero determinant (for square matrices) is the practical test. In regression, invertibility of the design matrix ensures unique solutions; rank-deficiency means infinitely many solutions or no solution—the necessity of regularization.

ML-centric perspective: Linear maps are the operational core of machine learning. Neural network layers (before nonlinearity) are linear maps. Regression and linear classification are linear maps composed with loss functions. Dimensionality reduction projects onto subspaces (projections are linear maps). Optimization searches for weights (parameters of linear maps). Kernel methods implicitly work in high-dimensional spaces where linear separability emerges. Understanding linear maps is understanding the foundation of modern ML.

What the Reader Should Now Be Able To Do

Upon completing this chapter, you should be able to:

Theoretical Competencies:

1. Characterize linear maps and their properties: Determine whether a function \(T: V \to W\) is linear, identify its kernel, image, rank, and nullity; verify injectivity and surjectivity and invertibility via row reduction and determinant tests.

2. Represent linear maps as matrices: Compute matrix representations by evaluating the map on basis vectors and expressing results in the target basis; ensure correctness via matrix-vector multiplication.

3. Perform basis changes and understand similarity: Compute representations in different bases using the change-of-basis formula \(A' = P^{-1}AP\); recognize geometric invariance under basis change and understand how to exploit basis choices to simplify matrices.

4. Compose linear maps and analyze rank: Compose compatible linear maps, represent as matrix products, and compute composition rank; apply rank-composition bounds to identify capacity-limiting bottlenecks in deep networks.

5. Apply rank-nullity to solution analysis: Use rank-nullity decomposition to determine solution existence/uniqueness in regression; understand how information flows through mappings and how regularization addresses degeneracy.

Practical Competencies:

6. Recognize and exploit mathematical structure: Identify symmetric, rank-1, and structured matrices; apply efficient solvers and low-rank approximations for computational efficiency and storage reduction.

7. Diagnose neural networks via linear algebra: Identify bottleneck layers through rank analysis; recognize singular/ill-conditioned weights; diagnose capacity limitations and information loss in layer compositions.

8. Design network architectures using rank constraints: Choose layer widths and connection patterns by reasoning about rank propagation; predict effective dimensionality at each layer; use skip connections to prevent information collapse.

9. Use rank and decomposition to guide regularization: Apply ridge regression, low-rank approximations, and rank constraints to improve stability and generalization; understand how regularization modifies the rank structure of solutions.

10. Connect deep learning training dynamics to linear maps: Understand gradient flow through composed linear transformations; analyze vanishing/exploding gradients in terms of Jacobian rank and singular values; predict convergence issues from weight matrix conditioning.

Structural Assumptions for Later Chapters

This chapter builds on prior foundational knowledge and makes assumptions for future extensions:

Assumptions from Earlier Chapters (Prerequisite Knowledge):

• Vector spaces, subspaces, bases, dimension, linear independence, and rank-nullity theorem from Chapter 2
• Gaussian elimination, row echelon form, and solution structure for homogeneous/non-homogeneous linear systems
• The concept that transformations can be represented and composed algebraically

Structural Assumptions Made in This Chapter:

1. Finite-dimensional vector spaces over ℝ or ℂ: All spaces are isomorphic to ℝⁿ for finite n; infinite-dimensional spaces (Banach spaces, Hilbert spaces) are deferred to advanced chapters requiring functional analysis and topological tools.

2. Basis-dependent matrix representations: Every matrix represents a transformation with respect to specific ordered pairs of bases; changing bases transforms the matrix via similarity; forgetting the basis leads to conceptual confusion.

3. Rank as a fundamental invariant: Rank is the key quantity determining whether maps are injective/surjective/invertible; rank is preserved under similarity and change of basis, making it basis-independent and fundamental.

Assumptions for Later Chapters (Forward Requirements):

• Chapter 4 extends to norms and inner products, adding geometric structure (angles, orthogonality) to supplement algebraic structure of transformations
• Chapter 5 develops eigenvalues and diagonalization, seeking special bases where linear maps become maximally simple (diagonal matrices)
• Chapter 6+ builds on rank, nullity, and composition principles, applying them to optimization, dimensionality reduction, and neural network design throughout

Limitations and Caveats Acknowledged:

• Basis representations are non-canonical: Different choices of bases produce different numerical matrices for the same geometric transformation; a matrix alone doesn’t encode geometry without knowing the basis pair.

• Rank and conditioning are numerically fragile: Near-machine-precision singular values create ambiguity between true rank deficiency and numerical noise; column scaling and perturbations can dramatically change apparent rank.

• Composition bounds on rank are loose: While rank composition satisfies \(\text{rank}(T_2 \circ T_1) \leq \min(\text{rank}(T_1), \text{rank}(T_2))\), the actual rank is often much smaller due to hidden algebraic structure in data.

• Finite-dimensional theory does not smoothly extend to infinite dimensions: Operator algebras and functional analysis require topological care; basis existence becomes non-constructive; intuition from finite dimensions can fail.

In Context

Algorithmic Development History

The theory of linear maps and matrix representations has evolved through centuries of mathematical practice, shaped by computational needs and theoretical insights:

Classical matrix algebra (1700s–1800s): Matrices emerged from the need to solve systems of linear equations. Gauss’s elimination method (ca. 1810) was the computational breakthrough—systematic row reduction to triangular form, followed by back-substitution. This method was not called “Gaussian elimination” until the 1930s, but the idea was implicit in earlier work (e.g., Chinese mathematical texts from ~250 CE). By the 1800s, mathematicians recognized patterns: determinants (Cramer’s rule for solving systems without elimination), matrix multiplication (composition of transformations), and inverse matrices. Lagrange and Laplace developed determinants; Cayley and Sylvester formalized matrix algebra as an abstract system.

Linear operator theory (early 1900s): The shift from concrete matrices to abstract linear maps came with the rise of functional analysis and the study of function spaces. Hilbert and his students developed operator theory in Hilbert spaces—infinite-dimensional generalizations of \(\mathbb{R}^n\). The Fredholm operator theory (studying integral equations as linear operators on function spaces) became foundational in PDE theory. The abstract perspective revealed universal principles: kernel, image, rank, invertibility criteria apply to matrices, differential operators, and even more exotic operators. Matrix analysis (finite-dimensional) became a special case of operator theory.

Matrix analysis formalization (mid-1900s): The 20th century saw rigorous formalization. Schur’s theorem (triangulation of matrices), the spectral theorem (diagonalization of symmetric/Hermitian operators), and the singular value decomposition (SVD, generalizing eigenvalue decomposition) emerged. The von Neumann-Schatten theory developed norms on operators (the operator norm, Frobenius norm, trace norm), enabling analysis of matrix approximation and perturbation. Perron-Frobenius theory (spectral properties of non-negative matrices) had applications in Markov chains and Google’s PageRank algorithm.

Numerical linear algebra and computational advances (1950s–1980s): As computers emerged, the practical challenges of Matrix computation became urgent. Wilkinson’s work on numerical stability showed that theoretical solutions (e.g., via Cramer’s rule determinants) are often numerically unstable. Iterative methods (Jacobi, Gauss-Seidel, conjugate gradient) outperform direct methods on large systems. The QR decomposition (Householder et al.) provides a numerically stable way to compute least-squares solutions. The SVD (widely adopted in the 1970s) became central to numerical linear algebra: computing low-rank approximations, regularizing ill-conditioned systems, and solving least-squares problems robustly.

Linear models in statistics (1900s–1970s): Statistics independently developed linear algebra (often under different names). Regression (Galton, Pearson, 1890s–1920s) is solving a linear system \(X\mathbf{w} = \mathbf{y}\) in the least-squares sense. The design matrix \(X\), the regression coefficients \(\mathbf{w}\), and the residuals \(\mathbf{y} - X\mathbf{w}\) are all linear-algebraic objects. Gauss-Markov theorem (linear estimators have minimal variance under certain assumptions) is a theorem about the geometry and properties of linear maps in function spaces. Principal component analysis (PCA, Hotelling, 1930s) is projection onto principal directions—the eigenvectors of the covariance matrix. Multiple regression, ANOVA, and general linear models unified under the lens of linear maps. In modern statistics, matrix computations (via the SVD, Cholesky decomposition) are standard for maximum likelihood estimation and Bayesian inference.

Modern deep learning (2010s–present): Deep neural networks revitalized interest in linear algebra fundamentals. Each network layer is a linear map (the weight matrix), followed by a nonlinearity. The stacking of layers creates composited nonlinear functions. Backpropagation (the gradient computation) involves composing Jacobian matrices (linear approximations of the map at each layer)—matrix chain rule. Modern challenges in deep learning are linear-algebraic:

• Vanishing/exploding gradients: Composing many Jacobians (all rank-1 due to ReLU saturation or singular due to initialization) leads to gradient magnitudes shrinking or exploding. Solution: initialization schemes (Xavier, He) ensure balanced scales; layer normalization uses linear transformations (whitening) to stabilize.

• Redundancy and pruning: Low-rank layers are identified and pruned (matrix rank reduction). Knowledge distillation compresses networks by learning a low-rank approximation of a large network’s function.

• Efficient inference: Tensor decomposition (generalizing low-rank matrix factorization) reduces model size. Pruning and quantization are applied to weight matrices.

• Regularization: Weight decay penalizes the Frobenius norm of weight matrices, implicitly biasing toward low-rank solutions. Dropout perturbs the linear map, preventing co-adaptation of features.

The rise of transformers (Vaswani et al., 2017) brought renewed focus on linear maps: the attention mechanism is a composition of (learnable) linear projections and soft projections (applying a softmax-normalized kernel—a matrix of inner products). Understanding attention requires linear algebra (dot products as inner products, projections as matrix multiplications).

Why This Matters for ML

Understanding linear maps is essential for modern machine learning because the entire computational graph of neural networks is built from linear maps (with nonlinearities interspersed). Let’s connect the theory directly to practice:

Linear layers as building blocks: In any neural network, a fully connected (dense) layer computes \(\mathbf{h} = W\mathbf{x} + \mathbf{b}\). Ignoring the bias and nonlinearity, this is the linear map \(T(\mathbf{x}) = W\mathbf{x}\). The weight matrix \(W \in \mathbb{R}^{m \times n}\) transforms \(n\)-dimensional input to \(m\)-dimensional output. The rank of \(W\) determines the effective output dimension:

• Full rank (\(\text{rank}(W) = m\)) means the layer can potentially output any \(m\)-dimensional vector (surjective). If the data’s essential features can be represented in \(m\) dimensions, this layer is not a bottleneck.

• Low rank (\(\text{rank}(W) < m\)) means outputs lie in a rank-\(r\) subspace. Subsequent layers are constrained to this subspace—they cannot access the \(m - r\) lost dimensions. This is a bottleneck.

In practice, untrained networks often have near-full rank weights (random initialization). During training, gradient descent may push weights toward lower rank (due to regularization or data structure), creating implicit compression. Analyzing the singular values of \(W\) (the SVD) reveals whether a layer is effectively lower-rank than nominally stated.

Rank and expressivity limits: A fundamental trade-off in neural networks is between expressivity (the function class the network can represent) and generalization (how well it performs on unseen data).

For linear models (no nonlinearity), expressivity is limited by rank: an \(m \times n\) matrix has rank \(\leq \min(m, n)\), limiting the output function class. Adding nonlinearity breaks this limitation—a composition of linear and nonlinear maps can represent much richer functions. However, each nonlinearity requires computation. Deep networks (many layers, potentially low rank between layers) achieve expressivity by composing nonlinearities. The trade-off is depth vs. width: given a fixed number of parameters, do you use many thin layers or fewer wide layers? A consequence of rank-nullity: if you have a bottleneck layer, you cannot recover the lost information in later layers (unless the nonlinearity can somehow “regenerate” lost dimensions—it cannot). So architecture design must ensure no critical information is compressed away.

Explicit ML application: Autoencoders have an encoder (projects input to a low-dimensional bottleneck) and a decoder (reconstructs from the bottleneck). If the bottleneck is linear (or has shallow depth), the best reconstruction is a low-rank approximation of the identity, capturing only directions of high variance in the data. VAEs extend this: the bottleneck is a stochastic noise layer, and the decoder has a KL divergence penalty to match a prior—a regularization technique (Chapter 13) that further constrains the learned representation.

Rank and generalization: A classical result in statistical learning theory is that low-rank (simple) models generalize better. Regularization methods (ridge regression, weight decay in neural networks) implicitly encourage low-rank solutions. Understanding how regularization (a perturbed loss function) relates to rank (via the SVD and the spectrum of the Hessian) is a bridge between optimization and generalization—a profound topic (Chapters 13, 15).

Forward Links to Norms and Inner Products

The next chapters extend linear maps with geometric structure via norms and inner products:

Norms (Chapter 6) measure vector magnitude and are used throughout ML: the \(L_2\) norm \(\|\mathbf{x}\|_2 = \sqrt{\sum_i x_i^2}\) measures Euclidean distance; the \(L_1\) norm \(\|\mathbf{x}\|_1 = \sum_i |x_i|\) encourages sparsity in regularization. The operator norm of a linear map (Theorem 6 preview, Example 7) is the largest factor by which the map stretches vectors—essential for understanding stability and conditioning (Chapter 11). In neural networks, the spectral norm (largest singular value) is often constrained via spectral normalization to stabilize training.

Inner products (Chapter 7) define angles and orthogonality. An inner product \(\langle \mathbf{u}, \mathbf{v} \rangle\) induces a notion of “perpendicularity” (\(\langle \mathbf{u}, \mathbf{v} \rangle = 0\)). Orthogonal projections (a special class of the projections in Example 8) project vectors onto subspaces such that the projection error is perpendicular to the subspace. This is dual to the concept of the orthogonal complement of the kernel (in the context of least-squares regression, Example 4). PCA (principal component analysis) finds the directions of maximum variance and project onto them—an orthogonal projection onto the span of the principal eigenvectors (Chapter 5). Kernel methods (Chapter 16) define inner products in high-dimensional (or infinite-dimensional) implicit spaces, enabling linear classification in transformed representations.

Applications in optimization (Chapter 8): Gradient descent searches for a minimizer of a loss function by moving in the direction of steepest descent (negative gradient). The gradient is defined via the inner product (the Fréchet or Riesz representation); the step direction is the negative gradient (a linear map on the loss landscape—the Hessian). Second-order methods (Newton, quasi-Newton) use the Hessian (the matrix of second derivatives, a linear operator on the parameter space). Understanding the spectrum (eigenvalues, Chapter 5) of the Hessian reveals the curvature landscape and predicts convergence rates. Ill-conditioned Hessians (large ratio of largest to smallest eigenvalue) slow convergence—a linear-algebraic characterization of optimization difficulty.

Deep learning and conditioning (Chapter 11): As neural networks get deeper, the composition of Jacobian matrices (linear maps) determines how gradients propagate backward. If each layer’s Jacobian has small singular values (e.g., due to saturation of ReLU in early layers), the overall Jacobian has exponentially small singular values—gradients vanish. Conversely, if singular values are large, gradients explode. Modern techniques (batch normalization, layer normalization, skip connections) address this by controlling the singular value spectrum of the composed map. Understanding singular values and operator norms is thus critical for designing and training deep networks.

Implicit connections: The machinery of linear algebra (rank, kernel, image, matrix representation) will appear implicitly throughout later chapters. Recognizing “this is really a linear map” or “this is a rank-deficient matrix problem” helps apply appropriate tools and intuition to seemingly different contexts.

End-of-Chapter Advanced Exercises

A. True / False (20)

A.1. For any linear map \(T: \mathbb{R}^5 \to \mathbb{R}^3\), the kernel of \(T\) is guaranteed to be nontrivial (i.e., contains more than just the zero vector).

A.2. If two matrices \(A\) and \(B\) have the same rank, then they represent the same linear map in some choice of bases.

A.3. The composition of two injective linear maps is always injective, but the composition of two surjective linear maps is not always surjective.

A.4. If the kernel of \(T: V \to W\) is one-dimensional and the image of \(T\) is two-dimensional, then \(\dim(V) = 3\).

A.5. A linear map \(T: \mathbb{R}^n \to \mathbb{R}^n\) is invertible if and only if \(\det(A) \neq 0\), where \(A\) is any matrix representation of \(T\).

A.6. For a neural network with a fully connected layer having weight matrix \(W \in \mathbb{R}^{128 \times 784}\) (128 output units, 784 input features), the maximum information capacity of the layer is at most 128 dimensions, regardless of the data distribution.

A.7. If \(T: V \to V\) is a linear map on a finite-dimensional space such that \(\text{rank}(T) = \dim(V) - 1\), then \(T\) cannot be invertible.

A.8. A projection operator \(P: V \to V\) satisfying \(P^2 = P\) has the property that \(\ker(P)\) and \(\text{im}(P)\) are complementary subspaces (i.e., \(V = \ker(P) \oplus \text{im}(P)\)).

A.9. In a deep linear network (composition of \(k\) linear layers), the rank of the overall map is the product of the individual layer ranks.

A.10. If a weight matrix \(W \in \mathbb{R}^{m \times n}\) has rank \(r < \min(m, n)\), then there exist two distinct input vectors \(\mathbf{x}_1 \neq \mathbf{x}_2\) such that \(W\mathbf{x}_1 = W\mathbf{x}_2\).

A.11. The singular value decomposition (SVD) of a matrix \(A = U \Sigma V^\top\) reveals that the rank of \(A\) equals the number of nonzero diagonal entries in \(\Sigma\).

A.12. A linear map \(T: V \to W\) between finite-dimensional spaces is surjective if and only if \(\text{rank}(T) = \dim(W)\).

A.13. For a regression problem with design matrix \(X \in \mathbb{R}^{n \times p}\) (n observations, p features), if \(\text{rank}(X) < p\), then the least-squares solution \(\hat{\mathbf{w}} = (X^\top X)^{-1} X^\top \mathbf{y}\) exists and is unique.

A.14. Two matrices \(A\) and \(B\) in \(\mathbb{R}^{n \times n}\) represent the same linear map (in different bases) if and only if they are similar, i.e., \(B = P^{-1}AP\) for some invertible matrix \(P\).

A.15. An autoencoder with a bottleneck layer (encoder output dimension smaller than input dimension) fundamentally cannot reconstruct inputs outside the image of the encoder’s linear component, even with optimal nonlinear decoders.

A.16. The operator norm of a linear map \(T: \mathbb{R}^n \to \mathbb{R}^m\) (induced by Euclidean norms) equals the largest singular value of its matrix representation.

A.17. For a fully connected neural network layer with ReLU activation, if the weight matrix \(W\) has rank less than the number of output units, then no amount of training can make the layer express a function that requires the full output dimensionality.

A.18. A change of basis transformation \(A' = P^{-1}AP\) preserves the rank of the original matrix \(A\).

A.19. In backpropagation through a deep linear network, the gradient with respect to an early layer’s weight matrix depends on the product of the Jacobian matrices of all subsequent layers, so a single low-rank layer can cause vanishing gradients for all earlier layers.

A.20. The kernel of a linear map \(T: V \to W\) is always a subspace of the domain \(V\), and the image of \(T\) is always a subspace of the codomain \(W\).

B. Proof Problems (20)

B.1. Prove that the kernel of a linear map \(T: V \to W\) is a subspace of \(V\). Be explicit about closure under addition and scalar multiplication.

B.2. Prove that if \(T: V \to W\) is a linear map with \(\text{rank}(T) = \dim(V)\), then \(T\) is injective.

B.3. For a fully connected neural network layer represented by weight matrix \(W \in \mathbb{R}^{m \times n}\), prove that if \(\text{rank}(W) = r < n\), then the image of the linear map \(T(\mathbf{x}) = W\mathbf{x}\) is at most \(r\)-dimensional, regardless of the input distribution.

B.4. Prove the rank-nullity theorem: For a linear map \(T: V \to W\) between finite-dimensional vector spaces, \(\dim(V) = \text{rank}(T) + \text{nullity}(T)\).

B.5. Prove that the image (range) of a linear map \(T: V \to W\) is a subspace of \(W\).

B.6. Let \(T: \mathbb{R}^n \to \mathbb{R}^m\) be a linear map with matrix representation \(A \in \mathbb{R}^{m \times n}\) (in the standard bases). Prove that \(\text{rank}(A) = \text{rank}(T)\) (i.e., rank is independent of the choice of bases).

B.7. Prove that a linear map \(T: V \to V\) on a finite-dimensional vector space is invertible if and only if it is both injective and surjective.

B.8. Let \(T_1: U \to V\) and \(T_2: V \to W\) be linear maps. Prove that \(\text{rank}(T_2 \circ T_1) \leq \min(\text{rank}(T_1), \text{rank}(T_2))\), and provide a concrete example showing each inequality can be tight.

B.9. For an autoencoder with an encoder \(E: \mathbb{R}^n \to \mathbb{R}^k\) (linear, with rank \(k < n\)) and a decoder \(D: \mathbb{R}^k \to \mathbb{R}^n\) (linear), prove that the composition \(D \circ E\) cannot equal the identity map, even if \(D\) is chosen optimally.

B.10. Prove that a change-of-basis transformation \(A' = P^{-1}AP\) preserves rank: \(\text{rank}(A) = \text{rank}(A')\) for any invertible matrix \(P\).

B.11. Prove that if \(T: V \to W\) is an injective linear map between finite-dimensional spaces with \(\dim(V) = \dim(W)\), then \(T\) is surjective (and hence bijective).

B.12. For a regression problem with design matrix \(X \in \mathbb{R}^{n \times p}\) (with \(n > p\)) and response vector \(\mathbf{y} \in \mathbb{R}^n\), prove that if \(\text{rank}(X) = p\), then the least-squares solution \(\hat{\mathbf{w}} = (X^\top X)^{-1} X^\top \mathbf{y}\) exists and is unique.

B.13. Prove that for a linear projection \(P: V \to V\) satisfying \(P^2 = P\), the kernel and image are complementary subspaces: \(V = \ker(P) \oplus \text{im}(P)\) (direct sum).

B.14. Prove that two matrices \(A, B \in \mathbb{R}^{m \times n}\) have the same rank if and only if they are related by a left-multiply by an invertible matrix and a right-multiply by an invertible matrix, i.e., \(B = MAQ\) for invertible \(M \in \mathbb{R}^{m \times m}\) and \(Q \in \mathbb{R}^{n \times n}\).

B.15. For a deep linear network composed of weight matrices \(W_1 \in \mathbb{R}^{m_1 \times n}\), \(W_2 \in \mathbb{R}^{m_2 \times m_1}\), …, \(W_k \in \mathbb{R}^{m_k \times m_{k-1}}\), prove that \(\text{rank}(W_k \cdots W_2 W_1) \leq \min(\text{rank}(W_1), \text{rank}(W_2), \ldots, \text{rank}(W_k))\), and explain the implications for information bottlenecks in deep networks.

B.16. Prove that if \(T: V \to W\) is a surjective linear map between finite-dimensional spaces, then for any subspace \(U \subseteq W\), the preimage \(T^{-1}(U) = \{v \in V: T(v) \in U\}\) satisfies \(\dim(T^{-1}(U)) = \dim(V) - \dim(W) + \dim(U)\).

B.17. Prove that a linear map \(T: V \to W\) is an isomorphism (bijective linear map) if and only if \(\text{rank}(T) = \dim(V) = \dim(W)\) (assuming finite-dimensional spaces).

B.18. Prove that for the SVD decomposition \(A = U \Sigma V^\top\) of a matrix \(A \in \mathbb{R}^{m \times n}\), the rank of \(A\) equals the number of nonzero diagonal entries in the diagonal matrix \(\Sigma\), and that \(\text{rank}(A) \leq \min(m, n)\).

B.19. For two linear maps \(T_1, T_2: V \to W\), prove that \(\text{rank}(T_1 + T_2) \leq \text{rank}(T_1) + \text{rank}(T_2)\), and provide a neural network interpretation (composition of layer outputs).

B.20. Prove that for a finite-dimensional vector space \(V\), a linear map \(T: V \to V\) is invertible if and only if \(\ker(T) = \{\mathbf{0}\}\), and also if and only if \(\text{im}(T) = V\). (Show all three characterizations are equivalent.)

C. Python Exercises (20)

This exercise set comprises 20 comprehensive computational problems designed to deepen understanding of linear maps, matrices, rank structure, kernel/image decomposition, and their applications to machine learning. Each exercise is structured with five components—Task, Purpose, ML Link, Hints, and What Mastery Looks Like—providing a complete pedagogical experience: from concrete computational goals to alignment with modern machine learning practice. All exercises avoid providing code or expected outputs, instead describing what successful completion should demonstrate. Approximately 19 of 20 are explicitly ML-relevant, with all 20 exploring aspects of matrix transformations, rank computation, and kernel/image structure fundamental to understanding linear algebra in neural networks.

C.1. Implementing Kernel Computation via Row Reduction

Task: Implement a function that computes the kernel (null space) of a given matrix by performing row reduction to reduced row echelon form (RREF) and extracting the free variables. Your implementation should handle matrices of arbitrary dimension and handle numerical stability by treating near-zero entries as zero. The function should return a basis for the kernel as a set of column vectors, and also return the rank of the matrix as a side product of the computation.

Purpose: The kernel is a fundamental object in linear algebra, and computing it via row reduction is the standard algorithmic approach. Understanding how RREF encodes the kernel (through the positions of free variables and the dependencies they satisfy) is essential for both theoretical understanding and practical computation. This exercise develops the skill of translating the mathematical definition of kernel into a concrete algorithm that works with floating-point arithmetic.

ML Link: In neural networks, the kernel of the weight matrix determines what inputs are “invisible” to the layer—they are annihilated and produce zero output regardless of value. In autoencoders, understanding the kernel of the encoder helps diagnose why certain input directions cannot be recovered. In dimensionality reduction, the kernel of a projection matrix defines the directions being discarded. In regularized regression, the kernel grows (the effective rank decreases) as regularization strength increases, explaining the bias-variance trade-off.

Hints: Use Gaussian elimination with partial pivoting to compute the RREF, which improves numerical stability. After obtaining RREF, the columns corresponding to free variables can be used to construct basis vectors for the kernel by setting each free variable to 1 (one at a time) and solving for the pivot variables. Consider using NumPy’s linear algebra routines for intermediate validation (e.g., np.linalg.svd to compute rank), but implement the RREF computation yourself. Test your implementation on matrices with known kernels (e.g., rank-deficient matrices where you can manually verify the kernel).

What mastery looks like: Your implementation correctly computes the kernel for various matrix dimensions and ranks, including full-rank matrices (kernel is trivial), rank-deficient matrices (nontrivial kernel), and edge cases like the zero matrix. The returned basis vectors are linearly independent and satisfy the equation \(A\mathbf{k} \approx \mathbf{0}\) (within numerical precision). The rank computed from the RREF matches the rank from other methods (e.g., counting nonzero singular values). For a 5×8 matrix with rank 3, your function correctly returns a 4-dimensional kernel (since nullity = 8 − 3 = 5… wait, that’s wrong, nullity = 8 − 3 = 5, but let me recalculate: rank-nullity says dim = rank + nullity, so 8 = 3 + nullity means nullity = 5, so a 5-dimensional kernel basis). Code is efficient (O(m²n) for m×n matrix, typical for Gaussian elimination) and handles numerical edge cases gracefully.

C.2. Computing Image (Column Space) and Verifying Rank-Nullity

Task: Implement a function that computes the image (column space) of a matrix by identifying a maximal set of linearly independent columns. Use multiple methods: (1) QR decomposition with column pivoting, (2) SVD-based approach (keeping columns corresponding to nonzero singular values), and (3) row-reduction-based approach (identifying pivot columns). The function should return a basis for the image, the rank, and verify that the rank-nullity theorem holds by comparing rank + nullity = n (where n is the number of columns).

Purpose: The image represents the set of all possible outputs of the linear map, and understanding how to compute it is crucial for analyzing what a transformation can and cannot do. The three methods reveal different algorithmic perspectives: QR connects to orthogonalization, SVD connects to spectral properties, and row reduction connects to the deterministic elimination procedure. Verifying rank-nullity in code reinforces the deep connection between kernel and image and validates the implementation.

ML Link: In neural networks, the image of a layer determines the maximum expressivity of all subsequent layers—a layer with rank-100 output can only influence features in a 100-dimensional subspace, no matter how wide downstream layers are. In transfer learning, the image of intermediate layers (features extracted by pre-trained models) forms the feature space available for downstream tasks. In compression and autoencoders, the image of the encoder is the latent space, and its dimension (the rank) determines the information capacity of the bottleneck. Understanding image dimension guides architecture decisions: does your bottleneck have enough rank to represent the data’s intrinsic dimensionality?

Hints: For QR with column pivoting, NumPy provides scipy.linalg.qr(..., mode='economic', pivoting=True), which directly identifies independent columns. For SVD, use np.linalg.svd and count nonzero singular values (with a numerical threshold). For row reduction, relate pivot columns to the RREF: the original pivot columns (before row operations) form a basis for the image. Implement code to verify rank + nullity = num_columns for test matrices of different ranks. Handle the case where the matrix is rank-deficient, full-rank, and rank-one (a special case for understanding outer products and low-rank approximations).

What mastery looks like: All three methods return the same rank and identify the same image dimension. The basis vectors returned are indeed linearly independent and span the image (verify by checking that any column of the matrix is in their span). The rank-nullity verification passes for matrices of various sizes and ranks, from 3×3 matrices (small, hand-verifiable) to 100×50 matrices (larger, requiring robust algorithms). When you perturb the matrix slightly (e.g., add a small random matrix), the rank changes appropriately (should stay the same if below numerical noise threshold, should increase/decrease if perturbation is significant). The code runs efficiently even for tall/wide matrices (numerical methods should not blow up in complexity).

C.3. Kernel and Image Decomposition in Neural Network Layers

Task: For a given weight matrix \(W \in \mathbb{R}^{m \times n}\) representing a neural network layer, implement a function that decomposes the domain into kernel and a complementary subspace (whose image is the layer’s image). Compute bases for the kernel and the orthogonal complement of the kernel (the subspace of inputs that are “seen” by the layer). Verify that these two subspaces are orthogonal complements and that input space can be written as their direct sum. Then, simulate feeding vectors through the layer and verify that only the component in the orthogonal complement of the kernel contributes to the output.

Purpose: This exercise makes concrete the abstract decomposition \(\mathbb{R}^n = \ker(T) \oplus \ker(T)^\perp\). The kernel represents “blind spots” of the layer—directions along which it has zero sensitivity. The orthogonal complement represents the “visible” directions. Decomposing any input into kernel and visible components reveals what information the layer preserves and what it discards. This is fundamental to understanding information flow in deep networks.

ML Link: In deep learning, understanding kernel decomposition helps diagnose why layers are not learning. If the kernel is large relative to the input space, the layer is effectively compressing information severely—a potential bottleneck. If the kernel is nearly the entire input space (rank nearly 0), the layer is nearly non-functional. In interpretability, kernel analysis reveals which input directions are “blocked” by a layer. In regularization, methods that encourage low-rank matrices are implicitly expanding the kernel. In adversarial robustness, adversarial perturbations often exploit directions in the kernel (where they are “invisible” to downstream layers).

Hints: Use the SVD or the QR decomposition to compute orthonormal bases. The right singular vectors (from SVD) corresponding to nonzero singular values form an orthonormal basis for \(\ker(T)^\perp\); those corresponding to zero singular values form a basis for the kernel. Alternatively, compute the kernel basis, then compute an orthonormal basis for the kernel, then use Gram-Schmidt to extend it to an orthonormal basis of the full space. Test with simple examples: a rank-1 matrix (large kernel), a full-rank matrix (trivial kernel), a projection matrix (kernel = null space of projection, image = projection subspace).

What mastery looks like: For any input vector, your code correctly decomposes it into kernel and visible components. When you apply the matrix to the input, the kernel component contributes zero to the output, and the visible component produces the expected output. The kernel and orthogonal complement bases are indeed orthogonal (dot products are near-zero within numerical precision) and span the entire space (the union of their bases has full rank). For a 10×8 matrix with rank 5, the kernel has dimension 3 and the orthogonal complement has dimension 5, and together they cover all 8 input dimensions. Modifying the matrix (e.g., zeroing a row) correctly changes the kernel and image; your code adapts appropriately.

C.4. Matrix Rank Estimation and Low-Rank Approximation

Task: Implement a function that estimates the effective rank of a matrix using various methods: (1) counting nonzero singular values, (2) examining the decay of singular values (e.g., when does the relative drop stabilize?), (3) using a hard threshold (singular values below a threshold are considered zero), and (4) using a relative threshold (singular values below a fraction of the maximum singular value). Return the effective rank for each method, the full SVD decomposition, and visualize (as text output describing the singular value spectrum) how the singular values decay. Compare the estimates.

Purpose: Rank is central to understanding matrix structure, but in practice, numerical data often exhibits approximate rank—not exactly rank-deficient, but effectively low-rank (most singular values are small). Learning to estimate effective rank is essential for denoising, compression, and identifying the intrinsic dimensionality of data. Different thresholding strategies reveal different aspects: hard thresholds are absolute, relative thresholds adapt to the matrix scale, and decay analysis reveals natural “knees” in the spectrum.

ML Link: Neural network weight matrices often have structure: they are approximately low-rank initially (random initialization spreads weight uniformly) or become low-rank during training (as the network learns to reduce redundancy). Pruning is based on the observation that many weights are near-zero (the effective rank can be reduced significantly with small performance loss). Principal component analysis (PCA) reduces dimensionality by keeping only the top-k singular vectors—the effective rank determines how many components are needed. In recommender systems, user-item interactions are approximately low-rank (a few latent factors explain most variation), and low-rank matrix completion (filling in missing entries) leverages this structure. Effective rank estimation guides decisions: how many components should you keep in PCA? How aggressively should you prune a network?

Hints: Use np.linalg.svd to compute the full singular value decomposition. For decay analysis, compute the ratio of consecutive singular values: a large ratio indicates a “knee” where the spectrum transitions from significant to negligible. Consider plotting (or describing in text) sigma[i] versus i, and also the ratio sigma[i]/sigma[0] (normalized singular value decay). Test on matrices with known rank (constructed from low-rank factors), full-rank matrices, and noisy perturbed versions of low-rank matrices. Use different thresholds and report how many singular values survive each threshold.

What mastery looks like: For a matrix constructed as a rank-5 matrix plus small noise, your code correctly identifies the effective rank as approximately 5 (all four methods should agree or give reasons why they differ). For a full-rank matrix, the estimates are reasonable (perhaps identifying it as full-rank or suggesting it becomes full-rank after removing numerical noise). The decay analysis correctly identifies “knees” in the singular value spectrum (e.g., if you construct a matrix with singular values 10, 9, 8, 1, 0.1, 0.01, the decay analysis should flag a transition around the third or fourth singular value). Relative threshold with threshold = 0.01 * max(singular_values) correctly adapts to the matrix scale (a matrix with singular values 100× larger has adjusted threshold 100× larger, capturing the same rank structure). For a 100×50 matrix, your code computes and reports all 50 singular values efficiently.

C.5. Rank and Expressivity in Fully Connected Layers

Task: Implement a function that takes a neural network layer weight matrix and analyzes its expressivity by computing: (1) the rank of the matrix, (2) the maximum output dimensionality (which equals the rank), (3) the “information loss” (how many dimensions are projected away), and (4) a tolerance analysis (if the rank drops by 1 due to noise or regularization, how does expressivity change?). For a given hidden layer dimension and input dimension, determine the “safe” or “optimal” layer width to avoid bottlenecks. Implement a function that checks whether a sequence of layer widths in a deep network forms a bottleneck (width decreases, then increases, but some layers have rank less than their width).

Purpose: Neural network architecture design requires balancing expressivity (width) with computational cost and generalization. Understanding rank-limited expressivity helps justify design choices. A common mistake is designing a network where one low-rank layer creates a bottleneck that no subsequent architectures can overcome. This exercise trains the intuition for rank-limited capacity and bottleneck diagnosis.

ML Link: Modern neural network design relies heavily on understanding when layers are bottlenecks. In ResNets, skip connections allow information to bypass potentially low-rank layers, preserving capacity. In autoencoders, the bottleneck is intentional—you want a low-rank layer to force compression. In transformers, the hidden dimension is typically 4× the embedding dimension in the feed-forward network, ensuring no bottleneck. In knowledge distillation, a small student model (often with smaller rank) must match a large teacher—the challenge is that the student’s effective rank limits what it can represent. Analyzing rank helps explain why student networks trained on teachers sometimes fail to match performance: the student’s rank is insufficient to approximate the teacher’s function.

Hints: Compute rank using np.linalg.matrix_rank with an appropriate tolerance, or count nonzero singular values from np.linalg.svd. For bottleneck detection, compute the rank of consecutive layers and check if any rank is less than min(input_dim, output_dim). Consider the case of a deep convolutional network where you need to analyze the rank of the weight matrices when reshaped appropriately. Create synthetic weight matrices (random matrices, low-rank matrices, truncated SVD matrices) to test your bottleneck detection.

What mastery looks like: For a fully connected layer with weight matrix 128×784 (as in A.6), your code correctly reports the maximum output dimensionality as min(128, 784) = 128 (if full-rank) or less (if rank-deficient). For a network with widths [100, 50, 200, 300], your code identifies that a 50-dimensional bottleneck at layer 2 limits all subsequent layers to at most 50 effective dimensions, even though layers 3 and 4 are nominally wider. When you artificially reduce a layer’s rank (e.g., by truncating its SVD), your code correctly reports the reduced expressivity and warns that downstream layers are bottlenecked. For a well-designed ResNet-like network where widths are consistent (or gradually decrease), your code reports no problematic bottlenecks.

C.6. Analyzing Rank in Convolutional Layers via Implicit Matrices

Task: Implement a function that takes a convolutional layer (kernel size \(k \times k\), input channels \(c_{in}\), output channels \(c_{out}\)) and implicitly constructs the corresponding matrix representation (for a single spatial location, or for the full image via the vectorized convolution matrix). Analyze the rank of this implicit matrix. Then, explore how “grouped convolutions” (which reduce parameter sharing and potentially reduce rank) affect the rank. Compare the rank of a standard convolution to grouped convolutions and depthwise convolutions (extreme grouping where each group is 1 input channel).

Purpose: Convolutional layers are not just matrix multiplications, but the same principles of rank apply to their implicit matrix representations. Understanding how architectural choices (grouping, kernel size, number of channels) affect rank is crucial for designing efficient convolutional networks. This exercise bridges the gap between the linear algebra of matrices and the convolutional structure of modern CNNs.

ML Link: Efficient neural networks often use depthwise-separable convolutions, which decompose a standard convolution into depthwise (channel-wise) and pointwise (1×1) convolutions. This decomposition is motivated by reducing parameters, but it also has a rank implication: the overall operation is a product of two lower-rank operations, which can limit expressivity but also reduce overfitting risk. MobileNets and other efficient models rely on this trade-off. By analyzing rank, you can predict when depthwise-separable convolutions will and won’t be effective. Neural Architecture Search (NAS) often optimizes the number of groups in grouped convolutions; understanding rank’s role guides design choices. In distillation, a teacher convolution can be approximated by a low-rank factorization (e.g., via SVD of its implicit matrix), enabling compression.

Hints: For a 2D convolution with kernel size k×k, input channels c_in, output channels c_out, a straightforward approach is to construct the full Toeplitz-like matrix (the circulant or zero-padded convolution matrix), which can be large. Alternatively, use the fact that a convolution can be represented as matrix multiplication after Im2Col (image-to-column) transformation: inputs are rearranged into a matrix where each column is a receptive field, and the kernel weights form a matrix. The rank of the weight matrix (reshaped appropriately) gives insights into the convolution’s rank. For grouped convolutions, the matrix becomes block-diagonal or block-structured. Compute and compare ranks for different grouping configurations.

What mastery looks like: For a standard 3×3 convolution with 32 input channels and 32 output channels, you construct (or describe) the corresponding matrix and analyze its rank. For depthwise convolutions (groups = num_channels), you show that the rank structure is fundamentally different (sparser, block-diagonal). When you compare a standard conv and a depthwise-separable conv (depthwise + pointwise), you demonstrate how (or whether) the composition’s rank relates to the individual operation ranks. For a 1×1 convolution (which is essentially a fully connected layer applied independently at each spatial location), you show that the rank structure is simpler and fully-determined by the kernel shape. Your analysis correctly predicts when depthwise-separable convolutions can fully represent a standard convolution (requires the product of ranks ≥ original rank) versus when information loss occurs (product of ranks < original rank).

C.7. Change of Basis and Matrix Diagonalization (Preview)

Task: Implement a function that takes a matrix \(A\) and a change-of-basis matrix \(P\), and computes the similarity transformation \(A' = P^{-1}AP\). Verify that the rank is preserved, and explore how the numerical conditioning of \(A\) changes under different basis choices (compute the condition number \(\kappa(A) = \sigma_{max}/\sigma_{min}\)). Then, implement a function that attempts to find a basis where \(A\) is nearly diagonal (preview of eigendecomposition, Chapter 5). For symmetric matrices, use eigenvector decomposition; for general matrices, use SVD and discuss why perfect diagonalization may not be possible.

Purpose: The insight that the same linear map can be represented by different matrices (in different bases) is profound. Change of basis connects abstract linear maps to concrete computations. Understanding how basis choice affects numerical stability (conditioning) is crucial for numerical linear algebra. This exercise transitions from the theory of rank-preserving transformations to the computational reality that the same transformation can be easy or hard to compute depending on the basis.

ML Link: In deep learning, understanding change of basis (and eventually diagonalization) is essential for analyzing learned representations. When a network learns, it discovers basis changes in input space that make the data easier to work with. Principal Component Analysis (PCA) finds a basis (the principal components) where the data is decorrelated and variance-aligned, in this basis, the covariance matrix is diagonal. In dimensionality reduction, you’re changing to a basis where only the top-k basis vectors are important, discarding the rest. In adversarial training, adversarial perturbations often exploit certain basis directions; understanding change of basis helps identify (and defend against) such directions. Conditioning analysis is relevant to optimization: ill-conditioned loss landscapes (large ratio of max to min curvature) slow gradient descent, and basis changes can improve conditioning.

Hints: Use np.linalg.inv to compute \(P^{-1}\) (but be aware of numerical stability; consider using solver methods for ill-conditioned matrices). Verify \(P^{-1}AP\) against the formula. For condition number, use np.linalg.cond or compute it from singular values. For symmetric matrices, use np.linalg.eigh to get eigenvalues and eigenvectors; for general matrices, use np.linalg.svd. Construct a basis using eigenvectors and compute the similarity-transformed matrix; for symmetric matrices, it should be diagonal (or nearly so if numerically approximate). Test with well-conditioned and ill-conditioned matrices to see how conditioning varies with basis choice.

What mastery looks like: For a given matrix \(A\) and change-of-basis matrix \(P\), you correctly compute \(A' = P^{-1}AP\) and verify that \(\text{rank}(A') = \text{rank}(A)\). For a symmetric matrix, the basis change to eigenvectors produces a diagonal (or nearly diagonal) matrix; the diagonal entries are the eigenvalues. For a general matrix, you use SVD and explain why you cannot achieve perfect diagonalization (right singular vectors and left singular vectors generally differ). The condition number correctly captures the aspect of how “stretched” or “skewed” the matrix transformation is; basis changes can sometimes improve conditioning (e.g., from \(\kappa = 1000\) to \(\kappa = 10\) for certain matrices), and you demonstrate this. For MNIST or a similar dataset, you compute the PCA basis and show how the condition number improves when you remove low-variance principal components.

C.8. Regression Rank Analysis and Solution Uniqueness

Task: Implement a function that analyzes the design matrix \(X \in \mathbb{R}^{n \times p}\) of a regression problem (n observations, p features). Compute the rank of \(X\), the rank-nullity relationship, and determine whether the least-squares problem has a unique solution. If \(X\) is rank-deficient, analyze the null space and explain what combinations of features are redundant. Implement Ridge regression (which adds regularization to ensure invertibility) and compare the solutions of least-squares (if rank-full) and Ridge. Visualize (or describe) how the regularization parameter \(\lambda\) affects the conditioning of the problem and the solution stability.

Purpose: Regression is one of the most practically important applications of linear algebra. Understanding rank and rank-deficiency in the regression context bridges theory and application. Rank-deficiency can arise from multicollinear features (correlated features), missing data, or redundant measurements. Ridge regression is a practical solution that trades small bias for reduced variance. This exercise grounds linear algebra in a common ML problem.

ML Link: Linear regression is ubiquitous in ML (as a model, as a component of more complex models, and in analysis). Many real-world datasets have multicollinearity—features that are correlated or redundant. Ridge regression (and other regularization methods like Lasso, which also address rank-deficiency in a different way) is essential in practice. Understanding that regularization makes the problem “full-rank” (invertible) helps explain why it improves generalization: by penalizing large weights, you implicitly restrict the solution to a smaller, more stable subspace. In neural networks, weight decay (L2 regularization) is Ridge regression applied to the weight matrix. In generalized linear models and logistic regression, regularization serves the same role. In high-dimensional settings (p > n, more features than observations), rank-deficiency is guaranteed, and regularization is mandatory.

Hints: Use np.linalg.lstsq for least-squares with automatic handling of rank-deficiency. Compute the SVD of \(X\) to analyze rank. For rank-deficient cases, the SVD reveals the null space (right singular vectors corresponding to zero singular values). For Ridge regression, the normal equations become \((X^\top X + \lambda I) \hat{\mathbf{w}} = X^\top \mathbf{y}\); implement this directly. Compare solutions for different \(\lambda\) values. Test on simulated data with known rank structure (full-rank, rank-deficient, or multi-collinear features).

What mastery looks like: For a design matrix constructed from n=100 observations of p=10 features, your code correctly identifies whether the rank is 10 (full rank, unique LS solution) or less. If rank < p, you correctly identify the redundant features and compute a basis for the null space. When you apply Ridge regression with \(\lambda = 0\), it reduces to standard LS (for full-rank \(X\)); as \(\lambda\) increases, the conditioning improves (smaller condition number) and the solution becomes more stable (smaller parameter values). For highly multi-collinear data (e.g., two features that are nearly identical), your code detects this and Ridge with appropriate \(\lambda\) produces a stable, regularized solution. Comparing LS and Ridge solutions shows that LS (when it exists) has lower training error, but Ridge often has better test error due to regularization.

C.9. Kernel Methods and Feature Space Rank

Task: Implement a function that computes the kernel matrix \(K \in \mathbb{R}^{n \times n}\) (where \(K_{ij} = k(\mathbf{x}_i, \mathbf{x}_j)\) for a kernel function \(k\)) for a dataset of n points. Analyze the rank of the kernel matrix for different kernels (linear kernel, Gaussian (RBF) kernel, polynomial kernel) and different hyperparameters (e.g., bandwidth for Gaussian kernel). For each kernel, compute the eigendecomposition of \(K\) and analyze the spectrum (eigenvalues). Discuss how the kernel hyperparameter affects the rank and the spectrum.

Purpose: Kernel methods are a powerful set of ML algorithms (support vector machines, kernel PCA, Gaussian processes) that implicitly work in a high-dimensional (or infinite-dimensional) feature space. The kernel matrix encodes the geometry of the data in this feature space. The rank of the kernel matrix determines the effective dimensionality in the feature space. Understanding kernel matrix rank helps diagnose whether a kernel is appropriate for the dataset and how to tune hyperparameters.

ML Link: In kernel PCA, the kernel matrix’s eigendecomposition reveals the principal components in feature space; the rank (number of nonzero eigenvalues) determines the intrinsic dimensionality. In support vector machines (SVMs), the kernel matrix determines the geometry of the decision boundary; ill-conditioned kernels (poor rank properties) can lead to numerical instability in the SVM solver. In Gaussian processes, the kernel matrix is the covariance matrix; its rank and conditioning affect prediction uncertainty and computational cost. Gaussian (RBF) kernels with small bandwidth produce high-rank (nearly full-rank) kernel matrices, leading to overfitting (the model memorizes training data); increasing bandwidth reduces rank (smoothing the feature space). Polynomial kernels have rank determined by the polynomial degree; understanding this guides hyperparameter selection.

Hints: Implement kernels as functions (e.g., linear_kernel(x, y), rbf_kernel(x, y, sigma), polynomial_kernel(x, y, degree)). Construct the kernel matrix by evaluating pairwise kernels. Use np.linalg.eigh to compute eigenvalues and eigenvectors (kernel matrix is symmetric positive semi-definite). Plot or describe the spectrum (eigenvalue decay). Vary hyperparameters and observe how the rank and spectrum change. Test on datasets of different sizes and dimensionalities.

What mastery looks like: For a linear kernel, the rank of the kernel matrix equals the rank of the data matrix (rank of the feature space). For Gaussian kernels with large bandwidth (smooth), the kernel matrix has lower rank (eigenvalues decay quickly). For Gaussian kernels with small bandwidth (sharp), the kernel matrix is nearly full-rank (eigenvalues decay slowly). You can explain the trade-off: small bandwidth gives flexible models but may overfit; large bandwidth gives smoother models but may underfit. For a dataset with n=1000 points, polynomial kernels of degree 1, 2, 3 have different rank structures; you can show how degree affects rank. For kernel PCA on a real dataset (e.g., MNIST digits), your analysis shows how many components you need to retain (based on rank) to capture 95% of the variance in feature space.

C.10. Bottleneck Detection in Deep Linear Networks

Task: Implement a function that takes a list of weight matrices representing layers of a deep linear network \(T = T_k \circ \cdots \circ T_2 \circ T_1\) and determines where the bottlenecks are. A bottleneck occurs where the rank drops unexpectedly or where the effective capacity is limited. Compute the rank after each layer composition (product of matrices) and track how rank evolves. Implement a function that computes the “information content” at each layer (based on rank) and predicts the output rank given the input. For a given input, trace how information flows through the network: how many dimensions are “alive” after each layer?

Purpose: Deep networks (even without nonlinearity) can have bottleneck layers that constrain all downstream layers. Understanding where bottlenecks occur is essential for network design. This exercise develops the intuition that composition of low-rank layers produces limited rank, and that a single low-rank layer creates a permanent bottleneck.

ML Link: In practice, neural networks often develop implicit bottlenecks during training (due to regularization or the structure of the data). In distillation, bottleneck layers are deliberately introduced to force compression and improve model efficiency. In AutoML and neural architecture search, bottleneck avoidance is often a design constraint. In Transformers, the rank of attention maps and feed-forward layers determines the model’s effective capacity. Analyzing rank helps explain why some architectures (like ResNets with skip connections) are more effective than others: skip connections preserve information by bypassing low-rank layers.

Hints: Compute the rank of each weight matrix using SVD. Compute the product of matrices step by step and track how rank changes. Use the fact that \(\text{rank}(AB) \leq \min(\text{rank}(A), \text{rank}(B))\) to predict the composed rank. Test with synthetic deep networks: a sequence of full-rank matrices should maintain full rank, but a sequence with one rank-5 matrix among rank-100 matrices should bottleneck to rank-5. Visualize the rank progression layer by layer.

What mastery looks like: For a 5-layer deep network with weight matrices of shapes 100×100, 100×50, 50×100, 100×100, 100×100, your code correctly identifies the bottleneck at layer 2 (rank ≤ 50). After layer 2, all subsequent layers remain bottlenecked to rank ≤ 50, no matter how wide they are. You can show that the overall network rank is at most 50, no matter how many layers you add (if they are all after the bottleneck). For a network where ranks are [100, 90, 80, 70, 100], your code should identify that even though layer 4 widens again, the rank is still constrained by previous layers (≤ 70). For a well-designed network with consistent ranks (or gradual tapering), no unexpected bottlenecks appear.

C.11. Autoencoder Bottleneck and Reconstruction Error

Task: Implement a function that takes an encoder matrix \(E \in \mathbb{R}^{k \times n}\) and a decoder matrix \(D \in \mathbb{R}^{n \times k}\) (with \(k < n\)), and analyzes the composition \(F = DE\) (the autoencoder mapping input to reconstruction). Compute the rank of \(F\), compute the reconstruction error theoretically (as a function of the subspace lost), and verify that \(DE\) is a projection onto the image of \(E\). Explore how the choice of \(E\) and \(D\) affects reconstruction: for a fixed bottleneck dimension \(k\), what is the optimal choice of \(E\) and \(D\) to minimize reconstruction error for given data?

Purpose: Autoencoders are a fundamental unsupervised learning architecture. Understanding bottleneck dimension as a rank constraint reveals that autoencoders are forced compression: they cannot reconstruct information in directions orthogonal to the bottleneck’s image. This exercise connects autoencoder architecture to linear algebra and helps design autoencoders that preserve important information.

ML Link: Autoencoders are widely used for unsupervised representation learning, denoising, and compression. The bottleneck dimension is a hyperparameter that controls the trade-off between compression and reconstruction accuracy. Understanding that the bottleneck is a rank-k linear constraint means you can use linear algebra to analyze what information is lost. Variational Autoencoders (VAEs) add stochasticity to the bottleneck; the analysis still applies but with probabilistic interpretation. Denoising autoencoders add noise to inputs; understanding the rank structure helps explain why they work (noise is often in high-variance directions, and the bottleneck preserves low-variance structure). In image compression, autoencoders have been largely replaced by learned codecs (like JPEG 2000, HEIF, AV1), but the principles remain the same: the bottleneck determines the compression rate and reconstruction quality.

Hints: Construct \(E\) and \(D\) from an SVD decomposition of a data matrix (for optimal compression, \(E\) and \(D^\top\) should be the principal components and their complements). Compute \(F = DE\) and verify it is a projection (satisfies \(F^2 = F\)). The rank of \(F\) should be k. For a data matrix \(X\), the reconstruction \(\hat{X} = FX\) is the projection of \(X\) onto the column space of \(E^\top\) (or equivalently, onto the image of \(D\)). The reconstruction error is \(\|X - FX\| = \|X - \hat{X}\|\), which equals the norm of the components discarded (orthogonal directions).

What mastery looks like: For an encoder with k=50 and an original data dimension of n=784 (like MNIST), the composition \(DE\) has rank 50 and reconstructs images in a 50-dimensional subspace. The reconstruction error is perfectly explained by the discarded directions (784-50=734 dimensions). When you train an autoencoder on MNIST data (or use the optimal choice of \(E\) via PCA), you show that a bottleneck of k=50 preserves enough information for reasonable reconstruction, while k=10 is too small. You can quantitatively predict reconstruction error from the geometry of the bottleneck: images with variance concentrated in the top-k principal directions reconstruct better than images with variance spread across many directions.

C.12. Dimensionality Reduction via Low-Rank Approximation

Task: Implement Principal Component Analysis (PCA) from scratch using SVD. Given a data matrix \(X \in \mathbb{R}^{n \times d}\) (n samples, d features), compute the SVD, extract the top-k principal components, and project data onto this subspace. Analyze the rank, the cumulative explained variance (in terms of summed singular values squared), and the reconstruction error of the low-rank approximation \(X_k = U_k \Sigma_k V_k^\top\). Implement a function that chooses k automatically based on a variance threshold (e.g., retain sufficient components to explain 95% of variance).

Purpose: PCA is one of the most fundamental dimensionality reduction techniques. It leverages the low-rank structure (or approximate low-rank structure) of real data to reduce dimensions while retaining most variance. Understanding PCA through the lens of rank and SVD provides deep insight into why it works and when it’s applicable.

ML Link: PCA appears in many ML algorithms: as preprocessing (reducing dimensionality before classification), in representation learning (learned representations are often low-rank), in exploration and understanding of high-dimensional data (visualizing the top 2–3 principal components), and in combination with other methods (PCA preprocessing before kernel methods, PCA regularization to enforce low-rank structure). The choice of k is crucial: too small, and you lose important information; too large, and you get no compression or noise reduction. Understanding that k determines the rank of the low-rank approximation guides this choice. In modern deep learning, deep representation learning (autoencoders, VAEs, contrastive methods) are more flexible than PCA, but the core insight remains: learning a low-rank representation that preserves task-relevant information.

Hints: Compute the mean of the data and center it (subtract the mean). Use np.linalg.svd to compute the SVD of the centered data matrix. The principal components are the left singular vectors (or the right singular vectors of the covariance matrix). The explained variance is the squared singular values divided by (n-1). The cumulative explained variance is the cumsum of these variances normalized by total variance. Implement a function that chooses k based on a variance threshold (e.g., keeping the first k where cumsum >= 0.95 * total). Project data onto the first k principal components and compute reconstruction error.

What mastery looks like: For a dataset like MNIST (n=60000, d=784), PCA shows that the top 100 components explain ~95% of variance, meaning the data is effectively 100-dimensional (in its principal component representation), much lower than the nominal 784 dimensions. Projecting onto these 100 components and reconstructing preserves most of the visual information in the digits. When you vary k (10, 50, 100, 200), the trade-off between dimensionality reduction and reconstruction error is clear: k=10 is too aggressive (digits become blurry), k=100 is well-balanced, k=200 gets most of the details. For a synthetic low-rank matrix (constructed from rank-10 factors with added noise), PCA correctly identifies that k=10 components are sufficient for near-perfect reconstruction (once you account for the noise). The cumulative explained variance curve correctly transitions from rapid increase (first few components) to plateau (later components).

C.13. Singular Value Decomposition and Compression

Task: Implement SVD-based compression for matrices (e.g., image matrices). Given a matrix \(A \in \mathbb{R}^{m \times n}\), compute the SVD \(A = U \Sigma V^\top\), keep only the top-k singular values and their corresponding singular vectors, and reconstruct the low-rank approximation \(A_k = U_k \Sigma_k V_k^\top\). Measure compression (ratio of parameters in \(A_k\) compared to \(A\)) and reconstruction error (Frobenius norm \(\|A - A_k\|\)). Implement a function that chooses k to achieve a target compression ratio or target error, and show how compression and error trade off. Apply this to image compression (read an image, compress it, and report statistics).

Purpose: SVD is the optimal low-rank approximation in terms of Frobenius norm: \(A_k\) is the closest rank-k matrix to \(A\). Understanding how to use SVD for compression is practical and reinforces thetheory that rank determines structure. This exercise combines linear algebra with a practical application.

ML Link: Model compression is crucial for deploying ML models (e.g., on mobile devices, edge devices, or when serving thousands of requests per second). Low-rank factorization is one of the key compression techniques. For neural network weights, SVD-based low-rank compression can significantly reduce model size (e.g., from 1000×1000 to 100×100 + 100×1000, saving from 1M to 100k+100k parameters, a 5× reduction). In recommendation systems, low-rank matrix factorization is the standard approach for matrix completion (predicting missing user-item interactions). Knowing how to compress via SVD helps design efficient models and understand compression-accuracy trade-offs.

Hints: Use np.linalg.svd to compute full decomposition. For low-rank approximation, keep only the first k singular values, rows of \(U\), rows of \(\Sigma\), and columns of \(V^\top\). Compute the compression ratio as: original parameters (m×n) vs. compressed parameters (m×k + k + k×n). For image compression, reshape the image matrix (height × width × 3 for RGB, or height × width for grayscale) into a 2D matrix (rows = height × 3 or height, columns = width), apply SVD compression, reshape back, and display. Compute reconstruction error norms.

What mastery looks like: For a 1000×1000 matrix, keeping the top 100 singular values gives a 10× compression ratio (1M parameters down to 100k + 100k = 200k). The reconstruction error is proportional to the 101st singular value; if the spectrum decays quickly, error is small; if it decays slowly, error is significant. For an image, visual quality is preserved for moderate compression (e.g., keeping 50% of singular values), but becomes noticeable for aggressive compression (e.g., 5% of singular values). You can show the trade-off curves: compression ratio vs. reconstruction error, or compression ratio vs. visual quality (PSNR, SSIM metrics, or subjective assessment).

C.14. Operator Norm and Spectral Properties

Task: Implement a function that computes the operator norm of a matrix (the induced 2-norm, also called the spectral norm) as the largest singular value. Verify this by computing the supremum of \(\|A\mathbf{x}\| / \|\mathbf{x}\|\) over random vectors \(\mathbf{x}\). Explore how the operator norm relates to eigenvalues (for square symmetric matrices, the spectral norm equals the largest absolute eigenvalue). Analyze how regularization (e.g., spectral normalization, which constrains the spectral norm to be ≤ 1) affects the operator norm. Implement spectral normalization for a matrix and verify that the resulting normalized matrix has operator norm ≈ 1.

Purpose: The operator norm is a measure of how much a linear map can stretch vectors. It’s fundamental to understanding stability, conditioning, and convergence rates in optimization. Spectral normalization is widely used in modern neural networks (especially GANs) to stabilize training. This exercise connects the theoretical operator norm to computational methods and practical applications.

ML Link: In neural networks, the operator norm of weight matrices controls how information flows through layers. Large operator norms can lead to gradient explosion (in backpropagation, you multiply Jacobians of successive layers, and large norms compound). Small operator norms can lead to gradient vanishing. Spectral normalization constrains each layer to have operator norm ≤ 1, ensuring stable gradient flow. In GANs, spectral normalization of the discriminator’s weight matrices is a key technique for training stability. In Lipschitz-constrained learning (e.g., adversarial robustness), you constrain the Lipschitz constant (which equals the operator norm for linear maps); spectral normalization is the way to enforce this. Understanding operator norm helps diagnose and fix training instability.

Hints: Compute the spectral norm as np.linalg.svd(A)[1][0] (the largest singular value) or np.linalg.norm(A, ord=2). To verify, compute \(\|A\mathbf{x}\| / \|\mathbf{x}\|\) for many random \(\mathbf{x}\) and check that the maximum approaches the spectral norm. For spectral normalization, divide the matrix by its spectral norm: \(A_{normalized} = A / \|A\|_2\). For square symmetric matrices, verify that the spectral norm equals the largest absolute eigenvalue (use np.linalg.eigh). Test on matrices with known spectral norms (e.g., diagonal matrices, orthogonal matrices with norm 1, scaled versions).

What mastery looks like: For a random 10×10 matrix, the computed spectral norm matches the largest singular value exactly (within numerical precision) and also matches the maximum of \(\|A\mathbf{x}\| / \|\mathbf{x}\|\) over random test vectors. For a symmetric matrix, the spectral norm equals the largest absolute eigenvalue. After spectral normalization, the matrix has norm approximately 1.0. For deep neural networks, measuring the spectral norm of weight matrices before and after spectral normalization demonstrates the effect: unconstrained weights may have norms > 10 or < 0.1, while spectral normalization keeps them ≈ 1. Applying spectral normalization to the discriminator in a GAN can be empirically shown to improve training stability (loss curves are smoother, fewer divergences).

C.15. Ill-Conditioning and Numerically Stable Computations

Task: Implement a function that computes the condition number \(\kappa(A) = \sigma_{max} / \sigma_{min}\) (from the SVD) of a matrix. Analyze how condition number relates to the stability of computations: given a least-squares problem \(A\mathbf{x} = \mathbf{b}\), show how errors in \(\mathbf{b}\) are amplified in the solution \(\mathbf{x}\). Implement least-squares solvers using two methods: (1) normal equations \((A^\top A) \hat{\mathbf{x}} = A^\top \mathbf{b}\) (unstable for ill-conditioned matrices) and (2) QR decomposition (more stable). For ill-conditioned matrices, demonstrate that the normal equations may fail or give inaccurate results, while QR succeeds. Analyze the condition number of \(A^\top A\) versus \(A\).

Purpose: Numerical stability is crucial in practice. An algorithm that is theoretically sound may fail catastrophically if ill-conditioned problems are not handled carefully. Understanding condition number helps diagnose numerical instability and choose appropriate algorithms. This exercise bridges pure linear algebra with numerical linear algebra (Chapter 11 preview).

ML Link: In machine learning, ill-conditioning appears frequently. Datasets with highly correlated features lead to ill-conditioned design matrices; regression becomes numerically unstable without regularization. In optimization, ill-conditioned Hessians slow gradient descent; second-order methods must carefully handle ill-conditioning to avoid step size problems. In neural networks, weights that grow during training can become ill-conditioned, explaining why techniques like gradient clipping and normalization (batch norm, layer norm) help. Understanding condition number motivates regularization methods: Ridge regression, dropout, and weight decay all improve conditioning (or reduce the effective condition number of the problem).

Hints: Use np.linalg.cond to compute condition number directly, or compute it from singular values. Use np.linalg.svd(A, compute_uv=False) to get singular values. Solve LS problems using both (A.T @ A)^{-1} A.T @ b (normal equations) and np.linalg.qr(A) followed by backsolving. Create ill-conditioned test problems (e.g., a matrix with singular values [1, 0.1, 0.01, 0.001, …]) and compare solutions. Verify that cond(A.T @ A) ≈ cond(A)^2 (condition number squares with normal equations). Demonstrate that small perturbations in \(\mathbf{b}\) cause large perturbations in \(\mathbf{x}\) if \(\kappa(A)\) is large.

What mastery looks like: For a well-conditioned matrix (condition number ≈ 1, e.g., orthogonal matrix), normal equations and QR give nearly identical solutions. For an ill-conditioned matrix (condition number ≈ 1e6), normal equations may produce wildly inaccurate results (especially when \(\mathbf{b}\) has small perturbations), while QR remains stable. You can show specific numbers: for \(\kappa(A) = 1000\), a 1% error in \(\mathbf{b}\) can amplify to 10% error in \(\mathbf{x}\) (factor of 10 amplification). The relationship \(\text{cond}(A^T A) \approx \text{cond}(A)^2\) is verified numerically: if \(\kappa(A) = 100\), then \(\kappa(A^T A) \approx 10000\).

C.16. Rank Deficiency in Testing and Validation

Task: Implement a function that detects rank deficiency in a design matrix \(X\) (which includes a column of 1s for the intercept in regression). Analyze common sources of rank deficiency: collinear features, dummy variable traps (in one-hot encoding, one category should be dropped), and rank-deficient feature engineering (e.g., constructing features from a lower-dimensional basis). For each source, implement detection code and demonstrate how it manifests as rank deficiency. Implement a function that automatically detects and removes redundant features (by iteratively QR-factorizing and removing ones corresponding to zero entries on the diagonal).

Purpose: Rank deficiency is a practical problem in applied ML. Understanding where it comes from and how to detect it prevents models from failing silently or producing unreliable results. This exercise develops practical debugging skills.

ML Link: In real-world ML pipelines, rank deficiency is common. In tabular data, features may be derived from other features (multicollinearity). In one-hot encoding, using all categories instead of dropping one creates a dependency. In feature engineering, creating polynomial features can lead to inadvertent dependencies. Ridge regression and other regularization methods handle rank deficiency gracefully, but understanding what’s happening is important for model interpretation and feature engineering. In some cases, rank deficiency indicates a real redundancy in the data (dropping one feature is safe); in other cases, it indicates misspecified features (feature engineering needs rethinking).

Hints: Use QR decomposition with pivoting to identify which columns are linearly independent (columns corresponding to nonzero entries on the diagonal are independent). Use np.linalg.matrix_rank to detect rank-deficious and compare to the number of features. For feature engineering, check rank before and after adding engineered features; a rank-deficient increase indicates problematic features. Implement tests for the one-hot-encoding dummy variable trap: construct a one-hot encoded matrix with all categories, verify it’s rank-deficient, then drop one category and verify rank is now full.

What mastery looks like: Given a dataset with 100 samples and 50 features where 5 features are perfectly collinear (they are linear combinations of other features), your code correctly identifies that the rank is 45 (not 50). When you apply one-hot encoding to a categorical variable with 10 categories without dropping one, your code detects the rank deficiency. When you engineer polynomial features from a low-rank base (e.g., 10 base features used to construct 100 polynomial features), your code detects that the resulting matrix is rank ≤ 10 (or slightly higher due to interactions, if you include interaction terms). After applying feature removal (QR-based or other), the resulting design matrix is full-rank.

C.17. Matrix Factorization for Recommendation Systems

Task: Implement basic matrix factorization for collaborative filtering. Given a user-item interaction matrix \(R \in \mathbb{R}^{m \times n}\) (m users, n items, entries are ratings or interaction counts, with many missing values), factorize it as \(R \approx U V^\top\) where \(U \in \mathbb{R}^{m \times k}\) and \(V \in \mathbb{R}^{n \times k}\) (with \(k \ll \min(m, n)\)). Implement a function that: (1) initializes \(U\) and \(V\) randomly, (2) iteratively updates them to minimize reconstruction error on observed entries (e.g., using gradient descent or alternating least squares), and (3) predicts missing entries by computing \(U V^\top\). Analyze the rank \(k\) and explain the trade-off between expressivity (larger \(k\) fits training data better) and generalization (smaller \(k\) may generalize better).

Purpose: Matrix factorization is a foundational ML technique for collaborative filtering (recommender systems). Understanding why it works (the low-rank assumption: user-item interactions are driven by a small number of latent factors) and how to implement it requires deep engagement with matrix decomposition and optimization. This exercise connects linear algebra to a practical, impactful ML application.

ML Link: Many recommendation systems (Netflix, Amazon, Spotify) use matrix factorization or variants. The assumption that user preferences are determined by a small number of latent factors (e.g., “likes action movies”, “prefers comedies”) is expressed mathematically as low-rank factorization. The rank \(k\) controls how many factors to learn; it’s a critical hyperparameter. Larger \(k\) allows fitting training interactions more accurately but risks overfitting. Smaller \(k\) provides regularization but may underfit. Advanced techniques (factorization machines, neural collaborative filtering) build on this foundation. Analyzing rank eigenvalues helps understand model capacity: if only a few large singular values appear, the data is genuinely low-rank, and matrix factorization is appropriate; if singular values decay slowly, low-rank is a simplification, and more complex models may be needed.

Hints: Implement alternating least squares (ALS): iteratively optimize \(U\) holding \(V\) fixed (a least-squares problem for each user row), then optimize \(V\) holding \(U\) fixed (least-squares for each item row). For efficiency, avoid computing the full matrix product \(UV^\top\) (which is dense); instead, compute entries as needed. Regularize by adding a penalty term (Frobenius norm of \(U\) or \(V\)) to prevent overfitting. Implement evaluation metrics: RMSE on test interactions, or ranking metrics like NDCG (normalized discounted cumulative gain) if the task is ranking rather than rating prediction.

What mastery looks like: For a small simulated user-item matrix (100 users, 50 items, 500 observed ratings), your matrix factorization method learns low-rank factors and reconstructs the training data with low error. When you hold out test ratings, the method predicts them reasonably (depending on rank and regularization). Increasing rank from 5 to 10 to 20 shows improving training error but potentially degrading test error (overfitting); you can identify the sweet spot for \(k\). For a real dataset (e.g., MovieLens), your method produces predictions and you can compute RMSE or other metrics. You can analyze the learned factors \(U\) and \(V\): do the users and items cluster based on the learned factors? Do the factors correspond to interpretable concepts (e.g., “comedy-lovers”, “action-fans”)?

C.18. Inverse and Pseudo-Inverse Computation and Stability

Task: Implement a function that computes the pseudo-inverse (Moore-Penrose inverse) \(A^\dagger\) of a matrix \(A\) using SVD: \(A^\dagger = V \Sigma^\dagger U^\top\) where \(\Sigma^\dagger\) is the diagonal matrix of reciprocals of nonzero singular values. Compare the pseudo-inverse to the standard inverse (for square full-rank matrices, they should agree). Analyze the condition number of the pseudo-inverse and explore numerical stability. Implement overdetermined and underdetermined least-squares problems (\(Ax = b\) with more equations than unknowns, or vice versa) and show how the pseudo-inverse finds optimal solutions in both cases.

Purpose: The pseudo-inverse is the generalization of the matrix inverse to rank-deficient and non-square matrices. It’s essential for solving least-squares problems in a mathematically clean way. Understanding its computation via SVD reinforces the power of SVD and its connection to fundamental linear algebra concepts like kernel and image.

ML Link: Least-squares is ubiquitous in ML (linear regression, least-squares SVM, neural network training with MSE loss). The pseudo-inverse directly solves LS problems; it’s the theoretical foundation. While optimization algorithms (gradient descent, Newton’s method) are more practical for large-scale problems, the pseudo-inverse provides intuition (finding the solution with minimum norm in underdetermined cases). In numerical linear algebra, understanding pseudo-inverse stability (via singular value structure) guides algorithm selection and regularization choices.

Hints: Use np.linalg.svd to compute \(U\), \(\Sigma\), \(V^\top\). Create \(\Sigma^\dagger\) by inverting nonzero singular values and setting zero values (or near-zero using a threshold) to zero. Use np.linalg.pinv to check your results. For full-rank square matrices, verify that the computed pseudo-inverse equals the standard inverse. For rank-deficient matrices, show that \(AA^\dagger A = A\) (part of the definition). For underdetermined LS (\(m < n\)), show that the pseudo-inverse solution \(\hat{\mathbf{x}} = A^\dagger \mathbf{b}\) is the solution with minimum norm.

What mastery looks like: For a full-rank square 5×5 matrix, your pseudo-inverse computes matches np.linalg.inv to high precision. For a rank-deficient 5×5 matrix (rank 3), your pseudo-inverse produces a matrix satisfying \(AA^\dagger A = A\) and \(A^\dagger AA^\dagger = A^\dagger\) (the Moore-Penrose conditions). For an overdetermined LS problem (10 equations, 5 unknowns), the pseudo-inverse solution matches the least-squares solution from QR decomposition. For an underdetermined LS problem (5 equations, 10 unknowns), the pseudo-inverse solution is the minimum-norm solution (verify: compute the norm and show that other solutions have larger norm).

C.19. Jacobian Matrix Analysis in Optimization

Task: Implement a function that computes the Jacobian matrix \(J \in \mathbb{R}^{m \times n}\) of a vectorial function \(\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m\) (computed either via numerical differentiation using finite differences, or analytical differentiation if you can derive it). Analyze the rank of the Jacobian and explain how it relates to local invertibility (implicit function theorem). Implement this for a neural network layer: the Jacobian is the matrix of partial derivatives of layer outputs with respect to layer inputs. Analyze how the Jacobian’s rank changes during neural network training, and relate rank changes to gradient flow (full-rank Jacobian means gradients flow cleanly; low-rank Jacobian indicates vanishing gradients or redundancy in layer outputs).

Purpose: The Jacobian is the linear approximation of a nonlinear function. Its rank determines whether the function is locally invertible and how sensitive outputs are to inputs. Understanding Jacobian rank is crucial for optimization, sensitivity analysis, and neural network training dynamics. This exercise bridges nonlinear functions (which are the domain of Chapter 15+) with linear algebra.

ML Link: In neural networks, backpropagation computes gradients by multiplying Jacobians of successive layers (chain rule). The Jacobian rank of each layer determines how well gradients propagate backward. Low-rank Jacobians lead to vanishing gradients (eigenvalues shrink with depth). Modern architectures (ResNets, batch normalization, layer normalization) address this by controlling Jacobian rank and spectrum. In sensitivity analysis (understanding how model predictions respond to input perturbations), the Jacobian’s singular values reveal sensitive and insensitive directions. In adversarial robustness, adversarial examples are crafted in the direction of maximum sensitivity (top left singular vector of the Jacobian). Understanding Jacobian rank guides design of robust models.

Hints: Implement numerical differentiation using finite differences: \(\frac{\partial f_i}{\partial x_j} \approx (f_i(\mathbf{x} + \epsilon e_j) - f_i(\mathbf{x})) / \epsilon\) (where \(e_j\) is the j-th unit vector). For neural networks, implement the forward-backward pass to compute Jacobians. Analyze the Jacobian rank using SVD. For a simple function like \(f(x) = \tanh(Wx)\) (single hidden layer), compute the Jacobian and rank as a function of \(W\) and see how the rank depends on weight magnitude and initialization.

What mastery looks like: For a simple function \(f(x) = (x_1^2, x_1 x_2)\) at a point \((1, 1)\), your Jacobian computation gives the correct matrix \(\begin{pmatrix} 2 & 0 \\ 1 & 1 \end{pmatrix}\) with rank 2 (full column rank if f is \(\mathbb{R}^2 \to \mathbb{R}^2\), in which case it’s locally invertible). For a neural network with a single hidden layer, training dynamics show how the Jacobian rank (related to the hidden layer’s rank) evolves: early in training, it may be low-rank; with proper initialization and training, it should increase (or stay reasonably full). For a deep neural network, you show how rank decreases with depth (vanishing gradient problem) and how architectural choices (skip connections, normalization) mitigate it.

C.20. Understanding Generalization Through Rank and Capacity

Task: Implement a function that analyzes the “capacity” of a linear model based on the rank of its weight matrix. Given a training and test dataset, fit a linear regression model (or a simple neural network with controlled rank) of various ranks (by constraining or regularizing the weight matrix rank). For each rank, measure: (1) training loss, (2) test loss, (3) generalization gap (test loss - training loss), (4) the calculated regularization penalty. Plot or describe how these metrics vary with rank. Show the U-shaped generalization curve: very low rank (high bias, underfitting) has poor training and test loss; medium rank is optimal; high rank (high variance, overfitting) has low training loss but high test loss.

Purpose: Understanding the bias-variance trade-off at the level of rank provides intuition for model selection and regularization. Low-rank models have high bias (simple, cannot fit complex patterns) but low variance (stable across datasets). High-rank models have low bias (flexible) but high variance (overfit). This exercise makes this trade-off concrete and quantifiable.

ML Link: Deep in modern ML lies this fundamental trade-off: more parameters (higher rank) enable fitting complex patterns, but risk overfitting. Regularization (L2 regularization as Ridge regression, or rank constraints) reduces complexity, improving generalization. Understanding the rank-capacity trade-off justifies many modern techniques: early stopping (cut training before the model reaches maximum rank), dropout (reduces effective rank by stochastically removing connections), batch normalization (adapts rank/scale dynamically), and weight decay (implicitly reduces rank). For large datasets, high rank is affordable; for small datasets, aggressive regularization (low rank) is necessary. Analyzing rank helps debug models: is your model underfitting or overfitting? Underfitting suggests rank is too low; overfitting suggests rank is too high.

Hints: For linear models, directly constrain or regularize rank using nuclear norm regularization (sum of singular values, approximates rank) or explicit rank truncation via SVD. For neural networks, use L2 regularization (weight decay) to implicitly reduce rank. Implement variants: no regularization (full rank), L2 regularization with various \(\lambda\), and explicit rank constraints (keep only top-k singular values in weights). Measure generalization gaps and optimal rank for different dataset sizes and complexities.

What mastery looks like: For a simple problem (e.g., polynomial regression on synthetic data), your analysis shows a clear U-shaped generalization curve: rank 1 is underfitting (high bias, poor fit), rank 5 is well-fit, rank 20 is overfitting (high variance, gap between train and test loss). For a real dataset like MNIST, you compare a low-rank model (e.g., linear classification with rank = num_classes) to high-rank (full-capacity neural network): low-rank achieves high training loss but is stable; high-rank achieves low training loss but poor test (shows overfitting for small data, or good generalization if data is large). By varying dataset size, you show that optimal rank scales with data (more data = can afford higher rank). Regularization strength effectiveness is demonstrated: weak regularization (small \(\lambda\)) allows overfitting; strong regularization (large \(\lambda\)) prevents it; intermediate \(\lambda\) is optimal.

Solutions

Solutions to A. True / False

A.1. For any linear map \(T: \mathbb{R}^5 \to \mathbb{R}^3\), the kernel of \(T\) is guaranteed to be nontrivial (i.e., contains more than just the zero vector).

Final Answer: TRUE

Full Mathematical Justification: This statement is a direct consequence of the rank-nullity theorem. For a linear map \(T: \mathbb{R}^5 \to \mathbb{R}^3\), we have \(\dim(\mathbb{R}^5) = \text{rank}(T) + \text{nullity}(T)\), which gives \(5 = \text{rank}(T) + \text{nullity}(T)\). Since the codomain is \(\mathbb{R}^3\), the image is a subspace of \(\mathbb{R}^3\), so \(\text{rank}(T) = \dim(\text{im}(T)) \leq \dim(\mathbb{R}^3) = 3\). Therefore, \(\text{nullity}(T) = 5 - \text{rank}(T) \geq 5 - 3 = 2\). This means the kernel contains at least a 2-dimensional subspace, so it must be nontrivial (larger than \(\{\mathbf{0}\}\)).

Counterexample if False: Not applicable; the statement is true.

Comprehension: Understanding this statement requires recognizing that rank cannot exceed the dimension of the codomain, and that rank-nullity then forces the nullity (kernel dimension) to be large. The key insight is that whenever you map a higher-dimensional space to a lower-dimensional space, information is necessarily lost, and that lost information is captured in the kernel.

ML Applications: In autoencoders with bottlenecks, if you compress from a 784-dimensional input to a 50-dimensional hidden layer (and then expand back), the information loss is inevitable—certain input directions are “invisible” to the bottleneck. In dimensionality reduction by projection onto a subspace, the kernel of the projection operator represents the directions being discarded. In neural networks that compress information, the kernel represents features the network is learning to ignore.

Failure Mode Analysis: Failing to recognize that a map from high-dimensional to low-dimensional spaces must have a nontrivial kernel can lead to misunderstanding model capacity. For example, if you design an autoencoder with a bottleneck smaller than the input, you might expect perfect reconstruction; the existence of a nontrivial kernel explains why this is impossible (for linear bottlenecks).

Traps: A common misconception is that a structured design (e.g., careful weight initialization or specific architecture) can overcome the inherent dimensionality constraint. No amount of engineering can make a linear bottleneck recover information lost in the lower-dimensional projection. Another trap is confusing “nontrivial kernel” with “large kernel”—here we only guarantee dimension ≥ 2, not that the kernel is “most” of the space.

A.2. If two matrices \(A\) and \(B\) have the same rank, then they represent the same linear map in some choice of bases.

Final Answer: FALSE

Full Mathematical Justification: Rank is a single scalar: it tells you the dimension of the image (and by rank-nullity, implicitly the kernel dimension). However, a linear map is characterized by much more structure than just its rank. Two maps with the same rank can differ in their detailed action—not just the dimensions of kernel/image, but how the map acts on different directions. For instance, the identity map on \(\mathbb{R}^3\) and a 90-degree rotation about the z-axis both have rank 3, but they are fundamentally different maps. Even in different bases, they represent different geometric transformations. Similarity (which is the relationship between matrices representing the same map in different bases) requires not just rank equality, but actual similarity: \(B = P^{-1}AP\) for invertible \(P\). Rank alone does not determine similarity; you also need the spectrum (eigenvalues, or more generally, the Julia form for non-diagonalizable matrices).

Counterexample if False: Consider the identity map \(I_3: \mathbb{R}^3 \to \mathbb{R}^3\) with matrix representation \(I = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}\) (rank 3) and a 90-degree rotation \(R = \begin{pmatrix} 0 & -1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}\) (rank 3). Both have rank 3, but they represent different maps. Any change-of-basis transformation \(P^{-1}RP\) will still be a rotation (with eigenvalues on the unit circle in the complex plane), while \(I\) has eigenvalues all equal to 1. Since similarity preserves eigenvalues, no change of basis can make \(I\) similar to \(R\).

Comprehension: Rank is a “coarse” invariant that only captures dimensionality of image and kernel. It ignores the geometric or spectral structure of the map. Just as two different terrain maps of the same region can show different features even though they have the same “dimension,” two maps can have the same rank but different qualitative behavior.

ML Applications: In neural networks, two layers with the same rank might have completely different learned features or weight distributions. Rank alone does not tell you what the layer computes. In transfer learning, a pre-trained network and a randomly initialized network might both be rank-full, but they extract very different features. In adversarial robustness, a neural network that is rank-deficient due to pruning might be vulnerable to different types of adversarial examples than an unpruned version, even if both the pruned and unpruned versions (if they had the same rank) might be expected to be similar by naive reasoning.

Failure Mode Analysis: Assuming rank equivalence implies similarity is a critical error in theoretical analysis. It would lead to incorrect conclusions about, for example, whether different neural network initializations can be transformed into each other via basis changes (they cannot, generally, even if they have the same rank).

Traps: The equivalence that is true is: “Two matrices represent the same linear map in different bases if and only if they are similar” (Definition 14). Do not confuse rank equality with similarity. Rank is just one number; similarity is a much more restrictive relationship.

A.3. The composition of two injective linear maps is always injective, but the composition of two surjective linear maps is not always surjective.

Final Answer: FALSE (The statement is partly true, partly false; it’s phrased as a conjunction, so it’s false overall.)

Full Mathematical Justification: The first claim is true: if \(T_1: U \to V\) and \(T_2: V \to W\) are injective, then \(T_2 \circ T_1\) is injective. Proof: suppose \((T_2 \circ T_1)(\mathbf{u}_1) = (T_2 \circ T_1)(\mathbf{u}_2)\). Then \(T_2(T_1(\mathbf{u}_1)) = T_2(T_1(\mathbf{u}_2))\). Since \(T_2\) is injective, \(T_1(\mathbf{u}_1) = T_1(\mathbf{u}_2)\). Since \(T_1\) is injective, \(\mathbf{u}_1 = \mathbf{u}_2\). So composition is injective.

However, the second claim is entirely false. The composition of surjective maps is always surjective (when it’s defined). Proof: if \(T_1: U \to V\) and \(T_2: V \to W\) are both surjective, then for any \(\mathbf{w} \in W\), there exists \(\mathbf{v} \in V\) with \(T_2(\mathbf{v}) = \mathbf{w}\) (by surjectivity of \(T_2\)). And there exists \(\mathbf{u} \in U\) with \(T_1(\mathbf{u}) = \mathbf{v}\) (by surjectivity of \(T_1\)). Thus \((T_2 \circ T_1)(\mathbf{u}) = T_2(T_1(\mathbf{u})) = T_2(\mathbf{v}) = \mathbf{w}\), so the composition is surjective.

Counterexample if False: The statement claims composition of surjective maps is “not always” surjective. Counterexample: let \(T_1: \mathbb{R}^3 \to \mathbb{R}^2\) be \(T_1(\mathbf{x}) = \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}\) (projection onto first two coordinates, surjective). Let \(T_2: \mathbb{R}^2 \to \mathbb{R}^2\) be \(T_2(\mathbf{y}) = \mathbf{y}\) (identity, surjective). Then \(T_2 \circ T_1: \mathbb{R}^3 \to \mathbb{R}^2\) is \((T_2 \circ T_1)(\mathbf{x}) = \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}\), which is surjective (any point in \(\mathbb{R}^2\) is the image of some point in \(\mathbb{R}^3\)). In fact, any composition of surjective maps is surjective.

Comprehension: The confusion likely arises from conflating injectivity and surjectivity. For injectives, composition preserves injectivity. For surjectives, composition also preserves surjectivity. The asymmetry in the problem statement is unwarranted; both composition properties work the same way (preservation of the property).

ML Applications: In neural networks, if each layer is surjective (onto its output space), the entire network is surjective. If each layer is injective, the network is injective. In information flow, surjectivity means every output is reachable; losing surjectivity at any layer means some outputs become unreachable downstream.

Failure Mode Analysis: Misunderstanding composition of surjections can lead to incorrect analysis of information flow in deep networks. If you believe surjectivity is not preserved under composition, you might incorrectly predict that information loss is inevitable, even though chain of surjections remains surjective.

Traps: The statement’s phrasing (“not always”) suggests a nuance. But there is no nuance here for surjections: composition always preserves surjectivity. The statement seems designed to trick you into second-guessing correct intuition.

A.4. If the kernel of \(T: V \to W\) is one-dimensional and the image of \(T\) is two-dimensional, then \(\dim(V) = 3\).

Final Answer: TRUE

Full Mathematical Justification: This is a direct application of the rank-nullity theorem: \(\dim(V) = \text{rank}(T) + \text{nullity}(T)\). The rank of \(T\) is the dimension of the image, which is given as 2. The nullity of \(T\) is the dimension of the kernel, which is given as 1. Therefore, \(\dim(V) = 2 + 1 = 3\). No ambiguity here—rank-nullity is a fundamental theorem that always holds for finite-dimensional spaces.

Counterexample if False: Not applicable; the statement is true.

Comprehension: This statement tests understanding of the rank-nullity theorem, which is central to linear algebra. The key is recognizing that rank (dimension of image) and nullity (dimension of kernel) must add up to the domain dimension, not the codomain dimension.

ML Applications: In autoencoders, if you have an input of dimension 784 and a bottleneck of dimension 50, the information loss (dimension of directions projected out) is 784 - 50 = 734. By rank-nullity, if the encoder is a linear map from \(\mathbb{R}^{784}\) to \(\mathbb{R}^{50}\), the kernel dimension is 784 - 50 = 734. In neural networks, understanding how information is partitioned into “preserved” (image) and “lost” (kernel) guides architecture design.

Failure Mode Analysis: Applying rank-nullity incorrectly (e.g., confusing domain and codomain) is a common error. Some might mistakenly think \(\dim(V) = \dim(W) =\) something, but the theorem constrains \(\dim(V)\), not \(\dim(W)\).

Traps: Be careful to apply rank-nullity correctly: \(\dim(\text{domain}) = \text{rank} + \text{nullity}\), not \(\dim(\text{codomain}) =\) anything specific.

A.5. A linear map \(T: \mathbb{R}^n \to \mathbb{R}^n\) is invertible if and only if \(\det(A) \neq 0\), where \(A\) is any matrix representation of \(T\).

Final Answer: TRUE

Full Mathematical Justification: Both directions: (1) If \(T\) is invertible, then it’s bijective (one-to-one and onto). This means its kernel is trivial and its image is all of \(\mathbb{R}^n\), so rank = n. By rank-nullity, nullity = 0. The rank-nullity result implies that the matrix \(A\) has full rank, which means all rows and columns are linearly independent, which is equivalent to \(\det(A) \neq 0\). (2) Conversely, if \(\det(A) \neq 0\), then \(A\) is invertible (has a matrix inverse), which means the linear map it represents is bijective, hence invertible.

Counterexample if False: Not applicable; the statement is true.

Comprehension: The determinant is a scalar invariant that captures whether a square matrix is invertible. Zero determinant means singular (linearly dependent rows/columns), non-zero means all rows/columns are independent and invertible. Additionally, \(\det(A)\) is independent of the choice of basis (similarity preserves determinant up to sign in some conventions, but actually \(\det(P^{-1}AP) = \det(A)\) always), so the condition makes sense.

ML Applications: In optimization, the Hessian (second derivative matrix) must be non-singular (positive definite or negative definite) for a critical point to be a strict local extremum. In neural networks, weight matrices with zero determinant have zero gradient flow in some directions (rank deficiency). Regularization (weight decay) prevents weights from becoming singular.

Failure Mode Analysis: Forgetting that the determinant is basis-independent is a subtle error. The statement says “\(A\) is any matrix representation,” emphasizing that the condition \(\det(A) \neq 0\) is the same regardless of basis choice (similarity transformation).

Traps: Be aware that for non-square matrices, determinant is not defined. This statement specifically restricts to \(\mathbb{R}^n \to \mathbb{R}^n\) (square case) for exactly this reason. For non-square maps, “invertible” is defined only for bijections (which is impossible if domain and codomain have different dimensions).

A.6. For a neural network with a fully connected layer having weight matrix \(W \in \mathbb{R}^{128 \times 784}\) (128 output units, 784 input features), the maximum information capacity of the layer is at most 128 dimensions, regardless of the data distribution.

Final Answer: TRUE

Full Mathematical Justification: The image of the linear map \(\mathbf{y} = W\mathbf{x}\) is a subspace of \(\mathbb{R}^{128}\) (the output space). The dimension of this image is at most \(\min(\text{rank}(W), 128) \leq 128\). So the output can access at most 128 dimensions. If data is high-dimensional but the layer outputs only 128-dimensional vectors, that output space is at most 128-dimensional. The subsequent nonlinearity (ReLU, etc.) cannot increase dimensionality beyond the 128-dimensional subspace that \(W\) produces—it can only apply a nonlinear transformation within that space or discard dimensions, not create new ones.

Counterexample if False: Not applicable; the statement is true.

Comprehension: This statement reflects the fundamental principle that rank (dimensionality of output) is bounded by the smaller of the input and output dimensions. A 128×784 matrix can map \(\mathbb{R}^{784}\) into at most a 128-dimensional subspace of \(\mathbb{R}^{128}\) (which is all of \(\mathbb{R}^{128}\) if the rank is 128, a proper subspace otherwise).

ML Applications: In convolutional neural networks, the number of output channels determines the maximum dimensionality of intermediate representations. In transfer learning, pre-trained layers bottleneck subsequent task-specific layers: if a pre-trained layer outputs 2048-dimensional features, subsequent layers cannot access more than 2048 dimensions of the input space. In knowledge distillation, a student network limited by fewer hidden units (smaller output dimension) can only represent a subset of the teacher’s function space.

Failure Mode Analysis: Assuming that with sufficient training data and optimization, a low-rank layer can somehow represent higher-dimensional features is a misconception. The rank bound is hard: data distribution cannot overcome it. No matter how clever the training, a 128-dimensional output is constrained to \(\mathbb{R}^{128}\).

Traps: Confusing the layer width (stated as 128 output units) with the layer’s effective rank (which could be less than 128 if weights become rank-deficient). The statement gives an upper bound (at most 128), not a promising that the layer achieves 128 dimensions—it might achieve fewer if regularization or initialization causes low-rank weights.

A.7. If \(T: V \to V\) is a linear map on a finite-dimensional space such that \(\text{rank}(T) = \dim(V) - 1\), then \(T\) cannot be invertible.

Final Answer: TRUE

Full Mathematical Justification: For \(T: V \to V\), invertibility requires the map to be bijective, which means it’s both injective and surjective. Injectivity requires trivial kernel, i.e., nullity = 0. Surjectivity requires the image to be all of \(V\), i.e., rank = \(\dim(V)\). By rank-nullity, \(\dim(V) = \text{rank} + \text{nullity}\). If rank = \(\dim(V) - 1\), then nullity = 1, so the kernel is one-dimensional (nontrivial). This means the map is not injective, hence not invertible.

Counterexample if False: Not applicable; the statement is true.

Comprehension: The nontrivial kernel makes inversion impossible. A map that annihilates an entire one-dimensional subspace cannot be inverted: two distinct vectors (orthogonal in some sense) that differ by a kernel element map to the same output, so the map is not one-to-one.

ML Applications: In neural networks, if a hidden layer has rank one less than its width, the layer’s output values lie in a proper subspace; the layer cannot be “recovered” by a later inverse layer, even if such a layer exists. Information is lost and unrecoverable.

Failure Mode Analysis: Beware of off-by-one errors: rank = \(\dim(V) - 1\) is not full rank; it’s rank-deficient by one dimension.

Traps: Comparing rank-theoretic invertibility (bijection) with matrix-based invertibility (\(\det(A) \neq 0\)). Both are equivalent for square matrices, but the statement uses the rank-theoretic definition.

A.8. A projection operator \(P: V \to V\) satisfying \(P^2 = P\) has the property that \(\ker(P)\) and \(\text{im}(P)\) are complementary subspaces (i.e., \(V = \ker(P) \oplus \text{im}(P)\)).

Final Answer: TRUE

Full Mathematical Justification: A projection (idempotent operator, \(P^2 = P\)) decomposes the space into kernel (annihilated directions) and image (preserved directions). Claim: these two are complementary (direct sum). Proof: (1) They are disjoint: if \(\mathbf{v} \in \ker(P) \cap \text{im}(P)\), then \(P\mathbf{v} = \mathbf{0}\) (kernel) and \(\mathbf{v} = P\mathbf{u}\) for some \(\mathbf{u}\) (image). Then \(\mathbf{v} = P\mathbf{u} \Rightarrow P\mathbf{v} = P^2\mathbf{u} = P\mathbf{u} = \mathbf{v}\) (using idempotence). But also \(P\mathbf{v} = \mathbf{0}\), so \(\mathbf{v} = \mathbf{0}\). (2) They span: for any \(\mathbf{v} \in V\), decompose \(\mathbf{v} = P\mathbf{v} + (\mathbf{v} - P\mathbf{v})\). The first term is in image (since \(P^2\mathbf{v} = P\mathbf{v}\)), and the second is in kernel (since \(P(\mathbf{v} - P\mathbf{v}) = P\mathbf{v} - P^2\mathbf{v} = P\mathbf{v} - P\mathbf{v} = \mathbf{0}\)). Thus every vector decomposes as kernel + image. This proves direct sum.

Counterexample if False: Not applicable; the statement is true.

Comprehension: Projections are fundamental in linear algebra because they split space into two independent pieces: “seen” (image, what the projection preserves) and “discarded” (kernel, what the projection annihilates). The idempotent property (\(P^2 = P\)) is precisely what ensures this clean decomposition.

ML Applications: In data analysis, projecting onto the span of principal components (PCA) discards variance-minimized directions (in the kernel of the projection) while preserving variance-maximized directions (in the image). In neural networks, layer normalization can be viewed as a projection to the affine subspace perpendicular to the mean. Feature extraction via autoencoders projects inputs onto a bottleneck subspace.

Failure Mode Analysis: Forgetting the idempotent property can lead to incorrect decomposition. If an operator satisfies \(P^2 \neq P\) but is otherwise projection-like, the complementarity breaks.

Traps: Be careful to distinguish projections (idempotent) from reflections or other orthogonal transformations. Orthogonal matrices are not idempotent (unless they’re projections themselves or identity/zero). Projections satisfy the specific algebraic property \(P^2 = P\), which forces the complementary decomposition.

[Continuing with A.9 through A.20 to complete the solutions…]

A.9. In a deep linear network (composition of \(k\) linear layers), the rank of the overall map is the product of the individual layer ranks.

Final Answer: FALSE

Full Mathematical Justification: The rank of a composition \(T_k \circ \cdots \circ T_2 \circ T_1\) is bounded by the minimum of the individual ranks, not the product. Formally, \(\text{rank}(T_2 \circ T_1) \leq \min(\text{rank}(T_1), \text{rank}(T_2))\). Proof: the image of \(T_2 \circ T_1\) is \(\{ T_2(T_1(\mathbf{x})) : \mathbf{x} \in \text{domain} \} = \{ T_2(\mathbf{y}) : \mathbf{y} \in \text{im}(T_1) \} \subseteq \text{im}(T_2)\). So the image of the composition is a subset of the image of \(T_2\), implying rank composition ≤ rank \(T_2\). Also, the composition maps the domain into a \(\text{rank}(T_1)\)-dimensional subspace (the image of \(T_1\)), so the composition has rank at most \(\text{rank}(T_1)\). Thus rank is at most min. The product of ranks is almost always larger than the min.

Counterexample if False: Let \(T_1: \mathbb{R}^3 \to \mathbb{R}^3\) with rank 2 (e.g., projection onto the first two coordinates), and \(T_2: \mathbb{R}^3 \to \mathbb{R}^3\) with rank 2 (e.g., projection onto the first two coordinates). The product of ranks is 2 × 2 = 4. But the composition \(T_2 \circ T_1\) is just projection onto the first two coordinates again, with rank 2 (not 4). In fact, any composition of two rank-2 maps has rank at most 2. More starkly: \(T_1\) with rank 10 composed with itself gives the same map with rank 10, not 10 × 10 = 100.

Comprehension: Composing two low-rank maps creates an even lower-rank map (or stays the same). Information loss is monotonic under composition. Each layer bottlenecks the next, limiting the overall dimensionality.

ML Applications: In deep neural networks, a single low-rank hidden layer (rank 50 in a 784-dimensional space) limits all subsequent layers to 50 effective dimensions. If a second layer also has rank 50, the composition still has at most rank 50, not 50 × 50 = 2500. This is the core issue in vanishing gradients: low-rank layers compose to create even lower-rank (or constant-rank) maps.

Failure Mode Analysis: Incorrectly assuming that composing two rank-\(r\) layers gives rank \(r^2\) would lead to massive overestimation of network capacity. This is a critical error in analyzing deep networks.

Traps: The statement uses the word “product,” which might invoke multiplication intuition. But rank composition is not multiplicative; it’s subadditive or min-bounded. Also, for full-rank layers (rank = min(input_dim, output_dim)), composition preserves full rank, but this is a special case, not evidence for product rule.

A.10. If a weight matrix \(W \in \mathbb{R}^{m \times n}\) has rank \(r < \min(m, n)\), then there exist two distinct input vectors \(\mathbf{x}_1 \neq \mathbf{x}_2\) such that \(W\mathbf{x}_1 = W\mathbf{x}_2\).

Final Answer: TRUE

Full Mathematical Justification: If rank \(r < n\) (the column dimension), then by rank-nullity, nullity = \(n - r > 0\), so the kernel is nontrivial. This means there exists \(\mathbf{k} \neq \mathbf{0}\) with \(W\mathbf{k} = \mathbf{0}\). Now, for any \(\mathbf{x}_1\), let \(\mathbf{x}_2 = \mathbf{x}_1 + \mathbf{k}\). Then \(W\mathbf{x}_2 = W(\mathbf{x}_1 + \mathbf{k}) = W\mathbf{x}_1 + W\mathbf{k} = W\mathbf{x}_1 + \mathbf{0} = W\mathbf{x}_1\), and \(\mathbf{x}_2 \neq \mathbf{x}_1\) (since \(\mathbf{k} \neq \mathbf{0}\)). Thus, distinct inputs produce the same output.

Counterexample if False: Not applicable; the statement is true.

Comprehension: Rank deficiency (rank < column count) directly implies nontrivial kernel, which directly implies the map is not injective (not one-to-one). Multiple distinct inputs map to the same output.

ML Applications: In neural networks, if a weight matrix is rank-deficient, the layer does not have injective input-output mapping. The layer “compresses” information—multiple input patterns produce the same hidden representation. This is sometimes desired (dimensionality reduction) but sometimes problematic (loss of information for downstream tasks).

Failure Mode Analysis: The condition is \(r < \min(m, n)\), not just \(r < m\). If \(m < n\) (more inputs than outputs), then rank \(r \leq m < n\), so the condition is satisfied. If \(m > n\), then rank \(r < n\) is needed, which is exactly the rank-deficiency condition that forces nullity > 0.

Traps: Confusing the rank condition with other properties. For example, \(r < m\) (rank less than row count) implies the map is not surjective (rows are not all independent, so image is not all of \(\mathbb{R}^m\)). But the statement asks for non-injectivity, which requires \(r < n\) (rank less than column count).

A.11. The singular value decomposition (SVD) of a matrix \(A = U \Sigma V^\top\) reveals that the rank of \(A\) equals the number of nonzero diagonal entries in \(\Sigma\).

Final Answer: TRUE

Full Mathematical Justification: The SVD decomposes \(A\) as a product of orthogonal matrices \(U\), \(V\) and a diagonal matrix \(\Sigma\) with singular values \(\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_{\min(m,n)} \geq 0\) on the diagonal. The rank of \(A\) is the dimension of the image, which can be computed from the SVD as follows: the columns of \(U\) corresponding to nonzero singular values are an orthonormal basis for the image (column space). Thus, rank = number of nonzero \(\sigma_i\). The null space is spanned by columns of \(V\) corresponding to zero singular values; its dimension is the number of zero \(\sigma_i\). This matches rank-nullity: rank + nullity = (number of nonzero) + (number of zero) = total columns = \(n\).

Counterexample if False: Not applicable; the statement is true.

Comprehension: The SVD is a canonical decomposition that makes rank transparent. The diagonal matrix \(\Sigma\) directly displays the “strength” of singular values; zero values immediately indicate rank deficiency.

ML Applications: In principal component analysis (PCA), the singular values of the data matrix reveal the explained variance along each principal direction. Keeping only the top-\(k\) singular values (and their corresponding vectors) discards low-variance directions, achieving dimensionality reduction. In spectral methods, singular values characterize the condition number and geometric properties of the matrix.

Failure Mode Analysis: Confusing “number of nonzero singular values” with “number of singular values less than a threshold.” Depending on numerical precision, near-zero singular values (1e-15, say) might appear as nonzero in floating-point arithmetic. In practice, numerical rank uses a threshold (e.g., singular values less than epsilon × max singular value are considered zero).

Traps: The SVD is often computed numerically, and numerical zero-detection depends on thresholds. A mathematically rank-5 matrix might appear rank-6 or rank-4 depending on the threshold chosen. For exact theoretical problems, rank = number of nonzero \(\sigma_i\) is correct; for numerical computation, use a threshold-based rank estimation.

A.12. A linear map \(T: V \to W\) between finite-dimensional spaces is surjective if and only if \(\text{rank}(T) = \dim(W)\).

Final Answer: TRUE

Full Mathematical Justification: Surjectivity means every element of \(W\) is the image of some element in \(V\), i.e., image = \(W\). The rank is the dimension of the image, so surjectivity is equivalent to \(\dim(\text{im}(T)) = \dim(W)\), which is exactly rank(T) = \(\dim(W)\).

Counterexample if False: Not applicable; the statement is true.

Comprehension: This is a direct consequence of definitions. Surjection (onto-ness) is the image-space being large enough; rank measures image dimension; so rank = codomain dimension is exactly surjectivity.

ML Applications: In neural networks, a layer is surjective if its output can take any value in its output space. If a layer has 128 outputs, it is surjective if rank(W) = 128 (the weight matrix has full column rank when viewed as a map from input space to output space). Non-surjective layers have “dead” output dimensions that are never activated.

Failure Mode Analysis: Confusing surjectivity with injectivity. Surjectivity bounds rank from below (rank ≥ \(\dim(W)\)), while injectivity is about kernel. For a square map (\(\dim(V) = \dim(W)\)), rank = full dimension (\(N\)) implies both injectivity and surjectivity, hence invertibility.

Traps: Be clear on the direction: surjectivity ⟺ rank ≥ \(\dim(W)\) for onto mapping, but the only way rank = the dimension of the image can also be the dimension of the codomain is if rank = \(\dim(W)\). If rank < \(\dim(W)\), the map is not surjective. If rank = \(\dim(W)\), the map is surjective.

A.13. For a regression problem with design matrix \(X \in \mathbb{R}^{n \times p}\) (n observations, p features), if \(\text{rank}(X) < p\), then the least-squares solution \(\hat{\mathbf{w}} = (X^\top X)^{-1} X^\top \mathbf{y}\) exists and is unique.

Final Answer: FALSE

Full Mathematical Justification: If rank(X) < p, then X does not have full column rank. The Gram matrix \(X^\top X\) is a \(p \times p\) matrix, and its rank equals rank(X) (by properties of matrix products). If rank(X) < p, then rank(\(X^\top X\)) < p, so \(X^\top X\) is singular (not invertible). Thus, the inverse \((X^\top X)^{-1}\) does not exist, and the formula cannot be evaluated. The least-squares problem \(\min_\mathbf{w} \| X\mathbf{w} - \mathbf{y} \|^2\) still has a solution (or infinitely many solutions) via the pseudo-inverse \(X^\dagger\), but the normal equations formula involving the inverse does not apply. If solutions exist, they are not unique (the solution set is an affine subspace).

Counterexample if False: Let \(X = \begin{pmatrix} 1 & 2 \\ 1 & 2 \\ 1 & 2 \end{pmatrix}\) (rank 1 < 2 columns). The two columns are proportional, so X is rank-deficient. Then \(X^\top X = \begin{pmatrix} 3 & 6 \\ 6 & 12 \end{pmatrix}\), which has determinant 36 - 36 = 0, so it’s singular. The inverse does not exist. The least-squares solutions form an affine line (not a unique point) in \(\mathbb{R}^2\).

Comprehension: The normal equations require \(X^\top X\) to be invertible, which requires full column rank for X. Rank deficiency breaks this requirement. However, the least-squares problem still has solutions (if the problem is consistent) or minimizers; they just are not unique.

ML Applications: In regression with multicollinear features (linearly dependent features), the design matrix is rank-deficient. The standard normal equations formula fails. Practitioners must either remove redundant features (reducing p to be less than the rank) or use regularization (Ridge regression), which adds a term to \(X^\top X\) that makes it invertible.

Failure Mode Analysis: This is a critical error in practical ML. Assuming the normal equations formula works without checking rank deficiency leads to singular matrix errors and crashes. Proper regression implementations check for rank deficiency first.

Traps: The problem statement says the solution “exists and is unique.” Even if a solution exists, rank deficiency means infinitely many solutions (not unique). Additionally, the formula requires existence of the inverse, which fails under rank deficiency. The statement is wrong on both fronts.

A.14. Two matrices \(A\) and \(B\) in \(\mathbb{R}^{n \times n}\) represent the same linear map (in different bases) if and only if they are similar, i.e., \(B = P^{-1}AP\) for some invertible matrix \(P\).

Final Answer: TRUE

Full Mathematical Justification: A linear map \(T: V \to V\) has a matrix representation in a given basis. If you change the basis via a change-of-basis matrix \(P\), the new matrix representation is \(A' = P^{-1}AP\). Conversely, if \(B = P^{-1}AP\), then \(A\) and \(B\) both represent the same linear map \(T\), just in different bases (one ordered basis and another related by the change of basis \(P\)). Thus, same map ⟺ similar matrices (Definition 14 from Chapter 3). Similarity is an equivalence relation on \(n \times n\) matrices that partitions them by the linear maps they represent.

Counterexample if False: Not applicable; the statement is true.

Comprehension: This is foundational: matrices are coordinate representations of abstract linear maps. The abstract map is basis-independent; the matrix representation is basis-dependent. The relationship between two representations of the same map is exactly similarity.

ML Applications: When you train a neural network, you learn weight matrices. If you later change the basis (e.g., via a change-of-basis layer or coordinate transformation), the learned weights become similar matrices, but they represent the same learned transformation. In transfer learning, pre-trained networks are “basis-compatible” in the sense that their representations can be adapted to new tasks via basis changes (fine-tuning, adaptation).

Failure Mode Analysis: Confusing similarity with other matrix relationships (e.g., row equivalence, column equivalence). Similarity is specifically for representing the same linear map in different bases; other equivalences have different meanings.

Traps: Be aware that similarity requires the change-of-basis matrix \(P\) to be invertible. Non-invertible matrices do not correspond to basis changes. Additionally, similarity is defined for square matrices; non-square matrices cannot be similar (though they can be equivalent via row and column equivalences).

A.15. An autoencoder with a bottleneck layer (encoder output dimension smaller than input dimension) fundamentally cannot reconstruct inputs outside the image of the encoder’s linear component, even with optimal nonlinear decoders.

Final Answer: FALSE

Full Mathematical Justification: The statement is overly restrictive. While it’s true that a linear encoder \(E: \mathbb{R}^{784} \to \mathbb{R}^{50}\) (linear, with rank ≤ 50) can only map inputs to a 50-dimensional subspace, a nonlinear decoder \(D: \mathbb{R}^{50} \to \mathbb{R}^{784}\) (nonlinear) can expand the 50-dimensional bottleneck back to 784 dimensions and potentially approximate the identity map on training data. The nonlinearity breaks the linear constraint. For example, the nonlinear function \(D(\mathbf{z}) = \mathbf{z} + f(\mathbf{z})\) (where \(f\) is a learned nonlinear mapping) can produce outputs far outside the image of the linear encoder. The autoencoder’s reconstruction quality depends on whether the nonlinear decoder can learn to invert the encoder, not on the encoder being linear.

Counterexample if False: Consider a nonlinear autoencoder where the encoder is linear (rank-50 bottleneck) but the decoder is a deep nonlinear network. This nonlinear decoder can, in principle, learn to approximate the identity map on training data, reconstructing inputs to arbitrary precision. Empirically, autoencoders with nonlinear decoders and linear bottlenecks (or nonlinear bottlenecks) achieve high reconstruction quality on benchmarks like MNIST, CIFAR, ImageNet.

Comprehension: The statement conflates the linear encoder’s limited image with the autoencoder’s overall capability. The nonlinear decoder can “compensate” for the bottleneck’s linearity by learning a complex decoder.

ML Applications: Variational Autoencoders (VAEs) use bottleneck layers (either linear or nonlinear) and nonlinear decoders. Despite the bottleneck constraint, VAEs learn to generate high-quality reconstructions through the nonlinearity of the decoder. Denoising autoencoders similarly pair linear or nonlinear bottlenecks with nonlinear decoders. The nonlinearity is key to the decoder’s expressivity.

Failure Mode Analysis: The statement makes a strong claim (“fundamentally cannot”), which is incorrect. The decoder’s nonlinearity can overcome bottleneck limitations for simple data distributions (high-variance directions), though not for truly “hidden” directions (those orthogonal to the bottleneck’s image, which are completely lost and cannot be recovered).

Traps: The statement is half-right in the sense that information orthogonal to the bottleneck’s image is truly lost and cannot be recovered (even nonlinearity can’t create information). But the statement says “cannot reconstruct inputs outside the image,” which is false if “reconstruct” means “approximately reconstruct on training data” (nonlinearity enables this). The misunderstanding is between hard information loss and reconstruction capability.

A.16. The operator norm of a linear map \(T: \mathbb{R}^n \to \mathbb{R}^m\) (induced by Euclidean norms) equals the largest singular value of its matrix representation.

Final Answer: TRUE

Full Mathematical Justification: The operator norm is defined as \(\|T\| = \max_{\mathbf{x} \neq \mathbf{0}} \frac{\|T(\mathbf{x})\|}{\|\mathbf{x}\|} = \max_{\|\mathbf{x}\|=1} \|T(\mathbf{x})\|\) (the maximum stretch under the map). For a matrix \(A\) with SVD \(A = U \Sigma V^\top\), we have \(A\mathbf{x} = U\Sigma V^\top \mathbf{x}\). For unit vector \(\mathbf{x}\), \(\|V^\top \mathbf{x}\| \leq 1\) (since \(V\) is orthogonal). Thus, \(\|A\mathbf{x}\| = \| U \Sigma V^\top \mathbf{x} \| = \| \Sigma V^\top \mathbf{x} \|\) (since \(U\) is orthogonal, doesn’t change norm). The maximum occurs when \(V^\top \mathbf{x}\) aligns with the first singular vector direction (the standard basis direction), giving \(\| \Sigma V^\top \mathbf{x} \| \leq \sigma_1 \|V^\top \mathbf{x}\| \leq \sigma_1\). The maximum \(\sigma_1\) is achieved. Thus, \(\|T\| = \sigma_1\), the largest singular value.

Counterexample if False: Not applicable; the statement is true.

Comprehension: The operator norm is a functional norm (norm of a function viewed as an operator), and the SVD makes it transparent: the largest singular value is the maximum factor by which the map stretches vectors.

ML Applications: The operator norm controls condition number: \(\kappa(A) = \sigma_1 / \sigma_n\) (largest / smallest singular value). In optimization, operator norm relates to Lipschitz continuity (gradient bounded by operator norm of Hessian). In neural networks, spectral normalization constrains weight matrix operator norms to stabilize training (often to norm 1, ensuring gradients don’t explode or vanish). In GANs, discriminator stability is improved via spectral norm constraint.

Failure Mode Analysis: Confusing operator norm with Frobenius norm (\(\|A\|_F = \sqrt{\sum \sigma_i^2}\)) or other norms (Frobenius is larger than operator norm in general). The operator norm is specifically the largest singular value.

Traps: The operator norm is also called the “spectral norm” (referring to the spectrum, i.e., singular values / eigenvalues of \(A^\top A\) or \(AA^\top\)). Be aware of terminology variations.

A.17. For a fully connected neural network layer with ReLU activation, if the weight matrix \(W\) has rank less than the number of output units, then no amount of training can make the layer express a function that requires the full output dimensionality.

Final Answer: TRUE

Full Mathematical Justification: The layer computes \(\mathbf{h} = \sigma(W\mathbf{x} + \mathbf{b})\), where \(\sigma\) is ReLU (element-wise max(·, 0)). The linear part \(W\mathbf{x}\) maps inputs to an at-most-\(\text{rank}(W)\)-dimensional subspace (the image of \(W\)). ReLU applies element-wise nonlinearity, which can expand dimensions into the “positive orthant” but cannot create new dimensions beyond the subspace spanned by \(W\)’s image. More precisely, the image of the ReLU layer is contained in a \(\text{rank}(W)\)-dimensional subspace of \(\mathbb{R}^{\text{num outputs}}\). Thus, if rank(W) < num outputs, the layer’s effective output dimension is rank(W), not num outputs, regardless of the bias or the nonlinearity of ReLU.

Counterexample if False: Let \(W \in \mathbb{R}^{100 \times 50}\) (100 outputs, 50 inputs) with rank 30. The layer outputs 100-dimensional vectors, but they all lie in a 30-dimensional subspace. No training (adjusting the weights, bias, or learning rates) can change this hard constraint. ReLU’s nonlinearity cannot expand the subspace.

Comprehension: Rank is a hard constraint on dimensionality. Nonlinearity cannot overcome dimensionality constraints, only modify the function’s behavior within the constrained subspace.

ML Applications: In neural networks, a layer with rank deficiency is a bottleneck. If rank(W) = 30 and num outputs = 100, the layer discards 70 dimensions worth of capacity. Subsequent layers cannot access those 70 dimensions, no matter how strong they are. This is a fundamental limitation guiding neural network design: avoid bottleneck layers unless intentional.

Failure Mode Analysis: Assuming that training long enough or using strong regularization can recover lost capacity, it cannot. Rank deficiency is a structural constraint that persists regardless of training.

Traps: Be careful to distinguish between “rank(W)” and “the number of active ReLU units.” ReLU activation can deactivate (output 0) for some dimensions, effectively reducing the output dimension, but this is temporary and depends on the input. The hard constraint is rank(W), not the number of active ReLU units.

A.18. A change of basis transformation \(A' = P^{-1}AP\) preserves the rank of the original matrix \(A\).

Final Answer: TRUE

Full Mathematical Justification: Rank is an intrinsic property of a linear map, not dependent on the choice of basis. Formally, the rank of a matrix is the dimension of its image (column space). Under similarity transformation \(A' = P^{-1}AP\), both \(A\) and \(A'\) represent the same linear map in different bases. Since rank is a property of the map (not the representation), rank(\(A\)) = rank(\(A'\)). Algebraically, rank(\(A'\)) = rank(\(P^{-1}AP\)) ≤ min(rank(\(P^{-1}\)), rank(\(A\)), rank(\(P\))), and since \(P\) is invertible, rank(\(P\)) = rank(\(P^{-1}\)) = n (full rank). Thus, rank(\(A'\)) ≤ rank(\(A\)). By symmetry (applying the inverse transformation \(A = PP^{-1}AP^{-1}(P)\)), rank(\(A\)) ≤ rank(\(A'\)). Thus, rank(\(A\)) = rank(\(A'\)).

Counterexample if False: Not applicable; the statement is true.

Comprehension: Similarity is an equivalence relation that preserves all geometric properties of the linear map: rank, trace, determinant, eigenvalues (for square matrices), etc. Similarity is precisely the relationship between two representations of the same map.

ML Applications: When neural network layers are re-parameterized (e.g., via basis changes or layer-wise changes), the rank of the transformation persists. This is important for analyzing pruning, quantization, and other transformations.

Failure Mode Analysis: Confusing similarity (\(B = P^{-1}AP\)) with other equivalences (e.g., row reduction, which changes rank). Row equivalence and column equivalence may change rank via row/column operations.

Traps: Be clear on what is being transformed. Similarity transforms both sides of the matrix; row or column operations do not. Similarity is a rigid relationship (invertible transformation); row/column operations are not.

A.19. In backpropagation through a deep linear network, the gradient with respect to an early layer’s weight matrix depends on the product of the Jacobian matrices of all subsequent layers, so a single low-rank layer can cause vanishing gradients for all earlier layers.

Final Answer: TRUE

Full Mathematical Justification: In a deep linear network \(\mathbf{y} = W_k \cdots W_2 W_1 \mathbf{x}\), the loss \(\ell(\mathbf{y})\) is a scalar. By the chain rule, \(\frac{\partial \ell}{\partial W_i}\) involves the product of all subsequent Jacobians: \(\frac{\partial \ell}{\partial W_i} \propto J_k \cdots J_{i+1}\), where \(J_j\) is the Jacobian of layer \(j\). If any \(J_j\) (for \(j > i\)) is low-rank (rank-deficient), the product \(J_k \cdots J_{i+1}\) has rank bounded by the minimum individual rank. A single low-rank layer makes the entire product low-rank, which means some directions in the gradient space are zero. The gradient vectors (w.r.t. \(W_i\)) are constrained to a low-rank subspace, leading to small gradient magnitudes or zero gradients in many directions. This is the vanishing gradient problem: a single bottleneck propagates backward and suppresses gradients for all earlier layers.

Counterexample if False: Let \(W_1 \in \mathbb{R}^{100 \times 50}\), \(W_2 \in \mathbb{R}^{50 \times 100}\) (rank 50), \(W_3 \in \mathbb{R}^{100 \times 50}\) (rank 50). The composition has rank at most 50. The Jacobian of layer 2’s output has rank ≤ 50, making the product \(J_3 J_2\) rank-bounded. The gradient for \(W_1\) is proportional to \(J_3 J_2\), which is low-rank. Many directions in the parameter space have zero gradient. This demonstrates vanishing gradients in a deep linear network.

Comprehension: Gradients propagate through the network as products of successive Jacobians. Low-rank Jacobians cause gradients to collapse to low-rank subspaces, leading to training difficulty.

ML Applications: This is the classic vanishing gradient problem, one of the most important challenges in deep learning. Solutions include skip connections (ResNets), which bypass bottleneck layers and allow gradients to flow directly through higher-rank paths. Batch normalization and layer normalization also address this by controlling the Jacobian’s spectrum and preventing extreme rank deficiency.

Failure Mode Analysis: Ignoring the rank constraint in gradient flow leads to the misconception that any deep network can be trained. In reality, deep networks with bottleneck layers suffer from vanishing gradients, requiring architectural solutions (skip connections, normalization) to address.

Traps: Confusing forward-pass bottlenecks (limiting the capacity of hidden representations) with backward-pass vanishing gradients (limiting the strength of gradient signals). Both are rank-related but are aspects of different directions of information flow.

A.20. The kernel of a linear map \(T: V \to W\) is always a subspace of the domain \(V\), and the image of \(T\) is always a subspace of the codomain \(W\).

Final Answer: TRUE

Full Mathematical Justification: Kernel = \(\ker(T) = \{ \mathbf{v} \in V : T(\mathbf{v}) = \mathbf{0} \}\). This is a subspace of \(V\) by the subspace criterion: (1) \(\mathbf{0} \in \ker(T)\) (since \(T(\mathbf{0}) = \mathbf{0}\) by linearity); (2) closure under addition: if \(T(\mathbf{v}_1) = T(\mathbf{v}_2) = \mathbf{0}\), then \(T(\mathbf{v}_1 + \mathbf{v}_2) = T(\mathbf{v}_1) + T(\mathbf{v}_2) = \mathbf{0} + \mathbf{0} = \mathbf{0}\), so \(\mathbf{v}_1 + \mathbf{v}_2 \in \ker(T)\); (3) closure under scalar multiplication: if \(T(\mathbf{v}) = \mathbf{0}\) and \(c\) is a scalar, then \(T(c\mathbf{v}) = cT(\mathbf{v}) = c \mathbf{0} = \mathbf{0}\), so \(c\mathbf{v} \in \ker(T)\). Similarly, image = \(\text{im}(T) = \{ T(\mathbf{v}) : \mathbf{v} \in V \} \subseteq W\) is a subspace by: (1) \(T(\mathbf{0}) = \mathbf{0} \in \text{im}(T)\); (2) if \(\mathbf{w}_1, \mathbf{w}_2 \in \text{im}(T)\), then \(\mathbf{w}_1 = T(\mathbf{v}_1)\), \(\mathbf{w}_2 = T(\mathbf{v}_2)\), so \(\mathbf{w}_1 + \mathbf{w}_2 = T(\mathbf{v}_1) + T(\mathbf{v}_2) = T(\mathbf{v}_1 + \mathbf{v}_2) \in \text{im}(T)\); (3) if \(\mathbf{w} \in \text{im}(T)\) and \(c\) is a scalar, then \(\mathbf{w} = T(\mathbf{v})\), so \(c\mathbf{w} = cT(\mathbf{v}) = T(c\mathbf{v}) \in \text{im}(T)\). Thus, both are subspaces.

Counterexample if False: Not applicable; the statement is true.

Comprehension: This is a foundational result in linear algebra, often stated early and developed throughout Chapter 3. It is definitional: the kernel and image are defined to be sets with specific properties, and it is proven that these properties guarantee subspace structure.

ML Applications: The kernel and image partition the domain and codomain in a meaningful way. In neural networks, the kernel represents “dead” or “invisible” input directions; the image represents “activated” output dimensions. Understanding both is crucial for analyzing information flow.

Failure Mode Analysis: None; this is a theorem, not a statement with failure modes.

Traps: Be careful to state which space (domain or codomain) the kernel and image belong to. Kernel ⊆ domain, image ⊆ codomain. A common mistake is confusing the kernel (in the domain) with the null space of the transpose (in the dual space).

Solutions to B. Proof Problems

B.1. Prove that the kernel of a linear map \(T: V \to W\) is a subspace of \(V\). Be explicit about closure under addition and scalar multiplication.

Full Formal Proof: We prove that \(\ker(T) = \{ \mathbf{v} \in V : T(\mathbf{v}) = \mathbf{0}_W \}\) is a subspace of \(V\) by verifying the three subspace criteria: (1) contains zero vector, (2) closed under addition, (3) closed under scalar multiplication.

(1) Zero vector: By linearity, \(T(\mathbf{0}_V) = \mathbf{0}_W\), so \(\mathbf{0}_V \in \ker(T)\).

(2) Closure under addition: Let \(\mathbf{v}_1, \mathbf{v}_2 \in \ker(T)\). Then \(T(\mathbf{v}_1) = \mathbf{0}_W\) and \(T(\mathbf{v}_2) = \mathbf{0}_W\). By linearity of \(T\), \(T(\mathbf{v}_1 + \mathbf{v}_2) = T(\mathbf{v}_1) + T(\mathbf{v}_2) = \mathbf{0}_W + \mathbf{0}_W = \mathbf{0}_W\). Thus, \(\mathbf{v}_1 + \mathbf{v}_2 \in \ker(T)\).

(3) Closure under scalar multiplication: Let \(\mathbf{v} \in \ker(T)\) and let \(c\) be a scalar (in the ground field). Then \(T(\mathbf{v}) = \mathbf{0}_W\). By linearity, \(T(c\mathbf{v}) = cT(\mathbf{v}) = c \mathbf{0}_W = \mathbf{0}_W\). Thus, \(c\mathbf{v} \in \ker(T)\).

Since all three criteria are satisfied, \(\ker(T)\) is a subspace of \(V\). \(\square\)

Proof Strategy & Techniques: This is a direct proof using the definition and properties of linear maps and subspace criteria. The strategy is: state the three conditions required for a subspace, then verify each one systematically. Key technique: leveraging linearity (additivity and homogeneity) to ensure the kernel inherits subspace structure. No contrapositive or induction is needed; straightforward verification suffices.

Computational Validation: For a 5×3 matrix \(A = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 2 & 1 & 0 \\ 0 & 0 & 0 \\ 1 & 2 & 3 \end{pmatrix}\), compute the kernel by solving \(A\mathbf{x} = \mathbf{0}\). Row reduction gives a basis for the kernel, say \(\{ \mathbf{k}_1, \mathbf{k}_2 \}\). Verify: (a) \(A\mathbf{k}_1 = \mathbf{0}\) and \(A\mathbf{k}_2 = \mathbf{0}\); (b) for any \(\mathbf{k} = c_1 \mathbf{k}_1 + c_2 \mathbf{k}_2\), we have \(A\mathbf{k} = c_1 A\mathbf{k}_1 + c_2 A\mathbf{k}_2 = \mathbf{0}\); (c) any combination is in the kernel.

ML Interpretation: In neural networks, the kernel represents input directions that the network layer completely ignores—they are annihilated and produce zero output. Understanding the kernel as a subspace means identifying all such “dead” directions simultaneously. These directions form a geometric object (subspace) rather than scattered points, which is geometrically cleaner and algorithmically more efficient.

Generalization & Edge Cases: This theorem holds for any linear map between arbitrary vector spaces over any field. Special cases: (1) kernel is trivial (\(\{ \mathbf{0} \}\)) iff the map is injective; (2) kernel is the entire domain iff the map is the zero map; (3) for infinite-dimensional spaces, the kernel is still a subspace, though possibly infinite-dimensional.

Failure Mode Analysis: A common error is claiming that the kernel is “all vectors mapping to zero,” then only verifying this for specific vectors rather than proving closure properties. Another mistake is applying the subspace criterion without rigorously checking linearity; for example, assuming addition is closed without proof. A third error: confusing the kernel (in the domain) with the nullspace (which is the same thing, but students sometimes conflate it with the cokernel or other dual objects).

Historical Context: The kernel is a fundamental concept in modern algebra, emerging from the study of homomorphisms in the early 20th century. The formal notion of subspace under addition and scalar multiplication crystallized in the work of Cayley, Sylvester, and later Peano and others. The kernel-image decomposition (via rank-nullity) is central to algebraic topology and homological algebra.

Traps: (1) Forgetting that the kernel is in the domain, not the codomain. (2) Assuming closure of a property without proving it rigorously. (3) Confusing “kernel contains zero” with “kernel is nontrivial.” (4) In numerical computation, due to floating-point errors, vectors solving \(A\mathbf{x} = \mathbf{0}\) might not be exactly zero; threshold-based detection is needed.

B.2. Prove that if \(T: V \to W\) is a linear map with \(\text{rank}(T) = \dim(V)\), then \(T\) is injective.

Full Formal Proof: We prove by using rank-nullity. By rank-nullity theorem, \(\dim(V) = \text{rank}(T) + \text{nullity}(T)\). Given \(\text{rank}(T) = \dim(V)\), we have \(\text{nullity}(T) = \dim(V) - \text{rank}(T) = 0\). Thus, \(\ker(T) = \{ \mathbf{0} \}\) (the kernel is trivial). A map is injective iff its kernel is trivial (if \(T(\mathbf{v}_1) = T(\mathbf{v}_2)\), then \(T(\mathbf{v}_1 - \mathbf{v}_2) = \mathbf{0}\), so \(\mathbf{v}_1 - \mathbf{v}_2 \in \ker(T) = \{ \mathbf{0} \}\), thus \(\mathbf{v}_1 = \mathbf{v}_2\)). Therefore, \(T\) is injective. \(\square\)

Proof Strategy & Techniques: The proof uses the rank-nullity theorem as the main tool. Strategy: (1) apply rank-nullity to decompose domain dimension, (2) simplify using the given rank condition, (3) conclude about kernel, (4) use the characterization of injectivity via trivial kernel. The technique is algebraic and definitional; it does not require careful case analysis or deep structural theory.

Computational Validation: For a 4×3 matrix \(A\) with columns \(\mathbf{a}_1, \mathbf{a}_2, \mathbf{a}_3\), if rank(\(A) = 3\) (full column rank), then the columns are linearly independent. The nullspace is trivial (only scalar multiple of columns giving zero is all zeros). Verify: (a) compute \(\text{rank}(A)\) via SVD or QR decomposition; (b) if rank = 3, verify that \(A\mathbf{x} = \mathbf{0}\) has only the zero solution.

ML Interpretation: A neural network layer \(T: \mathbb{R}^n \to \mathbb{R}^m\) is injective if rank = n ≤ m (input dimension). This means distinct input vectors always produce distinct outputs (no information loss via collision). Injectivity is desirable in compression (bottleneck layers are intentionally non-injective) but undesirable in feature extraction (losing information is bad). Understanding when a layer is injective guides architecture design.

Generalization & Edge Cases: The theorem requires only finite-dimensional vector spaces and a linear map. It generalizes: for infinite-dimensional spaces, the statement “rank = domain dimension” is replaced by “codimension of kernel = 0” or “image is a proper subspace.” Edge case: dimension 0 (the zero space); a map from \(\{ \mathbf{0} \}\) is vacuously injective.

Failure Mode Analysis: A common mistake is confusing “rank = dim(V)” with “rank = dim(W)”. Only the former guarantees injectivity. Another error: forgetting that the theorem uses rank-nullity, which requires finite-dimensionality; it does not apply to infinite-dimensional spaces without care. A third error: assuming injectivity without checking the rank condition; injectivity is a stronger requirement (it’s equivalent to rank = domain dimension when domain and codomain have compatible dimensions).

Historical Context: Rank-nullity theorem (also called the fundamental theorem of linear algebra) is attributed to Sylvester in the mid-1800s, though the concept evolved through the work of many mathematicians. The connection between kernel dimension and injectivity was formalized in the early 20th century as part of the abstract algebra revolution.

Traps: (1) Stating “rank = dim(V)” when you mean something else numerically; be precise. (2) Forgetting the theorem applies with strict equality; a rank less than dim(V) does not imply injectivity. (3) Numerical computation: small singular values (or nearly zero eigenvalues) might appear as nonzero, making rank detection ambiguous.

B.3. For a fully connected neural network layer represented by weight matrix \(W \in \mathbb{R}^{m \times n}\), prove that if \(\text{rank}(W) = r < n\), then the image of the linear map \(T(\mathbf{x}) = W\mathbf{x}\) is at most \(r\)-dimensional, regardless of the input distribution.

Full Formal Proof: The image of \(T\) is \(\text{im}(T) = \{ W\mathbf{x} : \mathbf{x} \in \mathbb{R}^n \}\). By definition, the rank of \(W\) is \(\text{rank}(W) = \dim(\text{im}(T))\) (the dimension of the column space). Thus, if \(\text{rank}(W) = r\), then \(\dim(\text{im}(T)) = r\). This means the image is a subspace of \(\mathbb{R}^m\) with dimension exactly \(r\). Therefore, the output of the layer lies in an \(r\)-dimensional subspace of \(\mathbb{R}^m\), regardless of the distribution of input vectors. The statement “regardless of the input distribution” emphasizes that this is a geometric constraint, not a probabilistic one: every possible output, for any input, lies in this \(r\)-dimensional subspace. \(\square\)

Proof Strategy & Techniques: The proof is a restatement of definitions. The key insight is recognizing that rank is defined as the dimension of the column space (image). The strategy is: (1) identify the image in terms of the matrix action, (2) note that rank, by definition, determines the dimension of the image, (3) conclude the bound. No sophisticated techniques are needed; understanding definitions is the core.

Computational Validation: For a 100×50 matrix \(W\) with rank 30, the image is 30-dimensional. Any output \(W\mathbf{x} \in \mathbb{R}^{100}\) lies in a 30-dimensional subspace. To verify: (a) compute the SVD of \(W\); the number of nonzero singular values is 30; (b) the first 30 left singular vectors form an orthonormal basis for the image; (c) for any input \(\mathbf{x}\), the output \(W\mathbf{x}\) is a linear combination of these basis vectors (verify by expansion).

ML Interpretation: In neural networks, this is the fundamental capacity constraint: a layer with rank \(r\) outputs can be expressed in at most an \(r\)-dimensional subspace, no matter what input distribution you have, no matter how clever the downstream layers are. This is a hard geometric bound, not a probabilistic or statistical one. Understanding this guides decisions: if you want a layer to express 100-dimensional features, the weight matrix must have rank ≥ 100. This bound is independent of where your data comes from (input distribution).

Generalization & Edge Cases: The theorem holds for matrices over any field (above we used \(\mathbb{R}\), but it works over \(\mathbb{C}\), \(\mathbb{F}_p\), etc.). Edge case: rank 0 means the matrix is all zeros, and the image is \(\{ \mathbf{0} \}\) (0-dimensional). Edge case: rank = n = m (square full-rank matrix) means the image is all of \(\mathbb{R}^m\). Edge case: rank < min(m, n) means the image is a proper subspace.

Failure Mode Analysis: A common misconception is believing that data distribution or training can overcome the rank bound. It cannot. No amount of input diversity or training dynamics changes the rank-imposed dimensional constraint. Another error: confusing “the output is 100-dimensional” with “can express 100-dimensional features”; the former is false if rank < 100, the latter is the correct statement. A third error: assuming that the number of nonzero activations (neuron saturations in ReLU) relates to the rank bound; it does not (activations are binary or continuous, rank is a linear-algebraic property).

Historical Context: The rank-dimensionality relationship is foundational to linear algebra, formalized in the 19th century through the work of Cayley, Sylvester, Rouché, and Frobenius. The modern interpretation in terms of image and kernel dimension came with the abstraction of vector spaces in the early 20th century. The application to neural networks is recent (late 20th century onward), recognizing rank as a bottleneck for expressivity.

Traps: (1) Confusing “dimension of the image” with “dimension of the parameter space.” The rank bounds the image dimension, not the number of parameters (which is \(m \times n\)). (2) Numerical trap: computing rank via SVD can be ambiguous near floating-point precision limits; use a tolerance-based threshold. (3) Forgetting that the bound applies to all possible inputs; there are no “clever” input distributions that escape the rank constraint.

[Continuing with B.4 through B.20, each with 8 detailed components. Due to length, I’ll provide comprehensive but more condensed versions for clarity while maintaining rigor.]

B.4. Prove the rank-nullity theorem: For a linear map \(T: V \to W\) between finite-dimensional vector spaces, \(\dim(V) = \text{rank}(T) + \text{nullity}(T)\).

Full Formal Proof: Let \(V\) have dimension \(n\). Let \(\{ \mathbf{k}_1, \ldots, \mathbf{k}_p \}\) be a basis for \(\ker(T)\) (so \(p = \text{nullity}(T)\)). Since this is a linearly independent set in \(V\), we can extend it to a basis of \(V\): \(\{ \mathbf{k}_1, \ldots, \mathbf{k}_p, \mathbf{v}_1, \ldots, \mathbf{v}_q \}\) where \(p + q = n\). Claim: \(\{ T(\mathbf{v}_1), \ldots, T(\mathbf{v}_q) \}\) is a basis for \(\text{im}(T)\).

Spanning: Any element of \(\text{im}(T)\) is \(T(\mathbf{v})\) for some \(\mathbf{v} \in V\). Write \(\mathbf{v} = \sum_{i=1}^p a_i \mathbf{k}_i + \sum_{j=1}^q b_j \mathbf{v}_j\). Then \(T(\mathbf{v}) = \sum_{i=1}^p a_i T(\mathbf{k}_i) + \sum_{j=1}^q b_j T(\mathbf{v}_j) = \sum_{j=1}^q b_j T(\mathbf{v}_j)\) (since \(T(\mathbf{k}_i) = \mathbf{0}\)).

Linear independence: If \(\sum_{j=1}^q c_j T(\mathbf{v}_j) = \mathbf{0}\), then \(T(\sum_{j=1}^q c_j \mathbf{v}_j) = \mathbf{0}\), so \(\sum_{j=1}^q c_j \mathbf{v}_j \in \ker(T)\). But \(\ker(T) = \text{span}\{ \mathbf{k}_1, \ldots, \mathbf{k}_p \}\), and \(\{ \mathbf{k}_1, \ldots, \mathbf{k}_p, \mathbf{v}_1, \ldots, \mathbf{v}_q \}\) is linearly independent, so \(\sum_{j=1}^q c_j \mathbf{v}_j = \mathbf{0}\) implies \(c_j = 0\) for all \(j\).

Thus, \(\{ T(\mathbf{v}_1), \ldots, T(\mathbf{v}_q) \}\) is a basis for \(\text{im}(T)\), so \(\text{rank}(T) = q\). Therefore, \(\dim(V) = p + q = \text{nullity}(T) + \text{rank}(T)\). \(\square\)

Proof Strategy & Techniques: The strategy is basis extension: start with a basis of the kernel, extend it to a basis of the domain, and show that the “extension part” maps to a basis of the image. This is a fundamental proof technique in linear algebra: using basis vectors to partition and control dimensions. The proof avoids coordinates; it works purely with abstract vector space arguments.

Computational Validation: For \(A = \begin{pmatrix} 1 & 2 & 3 \\ 0 & 1 & 2 \end{pmatrix}\), compute rank (via row reduction: rank = 2) and nullity (via kernel basis: nullity = 1). Verify: rank + nullity = 2 + 1 = 3 = num columns ✓.

ML Interpretation: In neural networks, rank-nullity partitions the input space into “lost” (kernel) and “preserved” (mapped to image) directions. For a layer that maps high-dim inputs to low-dim outputs, the large kernel captures all the discarded information. Understanding the split is key to diagnosing information flow.

Generalization & Edge Cases: Holds for any finite-dimensional vector spaces over any field. For infinite-dimensional spaces, a generalization exists but requires care with infinite sums and convergence. Edge case: if \(\ker(T) = V\) (e.g., zero map), then rank = 0, nullity = dim(V).

Failure Mode Analysis: Students sometimes swap rank and nullity or forget to include both in the sum. Another error: applying the theorem to infinite-dimensional spaces without understanding the necessary modifications. Some incorrectly try to apply it to non-linear maps.

Historical Context: Formulated explicitly by Sylvester in the mid-1800s, though the ideas predate this. Modern proof via basis extension became standard in 20th-century algebra texts. Central to homological algebra and category theory.

Traps: (1) Confusing “dim(V)” with “dim(W)”; the theorem is about domain dimension, not codomain. (2) Numerical rank detection: if a matrix is nearly singular, the rank is ambiguous near floating-point limits. (3) The theorem gives rank in terms of nullity (or vice versa), not absolute bounds on either; both must sum to domain dimension.

B.5. Prove that the image (range) of a linear map \(T: V \to W\) is a subspace of \(W\).

Full Formal Proof: The image is \(\text{im}(T) = \{ T(\mathbf{v}) : \mathbf{v} \in V \}\). We verify three subspace criteria.

(1) Zero vector: \(T(\mathbf{0}_V) = \mathbf{0}_W\) by linearity, so \(\mathbf{0}_W \in \text{im}(T)\).

(2) Closure under addition: If \(\mathbf{w}_1, \mathbf{w}_2 \in \text{im}(T)\), then \(\mathbf{w}_1 = T(\mathbf{v}_1)\) and \(\mathbf{w}_2 = T(\mathbf{v}_2)\) for some \(\mathbf{v}_1, \mathbf{v}_2 \in V\). Then \(\mathbf{w}_1 + \mathbf{w}_2 = T(\mathbf{v}_1) + T(\mathbf{v}_2) = T(\mathbf{v}_1 + \mathbf{v}_2) \in \text{im}(T)\) by linearity.

(3) Closure under scalar multiplication: If \(\mathbf{w} \in \text{im}(T)\) and \(c\) is a scalar, then \(\mathbf{w} = T(\mathbf{v})\) for some \(\mathbf{v}\). Then \(c\mathbf{w} = cT(\mathbf{v}) = T(c\mathbf{v}) \in \text{im}(T)\) by linearity. \(\square\)

Proof Strategy & Techniques: The strategy mirrors the kernel proof: apply the subspace criterion directly, using linearity to establish closure properties. The key insight is that linearity (additivity and homogeneity) is precisely the structure needed to ensure the image inherits subspace properties.

Computational Validation: For \(T(\mathbf{x}) = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \mathbf{x}\), the image is spanned by the columns \(\begin{pmatrix} 1 \\ 3 \end{pmatrix}\) and \(\begin{pmatrix} 2 \\ 4 \end{pmatrix}\). Verify: (a) take two outputs, their sum is in the span; (b) scaling an output keeps it in the span. Direct computation confirms closure.

ML Interpretation: The image of a layer acts as the “effective output space”—the manifold of all possible outputs the layer can produce. Understanding this as a subspace means the layer compresses the input into a structured subspace. In deep networks, successive layers restrict to increasingly constrained output subspaces.

Generalization & Edge Cases: Holds for any linear map over any field. Edge case: image = \(\{ \mathbf{0} \}\) iff the map is the zero map. Edge case: image = \(W\) iff the map is surjective.

Failure Mode Analysis: A common error is assuming the image is always closed under addition without proof. Another mistake: confusing the image with the “range” in terms of attainable values not in the codomain; the image is always contained in the codomain. A third error: conflating the image with the column space in a matrix context; they are the same, but the conceptual distinction matters.

Historical Context: The formal notion of image (or range) as a subspace emerged with the abstract formalization of linear algebra in the early 20th century. Hausdorff and others developed the machinery to treat images and kernels symmetrically.

Traps: (1) Forgetting that the image is in the codomain, not the domain. (2) Assuming the image is always full-rank or full-dimensional; it need not be. (3) Confusing the image with the kernel; they partition the geometry of the map differently.

B.6. Let \(T: \mathbb{R}^n \to \mathbb{R}^m\) be a linear map with matrix representation \(A \in \mathbb{R}^{m \times n}\) (in the standard bases). Prove that \(\text{rank}(A) = \text{rank}(T)\) (i.e., rank is independent of the choice of bases).

Full Formal Proof: Rank is defined as \(\text{rank}(T) = \dim(\text{im}(T))\). The image of \(T\) is the set of all \(T(\mathbf{x}) = A\mathbf{x}\), which is exactly the column space of \(A\). By definition, the rank of a matrix is the dimension of its column space. Thus, \(\text{rank}(A) = \dim(\text{Col}(A)) = \dim(\text{im}(T)) = \text{rank}(T)\). This holds regardless of basis choice because the definition of the linear map is basis-independent; changing bases changes the matrix representation, but the linear map (and its image dimension) remains invariant. \(\square\)

Proof Strategy & Techniques: This proof hinges on understanding that rank has two equivalent definitions: (1) dimension of the column space (matrix definition), (2) dimension of the image (map definition). The strategy is to show they are the same for the standard representation. The key technique is recognizing that the matrix entries depend on bases, but the image dimension does not.

Computational Validation: For \(A = \begin{pmatrix} 1 & 2 & 3 \\ 0 & 1 & 4 \\ 1 & 3 & 7 \end{pmatrix}\), compute rank via row reduction (rank = 2 because the third row is redundant). Now compute the image via T: Span of columns is 2-dimensional (the third column is a linear combination of the first two). For a different basis \(P\), rank(\(P^{-1}AP\)) = 2 (same). Numerically verify via SVD in both coordinate systems.

ML Interpretation: In neural networks, the rank of a layer is a geometric quantity—the effective dimensionality of the output. This rank does not depend on how we represent the weight matrix (choice of basis); it’s an intrinsic property of the linear transformation. This invariance is crucial for understanding generalization: the capacity of a layer is determined by rank, not by parameterization.

Generalization & Edge Cases: For any finite-dimensional vector spaces and any linear map, rank (as the dimension of the image) is basis-independent. This is why rank is such a fundamental quantity: it captures intrinsic geometry, not representation.

Failure Mode Analysis: A common error is believing that the rank of a matrix representation changes when you change bases. It does not (the dimension of the image is basis-independent). Another error: confusing this with the fact that the matrix entries change with basis; they do, but the rank (dimensionality) stays the same. A third error: applying row reduction to matrices in non-standard bases and expecting the rank to match the standard representation without verifying basis changes.

Historical Context: Rank as a basis-independent concept became clear through work on abstract linear algebra and representation theory in the early-to-mid 20th century. The invariance of rank under similarity transformations is central to the structure theory of matrices.

Traps: (1) Assuming the matrix representation rank is basis-dependent; it is not. (2) Confusing “row rank = column rank” with “rank is basis-independent”; both are true but conceptually distinct. (3) Numerical pitfall: near-zero eigenvalues or singular values might make rank ambiguous; use a tolerance.

B.7. Prove that a linear map \(T: V \to V\) on a finite-dimensional vector space is invertible if and only if it is both injective and surjective.

Full Formal Proof: We prove both directions.

(\(\Rightarrow\)) If \(T\) is invertible, then it is bijective (by definition of invertibility). Injectivity and surjectivity are the defining components of bijectivity.

(\(\Leftarrow\)) If \(T\) is injective and surjective, then it is invertible. By rank-nullity, \(\dim(V) = \text{rank}(T) + \text{nullity}(T)\). Injectivity means \(\ker(T) = \{ \mathbf{0} \}\), so \(\text{nullity}(T) = 0\), thus \(\text{rank}(T) = \dim(V)\). Surjectivity means \(\text{im}(T) = V\), so \(\dim(\text{im}(T)) = \dim(V) = \text{rank}(T)\). Since \(T: V \to V\) is surjective, the image is all of \(V\). Since \(T\) is also injective, the map is bijective. Bijectivity is equivalent to the existence of an inverse \(T^{-1}: V \to V\) defined by \(T^{-1}(T(\mathbf{v})) = \mathbf{v}\) and \(T(T^{-1}(\mathbf{w})) = \mathbf{w}\) for all \(\mathbf{v}, \mathbf{w} \in V\). Thus, \(T\) is invertible. \(\square\)

Proof Strategy & Techniques: The forward direction is definitional. The reverse direction uses rank-nullity to translate injectivity and surjectivity into rank conditions, then combines them to conclude invertibility. The key insight is that for an endomorphism (map from space to itself), injectivity and surjectivity are equivalent when the domain and codomain have the same dimension.

Computational Validation: For \(T(\mathbf{x}) = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix} \mathbf{x}\), compute: rank = 2 (full), nullity = 0 (injection), image = \(\mathbb{R}^2\) (surjection). Find the inverse: \(T^{-1} = \begin{pmatrix} 1 & -1 \\ 0 & 1 \end{pmatrix}\). Verify \(T \circ T^{-1} = I\).

ML Interpretation: In neural networks, invertibility is rare (most layers are non-invertible bottlenecks). However, when designing invertible architectures (flow-based models, reversible networks), this criterion guides design: ensure each layer is bijective, preserving dimensionality throughout and enabling invertible transformations. Invertible layers enable efficient sampling and likelihood computation.

Generalization & Edge Cases: The theorem assumes finite-dimensional spaces (critical: in infinite dimensions, injectivity does not imply surjectivity for endomorphisms). Edge case: 0-dimensional space; the zero map is both injective and surjective.

Failure Mode Analysis: A common error is assuming injectivity alone implies invertibility (it does not without surjectivity). Another error: applying the theorem to maps \(T: V \to W\) with dim(V) ≠ dim(W); the result does not hold. A third error: assuming invertibility without checking both injectivity and surjectivity.

Historical Context: This characterization formalized through the development of abstract linear algebra and became standard in 20th-century linear algebra texts. The equivalence is a cornerstone of finite-dimensional linear algebra.

Traps: (1) Forgetting the theorem requires the domain and codomain to be the same space and have the same dimension. (2) In infinite dimensions, injectivity + surjectivity does not guarantee continuity of the inverse; additional assumptions are needed. (3) For matrices, det(A) ≠ 0 is equivalent, but the determinant is less intuitive than rank-nullity.

B.8. Let \(T_1: U \to V\) and \(T_2: V \to W\) be linear maps. Prove that \(\text{rank}(T_2 \circ T_1) \leq \min(\text{rank}(T_1), \text{rank}(T_2))\), and provide a concrete example showing each inequality can be tight.

Full Formal Proof: The composition \(T_2 \circ T_1\) maps \(U \to W\). The image of the composition is \(\text{im}(T_2 \circ T_1) = \{ T_2(T_1(\mathbf{u})) : \mathbf{u} \in U \} = \{ T_2(\mathbf{v}) : \mathbf{v} \in \text{im}(T_1) \} = T_2(\text{im}(T_1))\). Thus, \(\text{im}(T_2 \circ T_1) = T_2(\text{im}(T_1)) \subseteq \text{im}(T_2)\).

Taking dimensions, \(\text{rank}(T_2 \circ T_1) = \dim(\text{im}(T_2 \circ T_1)) \leq \dim(\text{im}(T_2)) = \text{rank}(T_2)\).

Additionally, \(\text{im}(T_2 \circ T_1) \subseteq T_2(\text{im}(T_1)) \subseteq T_2(V) = \text{im}(T_2)\), but more specifically, \(\text{im}(T_2 \circ T_1) = T_2(\text{im}(T_1))\). The dimension of the image of a restriction is at most the dimension of the domain of restriction, so \(\dim(T_2(\text{im}(T_1))) \leq \dim(\text{im}(T_1)) = \text{rank}(T_1)\).

Thus, \(\text{rank}(T_2 \circ T_1) \leq \min(\text{rank}(T_1), \text{rank}(T_2))\). \(\square\)

Concrete examples: (1) Example where rank(\(T_2 \circ T_1\)) = min(rank(\(T_1\)), rank(\(T_2\))): Let \(T_1: \mathbb{R}^3 \to \mathbb{R}^2\) with rank 2, and \(T_2: \mathbb{R}^2 \to \mathbb{R}^3\) with rank 2. Then rank(\(T_2 \circ T_1\)) can be 2 (tight on both). (2) Example where rank(\(T_2 \circ T_1\)) < min: \(T_1: \mathbb{R}^3 \to \mathbb{R}^2\) with rank 2 (image is 2D subspace), and \(T_2: \mathbb{R}^2 \to \mathbb{R}^2\) that projects onto a 1D subspace (rank 1). Then rank(\(T_2 \circ T_1\)) = 1 < min(2, 1) = 1 (actually equal here, but the composition could be smaller if T_2 maps the image of T_1 to a smaller subspace).

Proof Strategy & Techniques: The key insight is that the image of the composition is the image of the restriction of \(T_2\) to the image of \(T_1\). Dimensionality arguments then follow: the image of a restriction cannot be larger than the domain of restriction or the image of the full map.

Computational Validation: Let \(A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{pmatrix}\) (rank 2, maps \(\mathbb{R}^2 \to \mathbb{R}^3\)) and \(B = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{pmatrix}\) (rank 2, maps \(\mathbb{R}^3 \to \mathbb{R}^2\)). Then \(BA = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}\) has rank 2. Verify: rank(BA) = 2 = min(2, 2) ✓.

ML Interpretation: In deep networks, this result explains the bottleneck phenomenon. Each composition of layers can only be as expressive as the most restrictive layer. A rank-10 bottleneck layer forces all subsequent compositions to have rank ≤ 10, regardless of the ranks of the downstream layers. This is why architecture design matters: identify and repair low-rank bottlenecks.

Generalization & Edge Cases: Holds for any composition of linear maps. Edge case: if either \(T_1\) or \(T_2\) is the zero map, rank = 0. Edge case: if both are injective and surjective (bijections), the rank is preserved through composition.

Failure Mode Analysis: A common error is assuming rank(\(T_2 \circ T_1\)) = rank(\(T_1\)) × rank(\(T_2)\) (multiplication is wrong; the bound is via min, not product). Another error: thinking the bound always equals the min (it can be strictly less). A third error: applying this to nonlinear compositions or assuming the bound without understanding the geometry.

Historical Context: Rank composition bounds emerged naturally from the theory of linear transformations and are central to the theory of linear codes and channel capacity in information theory.

Traps: (1) Confusing the min rule with product. (2) Thinking rank(\(T_2 \circ T_1\)) = min always (it can be strictly less). (3) Assuming commutativity or other algebraic properties that don’t hold for composition; \(T_1 \circ T_2 \neq T_2 \circ T_1\) in general.

[Continuing B.9–B.20 with full 8-component detail. Each comprises ~300-400 words covering all 8 components in depth.]

B.9. For an autoencoder with an encoder \(E: \mathbb{R}^n \to \mathbb{R}^k\) (linear, with rank \(k < n\)) and a decoder \(D: \mathbb{R}^k \to \mathbb{R}^n\) (linear), prove that the composition \(D \circ E\) cannot equal the identity map, even if \(D\) is chosen optimally.

Full Formal Proof: The composition \(D \circ E\) maps \(\mathbb{R}^n \to \mathbb{R}^n\). By the rank composition bound, \(\text{rank}(D \circ E) \leq \min(\text{rank}(E), \text{rank}(D)) \leq \min(k, k) = k < n\) (using the given constraint rank(\(E) = k < n\) and the fact that rank(\(D\)) ≤ min(output dimension of E, n) = min(k, n) = k). Since the rank of \(D \circ E\) is strictly less than \(n\), the map is not surjective onto \(\mathbb{R}^n\); its image is a proper subspace. The identity map \(I: \mathbb{R}^n \to \mathbb{R}^n\) has rank \(n\). Therefore, \(D \circ E \neq I\). \(\square\)

Proof Strategy & Techniques: The proof uses the rank composition bound as the central tool. The strategy is: (1) apply the rank bound to the composition, (2) show that rank(\(D \circ E)\)) ≤ \(k < n\), (3) conclude that it cannot be the identity (which has rank \(n\)). The key insight is that dimensionality is a hard constraint: you cannot create new dimensions through linear transformations.

Computational Validation: For \(E = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{pmatrix}\) (encodes 3D to 2D, rank 2) and \(D = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{pmatrix}\) (decodes 2D to 3D, rank 2), the composition is \(DE = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}\) with rank 2 < 3. The third dimension is lost in the bottleneck; no choice of \(D\) fixes this.

ML Interpretation: Autoencoders compress information through a bottleneck (low-rank encoder). The decoder cannot recover the lost information no matter its design, as long as it is linear. This is why: (1) we accept lossy compression (not all information can be recovered), (2) nonlinear decoders can help (but are not our focus), (3) the bottleneck dimension limits expressivity fundamentally. This geometric fact is at the heart of dimensionality reduction and information bottleneck theory.

Generalization & Edge Cases: The result holds for any linear encoder/decoder with rank(\(E) < n\). If rank(\(E) = n\), then the encoder is injective, and an optimal decoder can be chosen to recover the identity (if the codomain of \(E\) and domain of \(D\) align). Edge case: if the encoder is surjective (rank = output dimension), then reconstruction is possible.

Failure Mode Analysis: A common error is assuming that with enough training a decoder can overcome the bottleneck. It cannot, if both encoder and decoder are linear. Another error: confusing this with the existence of a pseudo-inverse; even a pseudo-inverse cannot recover lost dimensions. A third error: assuming the identity can be approximated arbitrarily closely; the gap is structural, not just numerical.

Historical Context: The relationship between rank and information loss is foundational to coding theory and signal processing, dating back to Shannon’s information theory. The bottleneck principle became formalized in deep learning through works on the information bottleneck method (Tishby et al., 2000s).

Traps: (1) Assuming nonlinear (learnable) decoders can recover information; they can, but linear ones cannot. (2) Confusing “cannot equal identity” with “useless”; autoencoders are still valuable for compression, even if they lose information. (3) Thinking the bound is tight in practice; real autoencoders may have additional constraints (Gaussian noise, sparsity) that affect reconstruction further.

B.10. Prove that a change-of-basis transformation \(A' = P^{-1}AP\) preserves rank: \(\text{rank}(A) = \text{rank}(A')\) for any invertible matrix \(P\).

Full Formal Proof: Multiplication by an invertible matrix does not change rank. Specifically, \(\text{rank}(A') = \text{rank}(P^{-1}AP)\). Since \(P^{-1}\) is invertible, multiplying on the left by \(P^{-1}\) does not change rank: \(\text{rank}(P^{-1}AP) = \text{rank}(AP)\) (because \(P^{-1}\) is bijective, its image has the same dimension as its domain). Similarly, multiplying on the right by the invertible matrix \(P\) does not change rank: \(\text{rank}(AP) = \text{rank}(A)\) (the image of \(AP\) is the image of \(A\) restricted to the column space of \(P\); since \(P\) is invertible, its column space is the entire codomain, so no information is lost). Thus, \(\text{rank}(A') = \text{rank}(A)\). \(\square\)

Proof Strategy & Techniques: The proof relies on the fact that left/right multiplication by invertible matrices is rank-preserving. This is because invertible matrices are bijections; they map subspaces to subspaces of the same dimension, preserving rank. The strategy is: (1) recognize \(P^{-1}AP\) as a sequence of rank-preserving operations, (2) apply associativity and rank preservation to each step, (3) conclude.

Computational Validation: Let \(A = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}\) (rank 2) and \(P = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}\) (invertible). Compute \(P^{-1} = \begin{pmatrix} 1 & -1 \\ 0 & 1 \end{pmatrix}\) and \(A' = P^{-1}AP\). By numerical computation, rank(\(A) =\) rank(\(A') = 2\) ✓.

ML Interpretation: When we change coordinate systems (bases) for neural network weights, the rank—an intrinsic property of the linear transformation—does not change. This is why rank is such a powerful tool: it is basis-independent, reflecting true geometric properties of the network layer regardless of parameterization. Different parameter initializations or training procedures might produce matrices in different bases, but the rank remains constant.

Generalization & Edge Cases: Holds for any invertible \(P\). If \(P\) is orthogonal, the transformation preserves additional structure (norms, angles). If \(P\) is singular, the result no longer holds; rank can decrease.

Failure Mode Analysis: A common error is assuming similarity violations affect rank properties. They do not. Another error: confusing rank preservation with invariance of all spectral properties (eigenvalues, determinant can change). A third error: applying the result to singular matrices \(P\); similarity is defined only for invertible \(P\).

Historical Context: Rank as a basis-invariant quantity emerged through the formalization of linear algebra and the theory of similarity transformations in the 19th and early 20th centuries. It is a cornerstone of Jordan normal form and spectral theory.

Traps: (1) Assuming other properties (eigenvalues, determinant) are preserved; many are not, even though rank is. (2) Thinking numerical errors in computing \(P^{-1}\) mean rank changes; it does not, though numerical error in detection might occur. (3) Confusing similarity with other matrix relationships (congruence, equivalence); each preserves different properties.

B.11. Prove that if \(T: V \to W\) is an injective linear map between finite-dimensional spaces with \(\dim(V) = \dim(W)\), then \(T\) is surjective (and hence bijective).

Full Formal Proof: Let \(n = \dim(V) = \dim(W)\). By rank-nullity, \(\dim(V) = \text{rank}(T) + \text{nullity}(T)\), so \(n = \text{rank}(T) + \text{nullity}(T)\). Injectivity of \(T\) means \(\ker(T) = \{ \mathbf{0} \}\), so \(\text{nullity}(T) = 0\). Thus, \(\text{rank}(T) = n = \dim(W)\). Since \(\text{rank}(T) = \dim(W)\), the image of \(T\) is an \(n\)-dimensional subspace of \(W\), which is an \(n\)-dimensional space. The only \(n\)-dimensional subspace of an \(n\)-dimensional space is the space itself, so \(\text{im}(T) = W\), meaning \(T\) is surjective. Since \(T\) is both injective and surjective, it is bijective. \(\square\)

Proof Strategy & Techniques: The proof uses rank-nullity and the fact that an \(n\)-dimensional subspace of an \(n\)-dimensional space is the entire space. The strategy is: (1) apply rank-nullity with injectivity to conclude rank = dim(W), (2) identify the image as an \(n\)-dimensional subspace of an \(n\)-dimensional space, (3) conclude surjectivity.

Computational Validation: For \(T(\mathbf{x}) = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \mathbf{x}: \mathbb{R}^2 \to \mathbb{R}^2\), the nullspace contains only \(\mathbf{0}\) (check \(\det = -2 \neq 0\)), so injection is confirmed. The rank is 2 = dim codomain, so surjectivity follows. The matrix is invertible ✓.

ML Interpretation: In deep networks, if a layer is injective (no information loss in a forward pass), and the input and output dimensions are equal, then the layer must be surjective (can reach all output points). This characterizes invertible layers precisely: injectivity + equal dimensions = invertibility. This is important for designing reversible networks and flow-based generative models.

Generalization & Edge Cases: The theorem is specific to finite-dimensional spaces with equal dimension. In infinite dimensions, injectivity does not imply surjectivity. Edge case: 0-dimensional spaces; the zero map is trivially bijective.

Failure Mode Analysis: A common error is applying the theorem to maps with dim(V) ≠ dim(W); it does not hold. Another error: confusing injectivity with surjectivity without the dimension assumption; injectivity alone does not guarantee surjectivity. A third error: assuming the converse (surjectivity + equal dimension ⟹ injectivity) without proof; it is true but must be verified.

Historical Context: This result is a direct consequence of rank-nullity and is standard in finite-dimensional linear algebra. It is a key tool in the Fredholm alternative and theory of operator equations.

Traps: (1) Forgetting the dimension equality assumption; it is crucial. (2) Assuming the result extends to infinite dimensions; it does not without additional conditions (Banach space theory provides generalizations via open mapping theorem). (3) Thinking injectivity alone is sufficient; it is not.

B.12. For a regression problem with design matrix \(X \in \mathbb{R}^{n \times p}\) (with \(n > p\)) and response vector \(\mathbf{y} \in \mathbb{R}^n\), prove that if \(\text{rank}(X) = p\), then the least-squares solution \(\hat{\mathbf{w}} = (X^\top X)^{-1} X^\top \mathbf{y}\) exists and is unique.

Full Formal Proof: The least-squares solution minimizes \(\| X\mathbf{w} - \mathbf{y} \|^2\). The normal equations are \(X^\top X \mathbf{w} = X^\top \mathbf{y}\). If rank(\(X) = p\), then the columns of \(X\) are linearly independent. This means \(X^\top X\) is an invertible \(p \times p\) matrix: (1) \(X^\top X\) is \(p \times p\); (2) rank(\(X^\top X\)) = rank(\(X\)) = p ) (because rank is preserved under matrix multiplication by full-rank matrices); (3) since \(X^\top X\) is \(p \times p\) with rank \(p\), it is invertible. Thus, the solution \(\hat{\mathbf{w}} = (X^\top X)^{-1} X^\top \mathbf{y}\) is well-defined and unique. \(\square\)

Proof Strategy & Techniques: The key insight is that full column rank of \(X\) translates to invertibility of the Gram matrix \(X^\top X\). The strategy is: (1) identify the normal equations, (2) interpret the rank condition, (3) conclude invertibility of \(X^\top X\), (4) solve for the unique solution.

Computational Validation: For \(X = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}\) (rank 2) and \(\mathbf{y} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\), compute \(X^\top X = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}\), which is invertible (det = 3). Solve for \(\hat{\mathbf{w}}\) explicitly.

ML Interpretation: In linear regression, the full column rank assumption ensures that the inverse problem has a unique solution. Violation of this assumption (rank deficiency, multicollinearity) leads to non-uniqueness or non-invertibility. Understanding this geometrically: the normal equations require solving a system in feature space; full rank ensures the design matrix spans the feature space sufficiently.

Generalization & Edge Cases: If rank(\(X) < p\), the system has infinitely many solutions (if consistent) or no solution. If \(n < p\), the system is underdetermined even with full rank.

Failure Mode Analysis: A common error is assuming the least-squares solution exists without checking rank(\(X) = p\). If rank(\(X) < p\) (multicollinearity), the normal equations have infinitely many solutions. Another error: confusing the existence of a least-squares solution (always exists geometrically) with uniqueness (requires full rank). A third error: numerically inverting \(X^\top X\) when it is ill-conditioned; regularization or other methods are better.

Historical Context: The least-squares method dates to Gauss and Legendre (early 1800s). The normal equations and their geometric interpretation developed through the 19th and 20th centuries. Modern regression theory is built on rank conditions and the invertibility of the Gram matrix.

Traps: (1) Assuming full rank without checking; multicollinearity is common in real data. (2) Numerically inverting \(X^\top X\) directly (unstable); use QR decomposition or SVD. (3) Confusing rank(\(X) = p\) with \(X\) having full rank in other senses; in the context of overdetermined systems, full rank = full column rank.

B.13. Prove that for a linear projection \(P: V \to V\) satisfying \(P^2 = P\), the kernel and image are complementary subspaces: \(V = \ker(P) \oplus \text{im}(P)\) (direct sum).

Full Formal Proof: We prove two conditions: (1) \(\ker(P) \cap \text{im}(P) = \{ \mathbf{0} \}\) (disjointness), and (2) \(\ker(P) + \text{im}(P) = V\) (spanning).

(1) Disjointness: Let \(\mathbf{v} \in \ker(P) \cap \text{im}(P)\). Then \(P(\mathbf{v}) = \mathbf{0}\) (from the kernel) and \(\mathbf{v} = P(\mathbf{u})\) for some \(\mathbf{u} \in V\) (from the image). Then \(\mathbf{0} = P(\mathbf{v}) = P(P(\mathbf{u})) = P^2(\mathbf{u}) = P(\mathbf{u}) = \mathbf{v}\) (using \(P^2 = P\)). Thus, \(\mathbf{v} = \mathbf{0}\).

(2) Spanning: Let \(\mathbf{v} \in V\). Write \(\mathbf{v} = P(\mathbf{v}) + (\mathbf{v} - P(\mathbf{v}))\). The first term \(P(\mathbf{v}) \in \text{im}(P)\) (by definition). For the second term, \(P(\mathbf{v} - P(\mathbf{v})) = P(\mathbf{v}) - P^2(\mathbf{v}) = P(\mathbf{v}) - P(\mathbf{v}) = \mathbf{0}\) (using \(P^2 = P\)), so \(\mathbf{v} - P(\mathbf{v}) \in \ker(P)\). Thus, every \(\mathbf{v} \in V\) is a sum of an element from \(\ker(P)\) and an element from \(\text{im}(P)\). \(\square\)

Proof Strategy & Techniques: The proof uses the projection property \(P^2 = P\) directly to establish disjointness and spanning. The key insight is the decomposition \(\mathbf{v} = P(\mathbf{v}) + (\mathbf{v} - P(\mathbf{v}))\), which naturally partitions the space. The strategy is: (1) use \(P^2 = P\) to verify disjointness, (2) apply the same property to justify the decomposition and spanning.

Computational Validation: For \(P = \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}\) (projection onto the \(x\)-axis, \(P^2 = P\) ✓), the kernel is the \(y\)-axis and the image is the \(x\)-axis. They are orthogonal, disjoint, and their sum is \(\mathbb{R}^2\).

ML Interpretation: Projections decompose a space into two complementary subspaces: what is “kept” (image) and what is “discarded” (kernel). In machine learning, this is used in dimensionality reduction: project high-dimensional data onto a low-dimensional subspace (image), and the kernel represents the “noise” or “irrelevant” directions. Understanding the direct sum emphasizes that the decomposition is complete and non-overlapping.

Generalization & Edge Cases: The result holds for any projection (operator with \(P^2 = P\)) on any vector space. Edge case: \(P = I\) (identity); image = V, kernel = \(\{ \mathbf{0} \}\). Edge case: \(P = \mathbf{0}\) (zero map); image = \(\{ \mathbf{0} \}\), kernel = V.

Failure Mode Analysis: A common error is assuming kernel and image are always disjoint; the projection property \(P^2 = P\) is crucial. Another error: not verifying both disjointness and spanning; both are needed for a direct sum. A third error: confusing orthogonal complement with direct sum complement; they are related but not identical.

Historical Context: Projections and their decomposition properties are central to functional analysis and the spectral theorem. The direct sum decomposition is a cornerstone of representation theory.

Traps: (1) Forgetting the \(P^2 = P\) condition; without it, kernel and image may not be disjoint. (2) Assuming orthogonal projections; the result holds for any projection. (3) Confusing complementary subspaces with orthogonal complements.

B.14. Prove that two matrices \(A, B \in \mathbb{R}^{m \times n}\) have the same rank if and only if they are related by a left-multiply by an invertible matrix and a right-multiply by an invertible matrix, i.e., \(B = MAQ\) for invertible \(M \in \mathbb{R}^{m \times m}\) and \(Q \in \mathbb{R}^{n \times n}\).

Full Formal Proof:

(\(\Rightarrow\)) Suppose rank(\(A) =\) rank(\(B) = r\). By rank-normal form, there exist invertible matrices \(P, Q\) such that \(A = P J_r Q^{-1}\) where \(J_r = \begin{pmatrix} I_r & 0 \\ 0 & 0 \end{pmatrix}\) (the canonical rank-\(r\) form). Similarly, \(B = P' J_r (Q')^{-1}\). Then \(B = P' J_r (Q')^{-1} = P' P^{-1} A Q J_r^{-1} J_r (Q')^{-1}\). Setting \(M = P' P^{-1}\) (invertible) and \(\tilde{Q} = Q J_r^{-1} J_r (Q')^{-1}\)… actually, a simpler argument: if rank(\(A) =\) rank(\(B)\), both can be transformed to the same canonical form \(J_r\) via row and column operations (invertible transformations). This gives \(MAQ_1 = J_r = M' B Q_1'\) for some invertible \(M, M', Q_1, Q_1'\). Thus, \(B = (M')^{-1} M A Q_1 (Q_1')^{-1}\), which is of the form \(B = M A Q\) for invertible \(M, Q\).

(\(\Leftarrow\)) If \(B = MAQ\) with \(M, Q\) invertible, then rank(\(B) =\) rank(\(MAQ) = \( rank(\( AQ)\) (since \(M\) is invertible) = rank(\(A)\) (since \(Q\) is invertible). Thus, rank(\(A) =\) rank(\(B)\). \(\square\)

Proof Strategy & Techniques: The forward direction applies the canonical (Smith normal) form; the backward direction uses rank preservation under invertible multiplications. The key insight is that rank is the only invariant of a matrix under simultaneous row and column operations (in the sense that two matrices have the same rank iff they can be transformed to the same canonical form).

Computational Validation: For \(A = \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}\) and \(B = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}\), both have rank 1. Find \(M = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}\) and \(Q = I\) such that \(B = MA\) (verify).

ML Interpretation: This characterization shows that rank is a complete invariant for matrix equivalence under row and column operations. In deep learning terms, this means two weight matrices with the same rank have the same “expressive capacity” in the sense that they perform equivalent linear transformations up to change-of-basis. This justifies analyzing networks by rank rather than by explicit matrix entries.

Generalization & Edge Cases: Holds for any finite-dimensional matrices over any field. The canonical form \(J_r\) is the same for all matrices of rank \(r\).

Failure Mode Analysis: A common error is thinking equivalence under row/column operations implies similarity (it does not; similarity is more restrictive). Another error: assuming \(A\) and \(B\) are related by multiplication of invertible matrices without the two-sided nature; both left and right multiplications are needed. A third error: confusing this with other equivalence relations (congruence, similarity).

Historical Context: The Smith normal form is attributed to Henry J. S. Smith (mid-1800s) and is a generalization of diagonalization to non-square matrices. It is fundamental in the theory of finitely generated abelian groups and (equivalently) module theory over \(\mathbb{Z}\).

Traps: (1) Thinking rank-equivalence is the same as similarity; it is more general. (2) Assuming the canonical form is unique (it is, up to permutation of diagonal entries). (3) Not recognizing that rank is the only invariant; all other properties (eigenvalues, norm, determinant) can differ.

B.15. For a deep linear network composed of weight matrices \(W_1 \in \mathbb{R}^{m_1 \times n}\), \(W_2 \in \mathbb{R}^{m_2 \times m_1}\), …, \(W_k \in \mathbb{R}^{m_k \times m_{k-1}}\), prove that \(\text{rank}(W_k \cdots W_2 W_1) \leq \min(\text{rank}(W_1), \text{rank}(W_2), \ldots, \text{rank}(W_k))\), and explain the implications for information bottlenecks in deep networks.

Full Formal Proof: We prove by induction on the number of matrices.

Base case: For \(k = 1\), rank(\(W_1) =\) rank(\(W_1)\), trivially true.

Inductive step: Assume rank(\(W_k \cdots W_1) \leq \min(\text{rank}(W_1), \ldots, \text{rank}(W_k))\). For \(k+1\) matrices, \(\text{rank}(W_{k+1} (W_k \cdots W_1)) \leq \min(\text{rank}(W_{k+1}), \text{rank}(W_k \cdots W_1))\) by the composition rank bound (B.8). By the inductive hypothesis, rank(\(W_k \cdots W_1) \leq\) min(rank(\(W_1), \ldots,\) rank(\(W_k)).\) Thus, \(\text{rank}(W_{k+1} (W_k \cdots W_1)) \leq \min(\text{rank}(W_{k+1}), \min(\text{rank}(W_1), \ldots, \text{rank}(W_k))) = \min(\text{rank}(W_1), \ldots, \text{rank}(W_{k+1}))\). \(\square\)

Proof Strategy & Techniques: Induction allows us to apply the pairwise composition bound repeatedly. The key insight is that the bottleneck (minimum rank) gets tighter as we compose more layers.

Computational Validation: For layers with ranks 10, 5, 20, 15, the composition has rank ≤ min(10, 5, 20, 15) = 5. The second layer (rank 5) is the bottleneck.

ML Interpretation: This is the information bottleneck theorem for linear networks: a single low-rank layer constrains the expressivity of the entire network downstream. For example, if layer 3 has rank 10, then the output of the network (after layer 5) cannot express features in more than 10 dimensions, regardless of the ranks of layers 4 and 5. This motivates: (1) identifying and repairing bottlenecks (ResNets add skip connections to bypass bottlenecks), (2) ensuring layer widths are sufficient to preserve information, (3) understanding why deep networks with bottlenecks underperform.

Generalization & Edge Cases: Holds for any number of layers. If any layer is rank 0 (zero map), the entire composition is rank 0. Edge case: if all layers are full-rank, the composition is also full-rank (up to dimension constraints).

Failure Mode Analysis: A common error is not recognizing which layer is the bottleneck. Another error: assuming that a layer has high rank based on its dimensions alone; dimensions do not guarantee rank. A third error: thinking that deep networks can overcome a bottleneck by learning; classically, they cannot (though nonlinearities can help).

Historical Context: The rank bottleneck in deep linear networks is a special case of the information bottleneck principle introduced by Tishby and Schwartz (1999). Subsequent work on information-theoretic analysis of deep learning built on this foundation.

Traps: (1) Confusing layer width with rank; width is an upper bound on rank, but rank can be much smaller. (2) Assuming the bottleneck’s position is fixed; rank can vary during training. (3) Thinking all layers with rank = min rank are equally responsible; the structure and composition order matter.

B.16. Prove that if \(T: V \to W\) is a surjective linear map between finite-dimensional spaces, then for any subspace \(U \subseteq W\), the preimage \(T^{-1}(U) = \{ \mathbf{v} \in V: T(\mathbf{v}) \in U \}\) satisfies \(\dim(T^{-1}(U)) = \dim(V) - \dim(W) + \dim(U)\).

Full Formal Proof: Since \(T\) is surjective, rank(\(T) = \dim(W)\). By rank-nullity, \(\dim(V) = \dim(W) + \text{nullity}(T)\), so \(\text{nullity}(T) = \dim(V) - \dim(W)\).

Now consider the restriction \(T|_{{T^{-1}(U)}}: T^{-1}(U) \to U\) (the restriction of \(T\) to the preimage). This map is surjective onto \(U\) (by definition of preimage). By rank-nullity applied to this restriction, \(\dim(T^{-1}(U)) = \text{rank}(T|_{{T^{-1}(U)}}) + \text{nullity}(T|_{{T^{-1}(U)}})\).

Rank of the restriction is \(\text{rank}(T|_{{T^{-1}(U)}}) = \dim(U)\) (since the restriction is surjective onto \(U\)).

Kernel of the restriction is \(\ker(T|_{{T^{-1}(U)}}) = T^{-1}(U) \cap \ker(T) = \ker(T)\) (since for any \(\mathbf{v} \in \ker(T)\), we have \(T(\mathbf{v}) = \mathbf{0} \in U\), so \(\mathbf{v} \in T^{-1}(U)\)). Thus, \(\text{nullity}(T|_{{T^{-1}(U)}}) = \dim(\ker(T)) = \text{nullity}(T) = \dim(V) - \dim(W)\).

Therefore, \(\dim(T^{-1}(U)) = \dim(U) + (\dim(V) - \dim(W)) = \dim(V) - \dim(W) + \dim(U)\). \(\square\)

Proof Strategy & Techniques: The key idea is to apply rank-nullity to the restriction of \(T\) to the preimage \(T^{-1}(U)\). This clever step converts the geometric problem (preimage dimension) into an algebraic one (rank and nullity of a restricted map).

Computational Validation: For \(T: \mathbb{R}^3 \to \mathbb{R}^2\) defined by \(T(\mathbf{x}) = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{pmatrix} \mathbf{x}\) (projection onto first two coordinates, rank 2, nullity 1), and \(U = \text{span}\{ \begin{pmatrix} 1 \\ 0 \end{pmatrix} \}\) (1-dimensional), the preimage \(T^{-1}(U)\) is \(\{ (a, 0, c): a \in \mathbb{R}, c \in \mathbb{R} \}\) (2-dimensional). Verify: dim(\(T^{-1}(U)) =\) 2 = 3 - 2 + 1 ✓.

ML Interpretation: In neural networks, this formula computes the dimension of the “effective input manifold” that maps to a given output subspace. For soft classifiers (output \(U\) = high-score region), this tells us the input manifold dimension of examples mapped to that region. Understanding preimage dimensions informs analysis of feature learning and how much input diversity is needed to saturate an output region.

Generalization & Edge Cases: Assumes surjectivity; without it, the restriction may not be surjective. If \(U = W\), then \(T^{-1}(U) = V\), so dim = dim(V) ✓. If \(U = \{ \mathbf{0} \}\), then dim(\(T^{-1}(U)) = \text{nullity}(T)\) ✓.

Failure Mode Analysis: A common error is forgetting the surjectivity assumption; the result does not hold without it. Another error: confusing dim(\(T^{-1}(U))\) with rank(\(T\)) restricted to \(U\); they are different quantities. A third error: not recognizing that the formula depends on the nullity, which is often overlooked.

Historical Context: This result is a consequence of rank-nullity applied to restrictions and is standard in functional analysis and the theory of linear equations. It is particularly important in numerical linear algebra (e.g., understanding solution manifolds of linear systems).

Traps: (1) Applying the formula without surjectivity. (2) Confusing dim(\(T^{-1}(U))\) with dim(\(U)\); the kernel contributes an additive term. (3) Not recognizing the geometric insight: the entire kernel \(\ker(T)\) is contained in every preimage.

B.17. Prove that a linear map \(T: V \to W\) is an isomorphism (bijective linear map) if and only if \(\text{rank}(T) = \dim(V) = \dim(W)\) (assuming finite-dimensional spaces).

Full Formal Proof:

(\(\Rightarrow\)) If \(T\) is an isomorphism, it is bijective. Bijectivity entails injectivity and surjectivity. Injectivity means ker(\(T) = \{ \mathbf{0} \}\), so nullity(\(T) = 0\). By rank-nullity, rank(\(T) = \dim(V) - 0 = \dim(V)\). Surjectivity means im(\(T) = W\), so rank(\(T) = \dim(W)\). Thus, rank(\(T) = \dim(V) = \dim(W)\).

(\(\Leftarrow\)) If rank(\(T) = \dim(V) = \dim(W)\), then by rank-nullity, nullity(\(T) = \dim(V) - \text{rank}(T) = 0\), so ker(\(T) = \{ \mathbf{0}\) } (injectivity). Also, rank(\(T) = \dim(W)\) means dim(im(\(T)) = \dim(W)\), so im(\(T) = W\) (surjectivity, since a subspace of \(W\) with the same dimension as \(W\) must be \(W\) itself). Thus, \(T\) is bijective, hence an isomorphism. \(\square\)

Proof Strategy & Techniques: The proof transforms the geometric definition (bijection) into the algebraic definition (rank-nullity conditions). The key is the equivalence between dimension-matching subspaces and the full space.

Computational Validation: For \(T: \mathbb{R}^2 \to \mathbb{R}^2\) with matrix \(\begin{pmatrix} 1 & 1 \\ 1 & 2 \end{pmatrix}\), rank = 2 = dim(domain) = dim(codomain). Thus, \(T\) is an isomorphism (invertible). The inverse exists: \(T^{-1} = \begin{pmatrix} 2 & -1 \\ -1 & 1 \end{pmatrix}\).

ML Interpretation: Isomorphic mappings preserve all geometric and algebraic structure; they are essentially “relabelings” of the space. In deep networks, isomorphic layers (bijections) are valuable for designing invertible architectures; they preserve dimensionality and allow lossless transformations. Understanding when a layer is an isomorphism is key to designing reversible networks and flow-based models.

Generalization & Edge Cases: The theorem is finite-dimensional. In infinite dimensions, dimension-matching does not guarantee isomorphism; continuity and open mapping theorems provide generalizations (Banach space theory).

Failure Mode Analysis: A common error is applying the theorem to maps with unequal domain/codomain dimensions. Another error: assuming rank = min(dim(V), dim(W)) is sufficient; it is not (must be equality). A third error: confusing isomorphism with similarity; isomorphisms preserve more structure.

Historical Context: Isomorphisms are central to algebraic structures and category theory. The rank-based characterization for finite-dimensional spaces is standard and ancient (in historical terms).

Traps: (1) Forgetting finite-dimensionality. (2) Assuming the rank condition without computing rank explicitly. (3) Not recognizing that an isomorphism is a very special map; in deep networks, most layers are non-isomorphic.

B.18. Prove that for the SVD decomposition \(A = U \Sigma V^\top\) of a matrix \(A \in \mathbb{R}^{m \times n}\), the rank of \(A\) equals the number of nonzero diagonal entries in the diagonal matrix \(\Sigma\), and that \(\text{rank}(A) \leq \min(m, n)\).

Full Formal Proof: The SVD writes \(A = U \Sigma V^\top\) where \(U \in \mathbb{R}^{m \times m}\) and \(V \in \mathbb{R}^{n \times n}\) are orthogonal (unitary), and \(\Sigma \in \mathbb{R}^{m \times n}\) is diagonal with entries \(\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_p \geq 0\) where \(p = \min(m, n)\).

The rank of \(A\) is the dimension of the column space of \(A\). Each column of \(A\) is a linear combination of the columns of \(U \Sigma\). The matrix \(U\) is full-rank (orthogonal), so the column space of \(A\) has the same dimension as the column space of \(U \Sigma\). Since \(U\) is full-rank, the rank of \(U \Sigma\) equals the rank of \(\Sigma\). The matrix \(\Sigma\) is diagonal, so its rank is the number of nonzero diagonal entries. Thus, rank(\(A) =\) number of nonzero \(\sigma_i\).

For the bound: \(\Sigma\) is \(m \times n\), so it has at most min(\(m, n)\) nonzero diagonal entries (the diagonal can have at most min(m, n) entries). Thus, rank(\(A) \leq \min(m, n)\). \(\square\)

Proof Strategy & Techniques: The proof uses the invariance of rank under multiplication by full-rank (orthogonal) matrices, then identifies rank with the number of nonzero diagonal entries in a diagonal matrix. The strategy is: (1) recognize \(A = U \Sigma V^\top\), (2) use rank preservation under orthogonal transformations, (3) count nonzero diagonals in \(\Sigma\).

Computational Validation: For \(A = \begin{pmatrix} 1 & 0 \\ 0 & 2 \\ 0 & 0 \end{pmatrix}\), the SVD is \(A = I \begin{pmatrix} 2 & 0 \\ 0 & 1 \\ 0 & 0 \end{pmatrix} I^\top\) (approximately; exact SVD requires orthogonal \(U, V\)). The rank is 2 (two nonzero singular values). Numerically compute SVD and verify.

ML Interpretation: The SVD reveals rank through the singular values. In deep learning, the spectrum of singular values reflects the effective dimensionality of a layer; a sharp cutoff in singular value decay indicates a low-rank structure (bottleneck). This is why SVD-based rank estimation is practical and reliable.

Generalization & Edge Cases: Holds for any matrix over any field (though singular value decomposition is specific to real/complex matrices). The bound rank ≤ min(m, n) is tight: full-rank matrices achieve it.

Failure Mode Analysis: A common error is thinking non-zero singular values in numerical computation are all truly nonzero; near-zero values might be numerical noise. Another error: assuming rank = min(m, n) without computing the SVD. A third error: confusing the rank (number of nonzero \(\sigma_i\)) with the sum or product of singular values.

Historical Context: SVD was formalized in the 19th century (Beltrami, Jordan, Sylvester) and became computationally central in the 20th century with the development of numerical algorithms. Modern SVD algorithms are among the most important tools in numerical linear algebra.

Traps: (1) Assuming all singular values are exactly zero or nonzero; numerical thresholding is needed. (2) Confusing singular values with eigenvalues; they are different. (3) Not recognizing that SVD is the most numerically stable way to compute rank.

B.19. For two linear maps \(T_1, T_2: V \to W\), prove that \(\text{rank}(T_1 + T_2) \leq \text{rank}(T_1) + \text{rank}(T_2)\), and provide a neural network interpretation (composition of layer outputs).

Full Formal Proof: The image of the sum is im(\(T_1 + T_2) = \{ (T_1 + T_2)(\mathbf{v}) : \mathbf{v} \in V \} = \{ T_1(\mathbf{v}) + T_2(\mathbf{v}) : \mathbf{v} \in V \} \subseteq \{ \mathbf{w}_1 + \mathbf{w}_2 : \mathbf{w}_1 \in \text{im}(T_1), \mathbf{w}_2 \in \text{im}(T_2) \} = \text{im}(T_1) + \text{im}(T_2)\) (the sum of subspaces).

By the dimension formula for sums of subspaces, \(\dim(\text{im}(T_1) + \text{im}(T_2)) \leq \dim(\text{im}(T_1)) + \dim(\text{im}(T_2)) = \text{rank}(T_1) + \text{rank}(T_2)\).

Thus, \(\text{rank}(T_1 + T_2) = \dim(\text{im}(T_1 + T_2)) \leq \text{rank}(T_1) + \text{rank}(T_2)\). \(\square\)

Proof Strategy & Techniques: The proof uses the subspace sum property and dimension bounds. The key insight is that the image of a sum is contained in the sum of images.

Computational Validation: For \(T_1(\mathbf{x}) = \begin{pmatrix} 1 \\ 0 \end{pmatrix} x_1\) and \(T_2(\mathbf{x}) = \begin{pmatrix} 0 \\ 1 \end{pmatrix} x_1\), each has rank 1. The sum \((T_1 + T_2)(\mathbf{x}) = \begin{pmatrix} 1 \\ 1 \end{pmatrix} x_1\) has rank 1 (equal to the sum of ranks). Numerically verify.

ML Interpretation: In neural networks with skip connections or ensemble methods, outputs from multiple layers (or models) are summed. The rank bound tells us that summing rank-\(r_1\) and rank-\(r_2\) features can produce at most rank-\(r_1 + r_2\) combined features. If the two features are orthogonal (disjoint images), the bound is tight; if they are aligned (overlapping images), the combined rank is smaller. This explains why diverse skip connections are valuable: they bring orthogonal information.

Generalization & Edge Cases: Holds for any linear maps to the same codomain. Equality holds when im(\(T_1) \cap \text{im}(T_2) = \{ \mathbf{0}\) }.

Failure Mode Analysis: A common error is assuming rank(\(T_1 + T_2) =\) rank(\(T_1) +\) rank(\(T_2)\) always (it can be strictly less). Another error: confusing operator sum with functional composition; they are different. A third error: not recognizing that the bound involves the image sum, not the domain.

Historical Context: Subspace sum properties and dimension bounds are foundational to linear algebra and functional analysis. The result is standard and ancient.

Traps: (1) Thinking the bound is always tight; it is tight only when images are disjoint. (2) Confusing T1 + T2 with T2 ∘ T1 (composition vs addition). (3) Assuming the rank of a sum relates to algebraic properties of individual ranks (it does not directly; the image geometry matters).

B.20. Prove that for a finite-dimensional vector space \(V\), a linear map \(T: V \to V\) is invertible if and only if \(\ker(T) = \{ \mathbf{0} \}\), and also if and only if \(\text{im}(T) = V\). (Show all three characterizations are equivalent.)

Full Formal Proof: Let \(n = \dim(V)\).

(1) \(T\) is invertible \(\Rightarrow\) ker(\(T) = \{ \mathbf{0} \}\)): If \(T\) is invertible, then \(T\) is injective (by definition). Injectivity means ker(\(T) = \{ \mathbf{0} \}\).

(2) ker(\(T) = \{ \mathbf{0} \}\) \(\Rightarrow\) im(\(T) = V\)): If ker(\(T) = \{ \mathbf{0}\) ), then nullity(\(T) = 0\). By rank-nullity, rank(\(T) = n - 0 = n\). Since rank(\(T) = \dim(V) = n\) and the image is a subspace of \(V\), the only \(n\)-dimensional subspace of \(V\) is \(V\) itself. Thus, im(\(T) = V\).

(3) im(\(T) = V\) \(\Rightarrow\) \(T\) is invertible): If im(\(T) = V\), then \(T\) is surjective. Combined with the fact that domain and codomain are equal (\(V\)), we apply B.11 (injectivity + surjectivity with equal-dim domain/codomain \(\Rightarrow\) isomorphic, hence invertible). Since surjectivity holds, we need to show injectivity. By rank-nullity, rank(\(T) + \text{nullity}(T) = n\). Surjectivity means rank(\(T) = n\), so nullity(\(T) = 0\), thus ker(\(T) = \{ \mathbf{0}\) ) (injectivity). Therefore, \(T\) is bijective, hence invertible.

We have shown: invertible \(\Rightarrow\) trivial kernel \(\Rightarrow\) surjectivity \(\Rightarrow\) invertibility. Thus, all three are equivalent. \(\square\)

Proof Strategy & Techniques: The proof chains three logical implications, forming a cycle. The key tools are rank-nullity and the equivalence between dimension-matching and full-dimensionality. Each implication uses a different facet: injectivity, rank-nullity, and dimension properties.

Computational Validation: For \(T(\mathbf{x}) = \begin{pmatrix} 2 & 1 \\ 1 & 1 \end{pmatrix} \mathbf{x}\), compute: ker(\(T) = \{ \mathbf{0} \}\) (via determinant ≠ 0); im(\(T) = \mathbb{R}^2\) (rank = 2 = dim); \(T\) is invertible (find \(T^{-1}\)).

ML Interpretation: This theorem is foundational for understanding invertibility in neural networks. An endomorphism (map from space to itself) is invertible iff it preserves dimensionality (im(\(T) = V)\) and has no null directions (ker(\(T) = \{ \mathbf{0} \}\)). In deep learning, designing invertible layers requires ensuring both properties hold throughout the network.

Generalization & Edge Cases: Specific to finite dimensions; infinite-dimensional generalizations require additional assumptions. Edge case: linear functionals (\(V \to \mathbb{R})\) cannot be invertible unless \(V = \mathbb{R}\).

Failure Mode Analysis: A common error is assuming one characterization (e.g., ker(\(T) = \{ \mathbf{0} \}\)) without verifying the others. Another error: applying the theorem to maps \(T: V \to W\) with V ≠ W; it does not hold in that generality (though B.11 provides a version for equal-dimensional domain/codomain). A third error: not recognizing that all three are equivalent; thinking one is weaker than the others.

Historical Context: These three characterizations are foundational and date back centuries. They form the basis of the theory of invertible matrices and endomorphisms.

Traps: (1) Assuming the result holds for \(T: V \to W\) with V ≠ W; it does not (though near-equivalences exist). (2) Thinking one condition (e.g., surjectivity) is sufficient without the others; for endomorphisms, they are equivalent, but the implication structure matters. (3) Not recognizing the deep geometric meaning: invertibility is a dual property (both null and “null complement” must be preserved).

Solutions to C. Python Exercises

C.1. Implementing Kernel Computation via Row Reduction

Code:

import numpy as np

def compute_kernel(A, tol=1e-9):
    """
    Compute the kernel (null space) of matrix A via row reduction to RREF.
    
    Args:
        A: numpy array of shape (m, n)
        tol: threshold for treating values as zero (numerical stability)
    
    Returns:
        kernel_basis: numpy array of shape (n, nullity) - column vectors form basis
        rank: rank of A
    """
    A = A.astype(float)
    m, n = A.shape
    
    # Perform Gaussian elimination with partial pivoting to get RREF
    rref, pivot_cols = sympy_rref(A, tol)  # Use SymPy for exact rational arithmetic, or implement custom RREF
    
    rank = len(pivot_cols)
    free_cols = [i for i in range(n) if i not in pivot_cols]
    
    # Construct kernel basis from free variable dependencies
    kernel_basis = np.zeros((n, len(free_cols)))
    
    for j, free_col in enumerate(free_cols):
        vec = np.zeros(n)
        vec[free_col] = 1.0
        for i, pivot_col in enumerate(pivot_cols):
            vec[pivot_col] = -rref[i, free_col]
        kernel_basis[:, j] = vec
    
    return kernel_basis, rank

def sympy_rref(A, tol):
    """Compute RREF using row operations."""
    from sympy import Matrix as SympyMatrix
    M = SympyMatrix(A.tolist())
    rref_M, pivot_cols = M.rref()
    rref_array = np.array(rref_M.tolist(), dtype=float)
    return rref_array, pivot_cols

Expected Output: For matrix \(A = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \end{pmatrix}\) (rank 1, nullity 2): - kernel_basis shape: (3, 2) - One basis vector: \(\begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix}\) - Another: \(\begin{pmatrix} -3 \\ 0 \\ 1 \end{pmatrix}\) - Verification: \(A \times \text{kernel\_basis} \approx \mathbf{0}_{2 \times 2}\)

Numerical / Shape Notes: - For a 5 × 8 matrix: expect nullity = 8 - rank (where rank ≤ 5) - Numerical stability: Use partial pivoting; treat entries with absolute value < \(10^{-9}\) as zero - Rank verified via rank-nullity: rank + nullity = n = 8 - Complexity: O(m²n) for RREF computation

C.2. Computing Image / Column Space

Code:

import numpy as np

def compute_image_basis(A, method='qr', tol=1e-9):
    """
    Compute orthonormal basis for the image (column space) of A.
    
    Args:
        A: numpy array of shape (m, n)
        method: 'qr', 'svd', or 'rref'
        tol: numerical tolerance
    
    Returns:
        image_basis: orthonormal basis (m, rank) array
        rank: rank of A
    """
    A = A.astype(float)
    m, n = A.shape
    
    if method == 'qr':
        Q, R = np.linalg.qr(A)
        rank = np.sum(np.abs(np.diag(R)) > tol)
        image_basis = Q[:, :rank]
    
    elif method == 'svd':
        U, s, Vt = np.linalg.svd(A, full_matrices=False)
        rank = np.sum(s > tol)
        image_basis = U[:, :rank]
    
    elif method == 'rref':
        # Use RREF to identify pivot columns; image is span of original pivot columns
        rref, pivot_cols = sympy_rref(A, tol)
        rank = len(pivot_cols)
        pivot_cols_list = list(pivot_cols)
        image_basis = A[:, pivot_cols_list]  # Unnormalized; optionally orthonormalize
        image_basis, _ = np.linalg.qr(image_basis)
    
    return image_basis, rank

# Verification
def verify_image(A, image_basis, tol=1e-7):
    """Check that image_basis is orthonormal."""
    gram = image_basis.T @ image_basis
    assert np.allclose(gram, np.eye(image_basis.shape[1]), atol=tol), "Not orthonormal"
    return True

Expected Output: For matrix \(A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{pmatrix}\): - rank = 2 - image_basis shape: (3, 2), columns form orthonormal basis of \(\mathbb{R}^2\) within \(\mathbb{R}^3\) - Orthogonality check: gram = image_basis.T @ image_basis = I_2

Numerical / Shape Notes: - SVD method most numerically stable; uses singular values to determine rank - QR method faster (O(mn²)); suitable for tall matrices - For 10 × 8 matrix with rank 6: image_basis is 10 × 6 - Verify: \(A A^\dagger A = A\) (where \(A^\dagger\) is pseudo-inverse)

C.3. Kernel and Image Decomposition in Neural Network Layers

Code:

import numpy as np

def decompose_layer_kernel_image(W, b=None, tol=1e-9):
    """
    For a linear layer y = Wx + b, decompose the weight matrix W into kernel and image.
    
    Args:
        W: weight matrix (output_dim, input_dim)
        b: bias (optional)
        tol: numerical tolerance
    
    Returns:
        U_image: orthonormal basis for image of W (output_dim, rank)
        V_kernel: orthonormal basis for kernel of W^T (input_dim, nullity)
        rank: rank of W
    """
    m, n = W.shape  # m = output_dim, n = input_dim
    
    # SVD: W = U * Sigma * V^T
    U, s, Vt = np.linalg.svd(W, full_matrices=True)
    
    rank = np.sum(s > tol)
    
    # U_image: left singular vectors for nonzero singular values
    U_image = U[:, :rank]
    
    # V_kernel: right singular vectors corresponding to zero singular values (from right side of V)
    V_kernel = Vt[rank:, :].T  # shape (n, n - rank)
    
    return U_image, V_kernel, rank

def verify_decomposition(W, U_image, V_kernel, tol=1e-7):
    """
    Verify:
    1. U_image columns are orthonormal
    2. V_kernel columns are orthonormal
    3. Columns are orthogonal: U_image^T @ any column of V_kernel ≈ 0
    4. Range(W) = span(U_image)
    5. Null(W^T) = span(V_kernel)
    """
    # Orthonormality checks
    assert np.allclose(U_image.T @ U_image, np.eye(U_image.shape[1]), atol=tol)
    assert np.allclose(V_kernel.T @ V_kernel, np.eye(V_kernel.shape[1]), atol=tol)
    
    # Orthogonality between image and kernel
    cross = U_image.T @ V_kernel
    assert np.allclose(cross, np.zeros_like(cross), atol=tol)
    
    return True

Expected Output: For \(W = \begin{pmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6 \end{pmatrix}\) (rank 1): - U_image shape: (3, 1) - one orthonormal direction in output space - V_kernel shape: (2, 1) - one direction in input space that maps to zero - rank = 1 - Verification: \(W \times V_{\text{kernel}} \approx \mathbf{0}_{3 \times 1}\) - Interpretation: A specific linear combination of inputs vanishes in the output

Numerical / Shape Notes: - For 64 × 128 weight matrix (output 64, input 128) with rank 50: U_image is 64 × 50, V_kernel is 128 × 78 - Rank-nullity: 50 + 78 = 128 ✓ - Relevance: ResNets use skip connections to bypass bottleneck layers; decomposition explains which input directions “survive” the bottleneck - Complexity: O(mn × rank) for SVD

C.4. Low-Rank Approximation and Reconstruction

Code:

import numpy as np

def low_rank_approximation(A, k):
    """
    Approximate matrix A by its best rank-k approximation using SVD.
    
    Args:
        A: input matrix (m, n)
        k: target rank
    
    Returns:
        A_k: rank-k approximation
        relative_error: ||A - A_k|| / ||A||
    """
    U, s, Vt = np.linalg.svd(A, full_matrices=False)
    
    # Truncate to rank k
    U_k = U[:, :k]
    s_k = s[:k]
    Vt_k = Vt[:k, :]
    
    A_k = U_k @ np.diag(s_k) @ Vt_k
    
    # Compute relative Frobenius norm error
    error = np.linalg.norm(A - A_k, 'fro')
    norm_A = np.linalg.norm(A, 'fro')
    relative_error = error / norm_A if norm_A > 0 else 0
    
    return A_k, relative_error

def rank_threshold(A, energy_threshold=0.95):
    """
    Find minimum rank to retain 'energy_threshold' fraction of spectral energy.
    
    Args:
        A: input matrix
        energy_threshold: target fraction of sum of squared singular values (e.g., 0.95 for 95%)
    
    Returns:
        k: minimum rank
        cumulative_energy: cumulative sum of squares of singular values
    """
    U, s, Vt = np.linalg.svd(A, full_matrices=False)
    
    total_energy = np.sum(s**2)
    cumsum_energy = np.cumsum(s**2)
    
    k = np.where(cumsum_energy / total_energy >= energy_threshold)[0][0] + 1
    
    return k, cumsum_energy / total_energy

Expected Output: For a 100 × 100 Hilbert matrix with decaying singular values: - Total singular values: 100 - To retain 95% energy: k ≈ 20–30 (depending on matrix condition number) - A_k at rank 20: relative_error ≈ 0.05 (or better) - Spectrum decay: singular values drop rapidly, showing good low-rank approximation potential

Numerical / Shape Notes: - Singular values decay as \(s_i \approx c / i^\alpha\) for specific matrix types - For 1000 × 1000 image: rank 50 typically retains > 95% information - Error threshold of 0.01 (1%) requires higher rank than 0.05 (5%) - Complexity: O(mn min(m,n)) for full SVD; O(mnk) for truncated SVD

C.5. Rank and Expressivity in Linear Models

Code:

import numpy as np

def rank_expressivity_analysis(X_train, y_train, X_test, y_test, max_rank=None):
    """
    Analyze how the rank constraint affects linear regression fit quality.
    
    Args:
        X_train, y_train: training data
        X_test, y_test: test data
        max_rank: maximum rank to test (default: min(X_train.shape))
    
    Returns:
        train_errors, test_errors, ranks: arrays of errors for each rank
    """
    m, n = X_train.shape
    if max_rank is None:
        max_rank = min(m, n)
    
    # Compute pseudo-inverse for least-squares: β = (X^T X)^{-1} X^T y
    U, s, Vt = np.linalg.svd(X_train, full_matrices=False)
    
    train_errors = []
    test_errors = []
    ranks_tested = []
    
    for k in range(1, min(max_rank + 1, len(s) + 1)):
        # Fit using rank-k pseudo-inverse
        U_k = U[:, :k]
        s_k = s[:k]
        Vt_k = Vt[:k, :]
        
        # Pseudo-inverse: X^+ = V_k Sigma_k^{-1} U_k^T
        Xplus_k = Vt_k.T @ np.diag(1 / s_k) @ U_k.T
        
        beta_k = Xplus_k @ y_train
        
        # Predictions and errors
        y_train_pred = X_train @ beta_k
        y_test_pred = X_test @ beta_k
        
        train_error = np.mean((y_train - y_train_pred)**2)
        test_error = np.mean((y_test - y_test_pred)**2)
        
        train_errors.append(train_error)
        test_errors.append(test_error)
        ranks_tested.append(k)
    
    return np.array(ranks_tested), np.array(train_errors), np.array(test_errors)

Expected Output: For synthetic data (n = 10 features, m = 100 samples, true rank ≈ 5): - rank 1: train_error ≈ 0.5, test_error ≈ 0.55 (underfitting) - rank 5: train_error ≈ 0.01, test_error ≈ 0.02 (good fit) - rank 10: train_error ≈ 0.001, test_error ≈ 0.05 (overfitting, gap widens) - Optimal rank: k = 5 (minimum test error)

Numerical / Shape Notes: - Generalization gap = test_error - train_error widens as rank → max - For underfitting (k too small): both train and test errors are high - For overfitting (k too large): train error low, test error high - Rank-capacity trade-off: lower rank = simpler model = better generalization (up to a limit)

C.6. Convolutional Layer Rank and Channel Decomposition

Code:

import numpy as np

def analyze_conv_weight_rank(W_conv, depthwise_separable=False):
    """
    Analyze rank of convolutional layer weights.
    
    For a conv layer with shape (out_channels, in_channels, kernel_h, kernel_w):
    Reshape to 2D: (out_channels, in_channels * kernel_h * kernel_w)
    
    Args:
        W_conv: conv weights (out_channels, in_channels, kernel_h, kernel_w)
        depthwise_separable: if True, analyze depthwise-separable decomposition
    
    Returns:
        rank: rank of the weight matrix (when reshaped to 2D)
        compression_ratio: (original params) / (rank-k approx params)
    """
    out_c, in_c, kh, kw = W_conv.shape
    
    # Reshape to 2D
    W_2d = W_conv.reshape(out_c, in_c * kh * kw)
    
    # Compute rank via SVD
    U, s, Vt = np.linalg.svd(W_2d, full_matrices=False)
    rank = np.sum(s > 1e-9)
    
    # Depthwise-separable decomposition approximation
    if depthwise_separable:
        # Approximate with rank-1: outer product of one singular vector pair
        if rank > 0:
            approx_rank_1 = np.outer(U[:, 0], Vt[0, :])
            error = np.linalg.norm(W_2d - approx_rank_1, 'fro') / np.linalg.norm(W_2d, 'fro')
            # Compression: full params vs. depthwise + pointwise
            full_params = out_c * in_c * kh * kw
            depthwise_separable_params = in_c * kh * kw + out_c * in_c  # depthwise + 1x1 conv
            compression_ratio = full_params / depthwise_separable_params
            return rank, compression_ratio, error
    
    return rank, None, None

def estimate_mobilenet_efficiency(layer_configs):
    """
    Estimate parameter reduction in MobileNet-style factorization.
    
    Args:
        layer_configs: list of (out_c, in_c, kh, kw) tuples
    
    Returns:
        total_compression: overall reduction factor
    """
    total_full = 0
    total_factored = 0
    
    for out_c, in_c, kh, kw in layer_configs:
        full = out_c * in_c * kh * kw
        factored = in_c * kh * kw + out_c * in_c  # depthwise + pointwise
        total_full += full
        total_factored += factored
    
    return total_full / total_factored

Expected Output: For a conv layer (64 out channels, 32 in channels, 3×3 kernel): - W_conv shape: (64, 32, 3, 3) - Reshaped W_2d shape: (64, 288) - Typical rank: 32–48 (rarely full rank 64) - Depthwise-separable compression ratio: ~2.5–3× (standard in MobileNets)

Numerical / Shape Notes: - Parameters: 64 × 32 × 3 × 3 = 18,432 - Depthwise (32 × 3 × 3) + pointwise (64 × 32): 288 + 2,048 = 2,336 (8× smaller) - Rank constraint suggests grouped convolutions can be highly effective - Practical: MobileNet achieves ~7× speedup with ~1% accuracy loss via depthwise-separable

C.7. Change of Basis and Matrix Conditioning

Code:

import numpy as np

def change_of_basis_effect_on_conditioning(A, P):
    """
    Analyze how a change of basis affects the condition number of a matrix.
    
    If T has matrix representation A in the original basis,
    its representation in the new basis (defined by invertible P) is:
    A' = P^{-1} A P
    
    Args:
        A: original matrix
        P: invertible matrix defining new basis
    
    Returns:
        A_new: A' = P^{-1} A P
        cond_orig: condition number of A
        cond_new: condition number of A'
        improvement: ratio cond_orig / cond_new
    """
    cond_orig = np.linalg.cond(A)
    
    P_inv = np.linalg.inv(P)
    A_new = P_inv @ A @ P
    
    cond_new = np.linalg.cond(A_new)
    
    improvement = cond_orig / cond_new if cond_new > 0 else np.inf
    
    return A_new, cond_orig, cond_new, improvement

def diagonalize_and_condition_check(A):
    """
    If A is diagonalizable, change basis to diagonal form and assess conditioning improvement.
    
    Returns:
        A_diag: diagonal matrix (eigenvalues)
        P: eigenvector matrix (change of basis)
        cond_P: condition number of P (affects numerical stability of basis change)
        cond_improvement: ratio of original to diagonal condition number
    """
    eigenvalues, P = np.linalg.eig(A)
    
    # Verify diagonalization and assess basis change stability
    A_diag_theory = np.diag(eigenvalues)
    
    # In new basis: A' = P^{-1} A P = diag(eigenvalues)
    P_inv = np.linalg.inv(P)
    A_diag_actual = P_inv @ A @ P
    
    cond_orig = np.linalg.cond(A)
    # Condition of diagonal matrix is max(|lambda|) / min(|lambda|)
    cond_diag = np.linalg.cond(A_diag_actual)
    
    cond_P = np.linalg.cond(P)  # Stability of basis change
    
    improvement = cond_orig / cond_diag if cond_diag > 0 else np.inf
    
    return A_diag_actual, P, cond_P, improvement, cond_orig, cond_diag

Expected Output: For a 3×3 symmetric matrix with eigenvalues [10, 5, 0.1]: - cond_orig ≈ 100 (ratio max/min) - After diagonalization: cond_new ≈ 100 (unchanged—symmetric matrices don’t improve via reordering) - cond_P: condition number of eigenvector matrix (well-conditioned for symmetric matrices, ~1–5) - Improvement: 1 (no change for symmetric matrices; diagonalization doesn’t reduce condition number)

Numerical / Shape Notes: - For non-symmetric A: diagonalization can reduce condition number or increase it (depends on eigenvector matrix condition) - Symmetric matrices: P is orthogonal (cond_P = 1), preserves conditioning - Ill-conditioned basis changes: cond_P >> 1 can amplify numerical errors despite theoretical benefits - PCA: orthogonal change of basis (unit condition number) followed by dimensionality reduction

C.8. Rank, Multicollinearity, and Regression Stability

Code:

import numpy as np

def detect_multicollinearity(X, method='vif'):
    """
    Detect multicollinearity in feature matrix X via VIF or condition number.
    
    Args:
        X: feature matrix (m, n)
        method: 'vif' (variance inflation factor) or 'condition_number'
    
    Returns:
        vif_values: VIF for each feature (method='vif')
        condition_number: condition number of X^T X (method='condition_number')
    """
    if method == 'vif':
        # VIF_j = 1 / (1 - R_j^2) where R_j^2 is R^2 from regressing X_j on other X's
        vif_values = np.zeros(X.shape[1])
        for j in range(X.shape[1]):
            X_others = np.hstack([X[:, :j], X[:, j+1:]])
            # Regress X[:, j] on X_others
            beta = np.linalg.lstsq(X_others, X[:, j], rcond=None)[0]
            y_pred = X_others @ beta
            ss_res = np.sum((X[:, j] - y_pred)**2)
            ss_tot = np.sum((X[:, j] - np.mean(X[:, j]))**2)
            r_squared = 1 - ss_res / ss_tot if ss_tot > 0 else 0
            vif_values[j] = 1 / (1 - r_squared) if r_squared < 1 else np.inf
        return vif_values
    
    elif method == 'condition_number':
        Xtilde = X.T @ X  # Gram matrix
        cond = np.linalg.cond(Xtilde)
        return cond
    
    return None

def ridge_regression_vs_multicollinearity(X_train, y_train, X_test, y_test, lambdas):
    """
    Compare OLS vs. Ridge regression as a function of regularization strength.
    
    Ridge: β = (X^T X + λI)^{-1} X^T y
    
    Returns:
        beta_ols, beta_ridge_list: coefficient vectors
        train_errors, test_errors: error for each λ
    """
    beta_ols = np.linalg.lstsq(X_train, y_train, rcond=None)[0]
    train_errors_ols = np.mean((y_train - X_train @ beta_ols)**2)
    test_errors_ols = np.mean((y_test - X_test @ beta_ols)**2)
    
    train_errors = [train_errors_ols]
    test_errors = [test_errors_ols]
    betas = [beta_ols]
    
    for lam in lambdas:
        Gram = X_train.T @ X_train
        beta_ridge = np.linalg.inv(Gram + lam * np.eye(X_train.shape[1])) @ X_train.T @ y_train
        
        train_error = np.mean((y_train - X_train @ beta_ridge)**2)
        test_error = np.mean((y_test - X_test @ beta_ridge)**2)
        
        train_errors.append(train_error)
        test_errors.append(test_error)
        betas.append(beta_ridge)
    
    return betas, train_errors, test_errors

Expected Output: For a 100-sample dataset with 10 features (3 are nearly collinear): - VIF values: [1.2, 1.3, 50, 45, 52, 1.1, 1.0, 1.5, 1.2, 1.4] (three features > 40, indicating multicollinearity) - Condition number of Gram matrix: ~500 (high, indicates ill-conditioning) - OLS coefficients: large magnitudes, high variance (unstable) - Ridge regression (λ=0.1): coefficients shrink, test error improves by ~5–15%

Numerical / Shape Notes: - VIF > 10: typically indicates problematic multicollinearity - Condition number > 100: ill-conditioned system - Gram matrix cond = (cond of X)² (important relationship) - Ridge λ ≈ 0.01–1.0 typically optimal (tune via cross-validation) - Rank: multicollinearity reduces effective rank below n

C.9. Kernel Methods and Eigendecomposition

Code:

import numpy as np

def rbf_kernel_gram_matrix(X, gamma=1.0):
    """
    Compute RBF (Gaussian) kernel Gram matrix.
    
    K[i,j] = exp(-gamma * ||X_i - X_j||^2)
    
    Args:
        X: data (m, n)
        gamma: RBF width parameter (higher = sharper)
    
    Returns:
        K: kernel Gram matrix (m, m)
    """
    # Efficient computation: ||x_i - x_j||^2 = ||x_i||^2 + ||x_j||^2 - 2 x_i^T x_j
    sq_norms = np.sum(X**2, axis=1, keepdims=True)
    K = sq_norms + sq_norms.T - 2 * (X @ X.T)
    K = np.exp(-gamma * K)
    return K

def eigendecompose_kernel(K):
    """
    Eigendecompose the kernel matrix and analyze effective rank.
    
    Returns:
        eigenvalues: sorted in descending order
        eigenvectors: corresponding eigenvectors
        effective_rank: number of eigenvalues > 1e-9
    """
    eigenvalues, eigenvectors = np.linalg.eigh(K)
    
    # Sort in descending order
    idx = np.argsort(eigenvalues)[::-1]
    eigenvalues = eigenvalues[idx]
    eigenvectors = eigenvectors[:, idx]
    
    effective_rank = np.sum(eigenvalues > 1e-9)
    
    return eigenvalues, eigenvectors, effective_rank

def kernel_svm_approximation(X_train, y_train, X_test, gamma=1.0, C=1.0):
    """
    Kernel SVM via eigendecomposition (Nyström or kernel PCA approach).
    
    For illustration, use kernel matrix eigendecomposition to approximate SVM solution.
    
    Returns:
        predictions: predictions on test set
        support_info: information about kernel structure
    """
    # Compute Gram matrix on training set
    K_train = rbf_kernel_gram_matrix(X_train, gamma)
    
    # Eigendecompose
    eigenvalues, eigenvectors, eff_rank = eigendecompose_kernel(K_train)
    
    # For test set: compute K_test based on training data
    # K_test[i,j] = K(X_test[i], X_train[j])
    sq_norms_train = np.sum(X_train**2, axis=1, keepdims=True)
    sq_norms_test = np.sum(X_test**2, axis=1, keepdims=True)
    dists_sq = sq_norms_test + sq_norms_train.T - 2 * (X_test @ X_train.T)
    K_test = np.exp(-gamma * dists_sq)
    
    # Use kernel eigendecomposition for classification (simplified)
    # Project test onto kernel principal components
    kernel_features_train = eigenvectors @ np.diag(np.sqrt(eigenvalues))
    kernel_features_test = K_test @ eigenvectors @ np.diag(1 / np.sqrt(eigenvalues + 1e-9))
    
    # Train linear classifier on kernel features
    beta = np.linalg.lstsq(kernel_features_train, y_train, rcond=None)[0]
    predictions = kernel_features_test @ beta
    
    return predictions, eff_rank

def analyze_kernel_bandwidth(X, gammas):
    """
    Analyze RBF kernel behavior for different gamma values.
    
    Returns:
        condition_numbers: cond(K) for each gamma
        ranks: effective rank of K for each gamma
    """
    condition_numbers = []
    ranks = []
    
    for gamma in gammas:
        K = rbf_kernel_gram_matrix(X, gamma)
        cond = np.linalg.cond(K)
        eigenvalues, _, eff_rank = eigendecompose_kernel(K)
        
        condition_numbers.append(cond)
        ranks.append(eff_rank)
    
    return condition_numbers, ranks

Expected Output: For 50-sample dataset with RBF kernel (gamma=0.1): - Kernel matrix K: (50, 50), all entries in (0, 1) - Eigenvalues: decay rapidly (first few >> rest) - Effective rank: ~15–25 of 50 (indicates data lies on lower-dimensional manifold) - Condition number: 100–1,000 (depends on gamma and data concentration) - For gamma = 0.01 (wider RBF): lower rank, smoother kernel - For gamma = 10.0 (narrow RBF): higher rank, sharper kernel

Numerical / Shape Notes: - RBF kernel matrices are typically low-rank (manifold hypothesis) - Gamma too small: K ≈ all 1s (rank 1, underfitting) - Gamma too large: K ≈ identity (rank = n, overfitting to noise) - Optimal gamma: cross-validation on rank/generalization trade-off - Complexity of eigendecomposition: O(m³) for m-sample kernel matrix

C.10. Bottleneck Detection in Deep Linear Networks

Code:

import numpy as np

def detect_bottleneck_layers(weight_matrices):
    """
    Analyze ranks of weight matrices in a deep linear network.
    
    Bottleneck layer: a layer where rank drops significantly.
    
    Args:
        weight_matrices: list of weight matrices for each layer
    
    Returns:
        ranks: rank of each layer
        rank_drops: absolute drop in rank from previous layer
        bottleneck_threshold: suggested threshold for bottleneck detection
    """
    ranks = []
    for W in weight_matrices:
        U, s, Vt = np.linalg.svd(W, full_matrices=False)
        rank = np.sum(s > 1e-9)
        ranks.append(rank)
    
    rank_drops = np.diff([weight_matrices[0].shape[0]] + ranks)  # Include input dimension
    
    avg_drop = np.mean(np.abs(rank_drops[1:]))  # Exclude first input dimension
    bottleneck_threshold = avg_drop + 1 * np.std(np.abs(rank_drops[1:]))
    
    bottlenecks = [i for i, drop in enumerate(rank_drops) if drop > bottleneck_threshold]
    
    return ranks, rank_drops, bottlenecks

def analyze_information_flow(weight_matrices):
    """
    Trace how information (dimensionality) flows through the network.
    
    Returns:
        effective_dimensions: effective rank at each layer
        information_retention: fraction of information retained per layer
    """
    effective_dims = []
    input_dim = weight_matrices[0].shape[1]
    effective_dims.append(input_dim)
    
    cumulative_rank = input_dim
    for W in weight_matrices:
        U, s, Vt = np.linalg.svd(W, full_matrices=False)
        rank = np.sum(s > 1e-9)
        output_dim = W.shape[0]
        effective_rank = min(cumulative_rank, rank, output_dim)
        effective_dims.append(effective_rank)
        cumulative_rank = effective_rank
    
    information_retention = np.array(effective_dims[1:]) / np.array(effective_dims[:-1])
    
    return effective_dims, information_retention

def design_network_For_expressivity(input_dim, output_dim, num_layers, target_min_rank_per_layer=None):
    """
    Design network layer dimensions to avoid unintended bottlenecks.
    
    Simple strategy: make each layer at least as wide as min(input, ranks needed for expressivity).
    
    Returns:
        layer_configs: recommended (output_dim, input_dim) for each layer
    """
    if target_min_rank_per_layer is None:
        target_min_rank_per_layer = max(input_dim, output_dim) // 2
    
    layer_configs = []
    current_dim = input_dim
    
    for i in range(num_layers - 1):
        next_dim = max(target_min_rank_per_layer, current_dim)
        layer_configs.append((next_dim, current_dim))
        current_dim = next_dim
    
    # Final layer: output dimension
    layer_configs.append((output_dim, current_dim))
    
    return layer_configs

Expected Output: For a 5-layer network: input 784 → 512 → 256 → 64 → 10 (output): - Ranks: [512, 256, 64, 10] (assuming full rank at each layer assuming enough data) - Rank drops: [272, 256, 192, 54] (largest drops at middle layers) - Bottleneck detected: layer 3 (512 → 256 → 64 represents aggressive compression) - Information retention: [1.0, 0.5, 0.25, 0.156] (exponential decay)

Numerical / Shape Notes: - Deep networks without bottlenecks: linearly decreasing dimensions (gradual compression) - With bottleneck: sharp drop → risk of information loss - Residual connections mitigate: allow high-rank paths around bottlenecks - Rank must eventually drop to output_dim (unavoidable compression for classification) - Complexity: O(min(m,n)³) per layer for SVD

C.11. Autoencoder Bottleneck and Reconstruction Error

Code:

import numpy as np

def linear_autoencoder_reconstruction(X, bottleneck_dim):
    """
    Autoencoders with linear encoder/decoder: optimal solution via PCA.
    
    Encoder: z = W_e @ x (bottleneck_dim × input_dim)
    Decoder: x_recon = W_d @ z (input_dim × bottleneck_dim)
    
    Optimal solution: W_e = top-k principal components, W_d = their transposes.
    
    Args:
        X: data matrix (m, n)
        bottleneck_dim: latent dimension k
    
    Returns:
        X_recon: reconstructed data
        reconstruction_error: ||X - X_recon||_F
        explained_variance_ratio: variance captured by bottleneck
    """
    m, n = X.shape
    X_centered = X - np.mean(X, axis=0, keepdims=True)
    
    # SVD for PCA
    U, s, Vt = np.linalg.svd(X_centered, full_matrices=False)
    
    # Top-k components
    V_k = Vt[:bottleneck_dim, :].T  # Top components
    
    # Encoder and decoder
    W_e = V_k.T  # (bottleneck_dim, n)
    W_d = V_k    # (n, bottleneck_dim)
    
    # Latent representation
    Z = X_centered @ W_e.T  # (m, bottleneck_dim)
    
    # Reconstruction
    X_recon = Z @ W_d.T
    
    # Error
    error = np.linalg.norm(X_centered - X_recon, 'fro')
    total_variance = np.sum(s**2)
    explained_variance = np.sum(s[:bottleneck_dim]**2)
    explained_ratio = explained_variance / total_variance
    
    return X_recon, error, explained_ratio, V_k

def compare_bottleneck_choices(X, bottleneck_dims):
    """
    Compare reconstruction error for different bottleneck dimensions.
    
    Returns:
        errors: reconstruction error for each bottleneck dimension
        explained_variances: explained variance ratio for each dimension
    """
    errors = []
    explained_variances = []
    
    for k in bottleneck_dims:
        X_recon, error, exp_var, _ = linear_autoencoder_reconstruction(X, k)
        errors.append(error)
        explained_variances.append(exp_var)
    
    return errors, explained_variances

def nonlinear_autoencoder_intuition(X_train, X_test, bottleneck_dim, hidden_units=128):
    """
    Intuitive description of nonlinear autoencoder training.
    (No actual neural network training code; for illustration of expected behavior.)
    
    Key insight: nonlinear autoencoders learn hierarchical features.
    Bottleneck forces compression, revealing essential structure in data.
    
    Expected behavior:
    - Early training: reconstruction error decreases rapidly
    - Mid training: bottleneck learns discriminative features
    - Late training: overfitting risk (train error → 0, test error plateaus)
    
    Regularization strategies:
    - Dropout: randomly zero units, preventing co-adaptation
    - Denoising: add noise to input, force bottleneck to filter it out
    - L2 regularization: penalize weight magnitudes
    - Batch normalization: stabilize learning
    """
    pass  # Illustrative; actual training deferred

Expected Output: For MNIST (28×28=784 features) with different bottleneck dimensions: - bottleneck_dim=10: explained_variance=0.45, reconstruction_error=large - bottleneck_dim=50: explained_variance=0.89, reconstruction_error=moderate - bottleneck_dim=100: explained_variance=0.97, reconstruction_error=small - Optimal: k≈50 balances compression and fidelity

Numerical / Shape Notes: - Linear autoencoder: equivalent to PCA truncation at dimension k - Reconstruction error ∝ sum of discarded singular values - For images: k≈10–20% of dimensionality typical (e.g., 50–200 for MNIST) - Nonlinear autoencoders: can achieve better compression (feature learning benefit) - VAE connection: Bayesian interpretation of bottleneck as latent distribution

C.12. Principal Component Analysis (PCA) and Variance Explanation

Code:

import numpy as np

def pca_variance_analysis(X, cumulative_variance_threshold=0.95):
    """
    Perform PCA and find minimum number of components for target variance.
    
    Args:
        X: data matrix (m, n)
        cumulative_variance_threshold: target cumulative variance (e.g., 0.95 for 95%)
    
    Returns:
        n_components: minimum components to reach threshold
        explained_variance_ratio: variance explained by each component
        cumulative_ratios: cumulative variance explained
        principal_components: top components (eigenvectors of covariance)
    """
    X_centered = X - np.mean(X, axis=0, keepdims=True)
    
    # Covariance matrix or SVD
    U, s, Vt = np.linalg.svd(X_centered, full_matrices=False)
    
    # Explained variance
    explained_variance = (s**2) / np.sum(s**2)
    cumsum_variance = np.cumsum(explained_variance)
    
    # Number of components for threshold
    n_components = np.argmax(cumsum_variance >= cumulative_variance_threshold) + 1
    
    # Principal components
    principal_components = Vt[:n_components, :]  # Top components
    
    return n_components, explained_variance, cumsum_variance, principal_components

def pca_dimensionality_reduction(X, n_components):
    """
    Project data onto top n principal components.
    
    Args:
        X: input data (m, n)
        n_components: number of components to retain
    
    Returns:
        X_reduced: projected data (m, n_components)
        reconstruction: reconstruction of X (m, n)
        reconstruction_error: ||X - reconstruction||
    """
    X_centered = X - np.mean(X, axis=0, keepdims=True)
    
    U, s, Vt = np.linalg.svd(X_centered, full_matrices=False)
    
    # Project onto top components
    X_reduced = U[:, :n_components] @ np.diag(s[:n_components])  # or X_centered @ Vt[:n_components, :].T
    
    # Reconstruct
    V_k = Vt[:n_components, :].T
    reconstruction = (X_reduced @ Vt[:n_components, :]) + np.mean(X, axis=0, keepdims=True)
    
    error = np.linalg.norm(X - reconstruction, 'fro')
    
    return X_reduced, reconstruction, error

def scree_plot_interpretation(X):
    """
    Compute scree plot data: variance explained vs. component number.
    Helps identify elbow where additional components add little value.
    
    Returns:
        component_indices: component numbers (1, 2, 3, ...)
        cumulative_variance: cumulative variance explained
    """
    X_centered = X - np.mean(X, axis=0, keepdims=True)
    U, s, Vt = np.linalg.svd(X_centered, full_matrices=False)
    
    cumsum_var = np.cumsum(s**2 / np.sum(s**2))
    component_indices = np.arange(1, len(cumsum_var) + 1)
    
    return component_indices, cumsum_var

Expected Output: For MNIST dataset (60,000 samples, 784 features): - Explained variance (top 10 components): [0.096, 0.074, 0.059, 0.048, 0.040, …] - Cumulative variance at n=50: ~0.95 (95% of information in 50 components, 6.4% of original) - Scree plot elbow: around 50–100 components - Dimensionality reduction: 784 → 50 (6.4% of size, ~95% of variance) - Reconstruction with 50 components: visually acceptable but blurry

Numerical / Shape Notes: - First component typically explains 10–15% of variance - Rapid decay: top 50 components often explain 90%+ variance - For 784 features: typical PCA reduction to 50–100 components for image data - Elbow: point where marginal gain per component drops sharply - Cross-validation: select n_components to optimize downstream task (e.g., classification)

C.13. SVD-Based Low-Rank Image Compression

Code:

import numpy as np

def svd_image_compression(image, rank):
    """
    Compress an image using SVD truncation.
    
    Args:
        image: 2D numpy array (grayscale) or (height, width, 3) for RGB
        rank: target rank for approximation
    
    Returns:
        image_compressed: rank-k approx
        compression_ratio: original_size / compressed_size
        mse: mean squared error ||image - image_compressed||^2 / num_pixels
    """
    if image.ndim == 3:
        # For RGB, process each channel separately
        channels = [svd_image_compression(image[:,:,c], rank) for c in range(3)]
        image_compressed = np.stack([ch[0] for ch in channels], axis=2)
        compression_ratio = channels[0][1]
        mse = np.mean([ch[2] for ch in channels])
        return image_compressed, compression_ratio, mse
    
    U, s, Vt = np.linalg.svd(image, full_matrices=False)
    
    # Truncate
    U_k = U[:, :rank]
    s_k = s[:rank]
    Vt_k = Vt[:rank, :]
    
    image_compressed = U_k @ np.diag(s_k) @ Vt_k
    
    # Compression ratio: original entries vs. compressed (U + s + Vt)
    h, w = image.shape
    original_entries = h * w
    compressed_entries = rank * (h + w + 1)  # rank × (h + w + 1) instead of h × w
    compression_ratio = original_entries / compressed_entries
    
    # MSE
    mse = np.mean((image - image_compressed)**2)
    psnr = 10 * np.log10(255**2 / mse) if mse > 0 else np.inf  # PSNR for 8-bit images
    
    return image_compressed, compression_ratio, mse

def compression_vs_quality_tradeoff(image, ranks):
    """
    Analyze compression ratio and quality for multiple rank choices.
    
    Returns:
        compression_ratios: compression ratio for each rank
        errors: MSE for each rank
        psnrs: PSNR in dB for each rank
    """
    compression_ratios = []
    errors = []
    psnrs = []
    
    for k in ranks:
        img_comp, comp_ratio, mse = svd_image_compression(image, k)
        compression_ratios.append(comp_ratio)
        errors.append(mse)
        psnr = 10 * np.log10(255**2 / mse) if mse > 0 else np.inf
        psnrs.append(psnr)
    
    return compression_ratios, errors, psnrs

def compare_jpeg_like_compression(image, rank_list):
    """
    Compare SVD compression with typical JPEG compression metrics.
    
    JPEG typically achieves 10:1–50:1 compression with good perceptual quality.
    
    Returns:
        svd_metrics: compression ratios and PSNR for SVD
        interpretation: how SVD compares to standard codecs
    """
    # SVD compression analysis
    comp_ratios, errors, psnrs = compression_vs_quality_tradeoff(image, rank_list)
    
    # Typical JPEG: quality 90 ≈ 10:1, quality 80 ≈ 20:1, quality 70 ≈ 30:1
    # PSNR: quality 90 ≈ 35–40 dB, quality 80 ≈ 30–35 dB
    
    optimal_psnr = 35  # Target ~35 dB (visually acceptable)
    idx_optimal = np.argmin(np.abs(np.array(psnrs) - optimal_psnr))
    
    return {
        'ranks': rank_list,
        'compression_ratios': comp_ratios,
        'psnrs': psnrs,
        'optimal_rank': rank_list[idx_optimal],
        'optimal_psnr': psnrs[idx_optimal]
    }

Expected Output: For a 256×256 grayscale image: - rank 10: compression_ratio ≈ 4.8, MSE ≈ 500, PSNR ≈ 21 dB (very blurry) - rank 30: compression_ratio ≈ 1.9, MSE ≈ 100, PSNR ≈ 28 dB (tolerable) - rank 50: compression_ratio ≈ 1.4, MSE ≈ 30, PSNR ≈ 33 dB (good quality) - rank 100: compression_ratio ≈ 0.8 (no actual compression, nearly original) - Tradeoff: higher rank sacrifices compression for fidelity

Numerical / Shape Notes: - For h×w image: SVD storage = rank × (h + w + 1) vs. h × w original - Compression ratio > 1: actual compression; < 1: storage overhead - PSNR > 30 dB: visually acceptable; < 20 dB: visible artifacts - Natural images: very low rank (rank 20–50 for 256×256), enabling high compression - Complexity: O(h² w) or O(h w²) depending on aspect ratio

C.14. Operator Norm and Spectral Normalization

Code:

import numpy as np

def compute_operator_norm(A, p='fro'):
    """
    Compute operator norm (matrix norm) of A.
    
    Common norms:
    - p='fro': Frobenius ||A||_F = sqrt(sum of A_ij^2)
    - p=2: Spectral ||A||_2 = largest singular value
    - p=1: Induced 1-norm (max absolute column sum)
    - p=np.inf: Induced infinity-norm (max absolute row sum)
    
    Args:
        A: matrix
        p: norm type
    
    Returns:
        norm: computed norm value
    """
    if p == 2:
        # Spectral norm = largest singular value
        return np.linalg.norm(A, ord=2)
    elif p == 'fro':
        return np.linalg.norm(A, ord='fro')
    elif p == 1 or p == np.inf:
        return np.linalg.norm(A, ord=p)
    return None

def spectral_normalization_layer(W, num_iterations=1, tol=1e-5):
    """
    Apply spectral normalization to weight matrix W.
    
    Normalizes W so that ||W||_2 = 1 (spectral norm is 1).
    
    Used in GANs to stabilize training by enforcing Lipschitz constraint.
    
    Args:
        W: weight matrix (output_dim, input_dim)
        num_iterations: power iteration steps (typically 1 per forward pass)
        tol: convergence tolerance
    
    Returns:
        W_normalized: weight with spectral norm = 1
        sigma: estimated largest singular value
    """
    # Initialize random vector
    u = np.random.randn(W.shape[0])
    u = u / np.linalg.norm(u)
    
    # Power iteration
    for _ in range(num_iterations):
        v = W.T @ u
        v = v / (np.linalg.norm(v) + tol)
        u = W @ v
        u = u / (np.linalg.norm(u) + tol)
    
    # Estimate spectral norm
    sigma = u @ W @ v
    
    # Normalize
    W_normalized = W / sigma
    
    return W_normalized, sigma

def gan_discriminator_lipschitz_constraint(weight_matrices, lipschitz_constant=1.0):
    """
    Enforce Lipschitz constraint on discriminator network via spectral norm.
    
    Discriminator: product of spectral norms of each layer must be ≤ lipschitz_constant.
    
    Args:
        weight_matrices: list of layer weight matrices
        lipschitz_constant: target upper bound
    
    Returns:
        normalized_weights: spectral normalized weights
        product_of_norms: product of spectral norms (should be ≈ lipschitz_constant)
    """
    normalized_weights = []
    product = 1.0
    
    for W in weight_matrices:
        W_norm, sigma = spectral_normalization_layer(W, num_iterations=1)
        normalized_weights.append(W_norm)
        product *= sigma
    
    # Scale final layer to enforce constraint
    scale = lipschitz_constant / product if product > 0 else lipschitz_constant
    normalized_weights[-1] = normalized_weights[-1] * scale
    
    return normalized_weights, lipschitz_constant

def condition_number_and_stability(A):
    """
    Analyze conditioning and its impact on numerical stability.
    
    Condition number: cond(A) = ||A|| × ||A^{-1}||
    For solving Ax = b:
    - cond(A) small: well-conditioned, solution stable
    - cond(A) large: ill-conditioned, solution sensitive to perturbations
    
    Returns:
        condition_number: cond(A)
        stability_analysis: interpretation of numerical stability
    """
    cond = np.linalg.cond(A)
    
    if cond < 100:
        stability = "Well-conditioned; expect reliable solutions"
    elif cond < 1e6:
        stability = "Moderate conditioning; acceptable but use care with floating-point"
    else:
        stability = "Ill-conditioned; expect large errors in solution; consider regularization"
    
    return cond, stability