Span explains expressivity: linear models produce outputs in the span of features; attention mixes value vectors to form outputs in their span. Understanding spans clarifies when adding features or values increases model capacity, and how constraints (e.g., nonnegativity, normalization) yield convex combinations with bounded, interpretable outputs. In practice, spanâbased reasoning underlies basis selection, dictionary learning, and projections used throughout ML.
Develop a clear, ML-first intuition for span: linear predictors $Xw$ live in the span of the feature columns, and attention outputs are weighted sums of value vectors. This lets you reason about what your model can express, how projections and constraints affect outputs, and why shapes and weights matter for stability and interpretation.
Show that (a) linear predictions lie in the span of feature columns and (b) attention outputs lie in the span of value vectors. Compute both explicitly and verify the span claim by expressing outputs as linear combinations.
For linear predictions $\hat y = Xw$ with $X \in \mathbb{R}^{n\times d}$ and $w \in \mathbb{R}^d$,
\[ Xw = \sum_{j=1}^d w_j\, X_{:j}, \]
so $\hat y \in \mathrm{span}\{X_{:1},\dots,X_{:d}\}$.
For attention with one query, $a = \mathrm{softmax}(QK^\top/\sqrt{d_k}) \in \mathbb{R}^{n_k}$, the output is a weighted sum of values
\[ o = a^\top V = \sum_{i=1}^{n_k} a_i\, v_i, \quad o \in \mathrm{span}\{v_1,\dots,v_{n_k}\}. \]
We use:
import numpy as np
from scripts.toy_data import softmax
X = np.array([[1., 2.],
[3., 4.],
[5., 6.]])
w = np.array([2., -1.])
yhat = X @ w
print("yhat:", yhat)
print("check combo:", 2*X[:,0] - 1*X[:,1])
Q = np.array([[1., 0.]])
K = np.array([[1., 0.],
[0., 1.],
[1., 1.]])
V = np.array([[1., 0.],
[0., 2.],
[1., 1.]])
scores = (Q @ K.T) / np.sqrt(2)
a = softmax(scores, axis=1).reshape(-1)
o = a @ V
print("a:", a, "sum:", a.sum())
print("o:", o)
print("o combo:", a[0]*V[0] + a[1]*V[1] + a[2]*V[2])This snippet illustrates two instances of outputs as linear combinations. First, with $X \in \mathbb{R}^{3\times 2}$ and $w \in \mathbb{R}^2$, we compute $y_{\text{hat}} = Xw$. Each entry is $y_i = \mathbf{x}_i^\top w$, so $y_{\text{hat}}$ lies in the span of the columns of $X$. With $w = [2, -1]^\top$, this is exactly $y_{\text{hat}} = 2\,X_{:,0} - 1\,X_{:,1}$, verified by the âcheck comboâ line.
Second, the attention part forms scores $S = QK^\top / \sqrt{2}$ (with $d_k=2$), applies row-wise softmax to get weights $a \in \mathbb{R}^3$ that are nonnegative and sum to 1, then mixes values via $o = a^\top V$. This makes $o$ a convex combination of the rows of $V$: $o = \sum_{i=1}^3 a_i V_i$. The prints confirm the probability constraint and that the explicit weighted sum matches $o$.
axis in reductions and normalizations. For attention-like ops, softmax over keys (row-wise) so rows sum to â1.(n,1) with (n,d)). Prefer reshape/expand-dims to make semantics clear.||Ax-b||, reconstruction error, row-sum â 1). Assert finiteness and expected monotonicity where applicable.solve, lstsq, Cholesky/QR/SVD.axis) and ensure broadcasts are intended.np.linalg.cond(A) when solutions look unstable; regularize (ridge) or rescale features to improve conditioning.||Ax-b||, reconstruction errors) and assert finiteness.lstsq vs. solve), check orthogonality (U.T @ U â I), PSD (x.T @ A @ x > 0), and residual norms within tolerance (~1e-12 for float64).softmax(scores, axis=1) to obtain $a \in \mathbb{R}^{n_k}$.Span-based reasoning dates to early linear algebra and approximation theory: representing signals as linear combinations of basis elements underlies Fourier analysis, polynomial approximation, and finite element methods. In statistics and ML, outputs in spans explain linear model expressivity, least-squares projections, and PCA reconstructions. In modern architectures, attention implements convex combinations to form interpretable, stable mixtures of value vectors, with variants (sparse/linear attention) optimizing cost while preserving the span structure.
Comments