Part 1: Core setup - Predictions and attention are in spans

State the objects, shapes, and target question for Predictions and attention are in spans. Name the data matrices or vectors, specify their dimensions, and clarify the transformation or comparison this example develops.

Part 2: Geometry and algebraic insight - Predictions and attention are in spans

Describe the geometric picture (subspaces, projections, bases, or decompositions) and the algebraic identities that make Predictions and attention are in spans work. Highlight how these structures constrain solutions and connect to earlier linear algebra tools.

Part 3: Numerics and ML practice - Predictions and attention are in spans

Give the computational recipe for Predictions and attention are in spans, note stability or conditioning checks, and tie to an ML use case. Mention parameter choices, common pitfalls, and quick sanity checks such as shape validation or reconstruction error.

• Shape discipline: check dimensions before manipulating formulas.
• Numerical note: prefer stable primitives (lstsq, QR/SVD, Cholesky for SPD) over explicit inverses.
• Interpretation: relate algebraic steps to geometry (subspaces, projections) and to ML behavior (generalization, stability).
• Linear Regression as an Unconstrained Linear Combination The first section computes predictions for a simple linear model. Given a design matrix $X \in \mathbb{R}^{3 \times 2}$ with 3 examples and 2 features, and weights $w = [2, -1]^\top$, the matrix-vector product yhat = X @ w produces predictions $\hat{y} = [0., 2., 4.]^\top$ by computing the inner product of each row with $w$. The key insight emerges from the second print statement: 2*X[:,0] - 1*X[:,1] produces the identical result by explicitly forming a linear combination of $X$’s columns. This reveals that $Xw = w_1 \mathbf{x}_1 + w_2 \mathbf{x}_2$, where $\mathbf{x}_1$ and $\mathbf{x}_2$ are the column vectors of $X$. The coefficients $w$ can be any real numbers (positive, negative, or unbounded in magnitude)—there are no constraints. The output lives in the column span of $X$: any vector that is a linear combination of the columns. This unconstrained linear combination is the essence of supervised learning: find weights that best fit the data, constrained only by the loss function and regularization.
• Attention as a Constrained Convex Combination The second section implements scaled dot-product attention, which appears throughout transformers and modern deep learning. A query $Q \in \mathbb{R}^{1 \times 2}$ matches against keys $K \in \mathbb{R}^{3 \times 2}$ to produce raw similarity scores via scores = (Q @ K.T) / np.sqrt(2). The $1/\sqrt{d_k}$ scaling (with $d_k = 2$) prevents saturation by controlling the magnitude of dot products. These raw scores are passed through softmax(scores, axis=1), which converts them to normalized attention weights $a \in \mathbb{R}^3$ that sum to 1.0 (verified by printing a.sum()). This is the crucial constraint: softmax ensures non-negative coefficients that form a probability distribution, making the output a convex combination rather than an arbitrary linear combination. The final step o = a @ V computes the weighted sum of value vectors $V \in \mathbb{R}^{3 \times 2}$, producing output $o \in \mathbb{R}^2$. The verification a[0]*V[0] + a[1]*V[1] + a[2]*V[2] explicitly shows this is a convex combination—$o = \sum_i a_i V_i$ where $a_i \geq 0$ and $\sum_i a_i = 1$. The output lives in the convex hull of the value vectors, a geometric shape bounded by interpolating between them.
• The Deep Connection: Both Are Linear Combinations The profound insight is that regression and attention are two points on a spectrum of linear combinations. Regression selects arbitrary coefficients $w$ to minimize prediction error; attention selects non-negative coefficients (via softmax) that sum to 1. Regression projects onto the column span of $X$; attention projects onto the convex hull of value vectors. Despite this difference, both operations answer the same question: “Which point in my span/convex hull best represents the data?” The span is determined by the data (features in regression, value embeddings in attention), while the coefficients emerge from the learning signal (loss gradient in regression, query-key similarity in attention). This unifying view appears throughout ML: ensemble methods average models via convex combinations, mixture models interpolate distributions via softmax weights, and neural networks alternate between linear combinations (matrix products) and nonlinear expansions (activations) to build hierarchical representations.
• Part 4: Numerical and Shape Notes Shape discipline is essential here. For regression: $X \in \mathbb{R}^{3 \times 2}$, $w \in \mathbb{R}^2$ produce $\hat{y} \in \mathbb{R}^3$. For attention: $Q \in \mathbb{R}^{1 \times 2}$, $K \in \mathbb{R}^{3 \times 2}$ produce scores $\in \mathbb{R}^{1 \times 3}$, then softmax across axis=1 normalizes each row independently to sum to 1.0. The .reshape(-1) flattens the $1 \times 3$ weight matrix to a $3$-dimensional vector for clarity. Printing intermediate results (yhat, a, o) and verifying constraints (a.sum(), explicit combo) are best practices for debugging: they confirm that shapes align and operations behave as expected. The softmax import from scripts/toy_data provides numerically stable softmax (using the log-sum-exp trick to prevent overflow), which is essential when $d_k$ is large and raw scores can exceed floating-point range.