Part 1: Core setup - One-hot basis vectors and coordinate representations

State the objects, shapes, and target question for One-hot basis vectors and coordinate representations. Name the data matrices or vectors, specify their dimensions, and clarify the transformation or comparison this example develops.

Part 2: Geometry and algebraic insight - One-hot basis vectors and coordinate representations

Describe the geometric picture (subspaces, projections, bases, or decompositions) and the algebraic identities that make One-hot basis vectors and coordinate representations work. Highlight how these structures constrain solutions and connect to earlier linear algebra tools.

Part 3: Numerics and ML practice - One-hot basis vectors and coordinate representations

Give the computational recipe for One-hot basis vectors and coordinate representations, note stability or conditioning checks, and tie to an ML use case. Mention parameter choices, common pitfalls, and quick sanity checks such as shape validation or reconstruction error.

• Shape discipline: check dimensions before manipulating formulas.
• Numerical note: prefer stable primitives (lstsq, QR/SVD, Cholesky for SPD) over explicit inverses.
• Interpretation: relate algebraic steps to geometry (subspaces, projections) and to ML behavior (generalization, stability).
• One-Hot Embedding Lookup as Basis Selection The first section shows the simplest possible basis operation: selecting a column from a matrix. With $E \in \mathbb{R}^{2\times 3}$ containing 2 basis vectors (rows) spanning a 2D subspace of the 3-category space, multiplying by a one-hot vector [0., 1., 0.] extracts the second column $E[:,1] = [2., -1.]$. This is basis indexing in disguise: the one-hot vector specifies coordinates $(0, 1, 0)$ in the standard basis of $\mathbb{R}^3$, and $E \cdot \text{one-hot}$ maps those coordinates into the 2D embedding space spanned by $E$’s columns. The beauty of one-hot encoding is computational: while matrix-vector multiplication is generally $O(d_{\text{emb}} \times |V|)$, the one-hot structure makes this a simple table lookup—just fetch column 1. This pattern appears throughout ML: embedding layers in neural networks use one-hot vectors to index into learned weight matrices, where each column is a learned representation for a discrete token (word, category, user ID). The embedding matrix $E$ is the learned basis for the categorical space.
• PCA as Optimal Low-Rank Basis Discovery The second section transitions from hand-crafted bases to data-driven basis learning. Loading 20 points from toy_pca_points(n=20, seed=1) produces $X \in \mathbb{R}^{20\times 2}$ with inherent directional structure. Centering via mu = X.mean(0) and Xc = X - mu shifts the origin to the data’s center of mass, removing translational bias so SVD discovers variance directions relative to the mean. The SVD U, S, Vt = svd(Xc) factorizes centered data as $X_c = U \Sigma V^\top$, where Vt (which is $V^\top$) contains the principal directions. Extracting V = Vt.T[:, :1] selects the top-1 principal component—the single direction in $\mathbb{R}^2$ that captures maximum variance. This $V \in \mathbb{R}^{2\times 1}$ matrix is the learned basis for 1D representation: instead of representing points with 2 coordinates, we can approximate them with 1 coordinate along the principal axis.
• Encoding and Decoding in the Learned Basis Once the basis $V$ is learned, the code demonstrates the encode-decode cycle that underpins all representation learning. For the first data point x = X[0], computing z = V.T @ (x - mu) projects the centered point onto the principal direction, producing a scalar coefficient (the “coordinate” in the 1D basis). This is the encoding step: map high-dimensional data into a low-dimensional representation. To decode, the operation xhat = mu + V @ z reverses the process: scale the basis vector $V$ by the coefficient $z$, then translate back by adding the mean $\mu$. The reconstructed point xhat lies in the 1D subspace defined by $V$, approximating the original $x$. The reconstruction error $\|x - \hat{x}\|$ (not printed but implicit) measures information loss—how much structure was discarded by projecting from 2D to 1D.
• Why This Matters: Basis Choice Determines Representation Efficiency The profound connection between these two examples is that both operations select a basis, then express data as coordinates in that basis. One-hot encoding uses a trivial basis (indicator vectors for categories), while PCA learns an optimal basis (directions of maximum variance). The one-hot basis is sparse (one nonzero entry per vector) but high-dimensional (dimension equals vocabulary size); the PCA basis is dense (all entries contribute) but low-dimensional (dimension chosen to retain desired variance). This trade-off appears throughout ML: word embeddings compress sparse one-hot vectors into dense low-dimensional representations; autoencoders learn nonlinear generalizations of PCA’s linear encoding; transformers learn context-dependent bases via attention. Understanding basis selection—discrete vs. continuous, fixed vs. learned, sparse vs. dense—is the key to designing representation learning systems. When debugging model performance, ask: “Is my basis capturing the right structure?” Too few dimensions (under-parameterized basis) loses signal; too many dimensions (over-parameterized basis) overfits noise. PCA provides a principled answer via explained variance ratio, guiding the choice of dimensionality $k$ for the basis $V[:, :k]$.