Part 1: Dot Product Similarity (Unnormalized)

The dot product $d_i^\top q$ computes the projection of document $d_i$ onto query $q$ (times $\|q\|$). From the Cauchy-Schwarz inequality: \[ d_i^\top q = \|d_i\| \|q\| \cos(\theta_i), \] so the dot product is the product of three factors: (1) document magnitude $\|d_i\|$, (2) query magnitude $\|q\|$, and (3) cosine of the angle $\cos(\theta_i)$. Implication: Doubling the document length doubles the similarity score, even if the direction (content) is unchanged. For text, this biases toward longer documents; for embeddings, it favors vectors with larger norms (which may encode confidence or frequency, but often are arbitrary). When to use: Dot product is appropriate when magnitude is informative (e.g., attention logits where larger values indicate stronger relevance, or when aggregating weighted votes where magnitude encodes weight).

Part 2: Cosine Similarity (Normalized, Scale-Invariant)

Cosine similarity $\frac{d_i^\top q}{\|d_i\| \|q\|}$ isolates the angular component, removing magnitude: \[ \cos(\theta_i) \in [-1, 1]. \] Values near 1 indicate alignment (small angle), near 0 indicate orthogonality (90°), and near -1 indicate opposition (180°). Implication: Doubling document length doesn’t change the score the direction matters. This is equivalent to projecting all vectors onto the unit sphere (normalize to length 1), then computing dot products. When to use: Cosine is standard when scale is arbitrary or irrelevant: text retrieval (document length varies), image embeddings (CNN features have varying norms), contrastive learning (embeddings trained with L2 normalization).

Part 3: Why Rankings Differ (Norm Variation)

In the example: $d_1 = [10, 0]^\top$, $d_2 = [0.5, 0.5]^\top$, $q = [1, 0.5]^\top$.

Dot products: - $d_1^\top q = 10$: Large norm ($\|d_1\| = 10$) dominates despite $\theta_1 \approx 26.6°$. - $d_2^\top q = 0.75$: Small norm ($\|d_2\| \approx 0.707$) suppresses score despite smaller angle $\theta_2 \approx 18.4°$. - Ranking: Document 1 > Document 2 (norm wins).

Cosine similarities: - $\cos(\theta_1) \approx 0.894$: Alignment moderate. - $\cos(\theta_2) \approx 0.949$: Alignment best (smallest angle). - Ranking: Document 2 > Document 1 (angle wins).

General principle: Dot product rankings mix magnitude and angle; cosine rankings are purely angular. When norms vary by orders of magnitude (common in sparse TF-IDF or unnormalized embeddings), dot product and cosine can produce opposite orderings.

Why This Matters for ML

Information retrieval: Search engines rank billions of documents against user queries. Cosine similarity prevents bias toward long documents (more words more relevant). Google’s early PageRank used TF-IDF + cosine for this reason.

Semantic search (BERT, Sentence-BERT): Dense embeddings from transformers encode meaning. Cosine similarity finds semantically related sentences, even if phrasing differs ("The cat sits on the mat" vs. "Feline rests on a rug" cosine despite different words).

Contrastive learning (SimCLR, MoCo, CLIP): Training objective maximizes cosine similarity for positive pairs (augmented views of same image, or image-caption pairs), minimizes for negatives. Loss functions like NT-Xent (normalized temperature-scaled cross-entropy) normalize embeddings to the unit sphere: \[ \\mathcal{L}_i = -\\log \\frac{\\exp(\\cos(z_i, z_j) / \\tau)}{\\sum_k \\exp(\\cos(z_i, z_k) / \\tau)}, \] where $z_i, z_j$ are positive pairs, $\\tau$ is temperature, and $\\cos(\\cdot)$ is cosine similarity.

Clustering (K-means variants): Spherical K-means uses cosine similarity instead of Euclidean distance, clustering on the unit sphere. Better for text (document length is noise) and directional data (wind directions, orientations).

Recommendation systems: Collaborative filtering compares user-item vectors. Cosine similarity finds similar users ("users who rated these items similarly") regardless of rating scale differences (one user rates 1, another rates 3).

ML Examples and Patterns

(Included in the Code Introduction section below for the detailed implementation examples)

Connection to Linear Algebra Theory

Inner product as projection: The dot product $d^\top q$ measures the projection of $d$ onto $q$ (times $\|q\|$). From linear algebra: \[ \\text{proj}_q(d) = \\frac{d^\top q}{\\|q\\|^2} q, \] so $d^\top q = \\|q\\|^2 \\cdot \\text{scalar projection of } d \\text{ onto } q$.

Cauchy-Schwarz inequality: Bounds the dot product: \[ |d^\top q| \\le \\|d\\| \\|q\\|, \] with equality when $d$ and $q$ are parallel. This implies: \[ \\frac{d^\top q}{\\|d\\| \\|q\\|} \\in [-1, 1], \] i.e., cosine similarity is always in the valid range for a cosine function.

Unit sphere normalization: Cosine similarity is equivalent to computing dot products on the unit sphere: \[ \\hat{d} = \\frac{d}{\\|d\\|}, \\quad \\hat{q} = \\frac{q}{\\|q\\|}, \\quad \\cos(\\theta) = \\hat{d}^\\top \\hat{q}. \] All vectors of the same direction collapse to a single point on the unit sphere (scale-invariant representation).

Orthogonality: Vectors with $\\cos(\\theta) = 0$ (dot product zero) are orthogonal (perpendicular). In high dimensions, random vectors are approximately orthogonal (concentration of measure).

Angular distance: Cosine similarity induces a distance metric (arc length on unit sphere): \[ d_{\\text{angular}}(d, q) = \\arccos\\left( \\frac{d^\\top q}{\\|d\\| \\|q\\|} \\right). \] This satisfies the triangle inequality (unlike raw dot product), making it a proper metric.

Numerical and Implementation Notes

Axis parameter in np.linalg.norm: - axis=1: Compute norm of each row (per document). Correct for document-query similarity. - axis=0: Compute norm of each column (per feature). Wrong for this task. - axis=None (default): Compute Frobenius norm (flatten entire matrix). Also wrong.

Broadcasting in division:

cos = dot / (doc_norms * q_norm)

• dot has shape (n,)
• doc_norms has shape (n,)
• q_norm is a scalar
• doc_norms * q_norm broadcasts to shape (n,) (scalar multiplies each element)
• Division is element-wise: cos[i] = dot[i] / (doc_norms[i] * q_norm)

Avoiding division by zero:

eps = 1e-8
cos = dot / (doc_norms * q_norm + eps)

Prevents nan or inf when norms are near zero (e.g., all-zeros document or query).

Pre-normalization for multiple queries:

# Normalize documents once
docs_norm = docs / np.linalg.norm(docs, axis=1, keepdims=True)

# For each query
q_norm = q / np.linalg.norm(q)
cos = docs_norm @ q_norm  # Already cosine similarity!

Saves repeated norm computation (amortizes cost over many queries).

Ranking descending (highest first):

indices = np.argsort(-scores)  # Negation flips to descending
top_k = indices[:k]  # Top k documents

Numerical stability for very high dimensions: For $d \\sim 10^4$ (e.g., BERT embeddings), naive norm computation can overflow. Use stable alternatives:

# Stable norm (computes sqrt(sum(x**2)))
norm = np.linalg.norm(x)  # Uses numerically stable BLAS routine

# Manual (less stable for large d)
norm = np.sqrt((x**2).sum())  # Can overflow if x has large values

Numerical and Shape Notes

Shape discipline: - $q \\in \\mathbb{R}^d$: query vector (e.g., $d=768$ for BERT-base) - $\\text{docs} \\in \\mathbb{R}^{n \\times d}$: document matrix ($n$ documents, each $d$-dimensional) - $\\text{dot} = \\text{docs} \\, q \\in \\mathbb{R}^n$: dot products (one per document) - $\\|\\text{docs}\\|_2 \\in \\mathbb{R}^n$: row-wise norms (np.linalg.norm(docs, axis=1)) - $\\|q\\|_2 \\in \\mathbb{R}$ (scalar): query norm - $\\cos(\\theta) \\in \\mathbb{R}^n$: cosine similarities (one per document)

Keepdims for broadcasting:

# Without keepdims: shape (n,), can't broadcast correctly for column-wise ops
doc_norms = np.linalg.norm(docs, axis=1)  # shape (n,)

# With keepdims: shape (n, 1), broadcasts correctly
doc_norms = np.linalg.norm(docs, axis=1, keepdims=True)  # shape (n, 1)
docs_normalized = docs / doc_norms  # broadcasts to (n, d)

Transposing for batch queries:

# Multiple queries: Q is (m, d), docs is (n, d)
dots = Q @ docs.T  # (m, d) @ (d, n) = (m, n)
# dots[i, j] = similarity of query i to document j

Memory efficiency for large corpora: For $n \\sim 10^9$ documents (web-scale), computing all similarities is infeasible. Use approximate nearest neighbors (ANN): - FAISS (Facebook AI Similarity Search): GPU-accelerated ANN with quantization - ScaNN (Google): compressed embeddings + learned distance metrics - Annoy (Spotify): tree-based partitioning

Pedagogical Significance

This example is the foundational demonstration of similarity metrics in ML:

Key takeaways: 1. Dot product biases toward magnitude: Large vectors win, even if misaligned. 2. Cosine similarity is scale-invariant: Normalization focuses on direction (angle). 3. Rankings can flip: Same data, different metric opposite orderings. 4. Unit sphere geometry: Cosine = dot product on normalized vectors. 5. Choose metric by task: Magnitude informative? Use dot product. Scale arbitrary? Use cosine.

Common misconceptions addressed: - "Dot product and cosine are equivalent": No scales with magnitude, cosine doesn’t. - "Normalization is optional": No’s critical when scale is uninformative (text, images). - "Cosine is always better": No on whether magnitude encodes useful information. - "Rankings are robust to metric choice": No example shows rankings can reverse entirely.

Connection to other examples: - Example 80 (Attention): Attention uses unnormalized $QK^\\top$; softmax provides implicit normalization. - Example 84 (Backprop): Gradients involve inner products ($X^\\top \\frac{\\partial L}{\\partial Y}$); gradient clipping normalizes by magnitude. - Example 83 (PCA): Principal components are unit vectors; projections use dot products but components are normalized.

Why this snippet is powerful: It isolates the core geometric distinction between magnitude-dependent and magnitude-independent similarity in minimal code. The ranking flip (Document 1 vs. Document 2) makes the difference visceral an abstract formula, but a concrete reversal of preferences. This is the gateway to understanding modern retrieval (Google Search, semantic search, recommender systems) and contrastive learning (SimCLR, CLIP), where cosine similarity is ubiquitous. Every embedding-based system (BERT, GPT, diffusion models, face recognition) uses these metrics. The principle when scale is noise beyond ML to any high-dimensional data analysis (PCA, clustering, regression). This is the most important similarity metric pattern in the entire book.