ZestCalc

Explainer

Low-Rank Approximation

Every matrix is a sum of simple rank-one pieces, and keeping only the biggest few keeps most of it. See an image rebuilt from a handful of singular values, ratings explained by two tastes, and why LoRA fine-tunes huge models with small updates.

A matrix is a sum of rank-one pieces

The singular value decomposition writes any matrix as A=UΣVTA = U \Sigma V^{T}, which the SVD Calculator computes for small matrices. Multiplied out, that is a sum of simple pieces:

A=σ1u1v1T+σ2u2v2T+⋯+σrurvrTA = \sigma_1 u_1 v_1^{T} + \sigma_2 u_2 v_2^{T} + \cdots + \sigma_r u_r v_r^{T}

Each piece is a column uiu_i times a row viTv_i^{T}, scaled by a singular value σi\sigma_i. A column times a row has rank one: every row of it is a multiple of the same row. The number of pieces rr is the rank of AA, which the Matrix Rank Calculator finds. The singular values come sorted, σ1≥σ2≥⋯≥0\sigma_1 \ge \sigma_2 \ge \cdots \ge 0, so the first pieces are the biggest.

Each piece is a column u times a row v, scaled by σ. A column times a row can only make a pattern of rows and columns, so every piece looks like a plaid. Added up, they rebuild the picture. In the piece, the column, and the row, middle gray is 0, lighter is positive, and darker is negative.

Keep the biggest pieces

Stop the sum after kk pieces and you get AkA_k, a matrix of rank kk. The Eckart–Young theorem says no other rank-kk matrix is closer to AA, measuring distance as the square root of the summed squared differences (the Frobenius norm). The error is set exactly by the pieces you dropped:

∥A−Ak∥∥A∥=σk+12+⋯+σr2σ12+⋯+σr2\frac{\lVert A - A_k \rVert}{\lVert A \rVert} = \sqrt{\frac{\sigma_{k+1}^2 + \cdots + \sigma_r^2}{\sigma_1^2 + \cdots + \sigma_r^2}}

The sum of the squared singular values is often called the matrix's energy. When the singular values fall off fast, a few pieces hold nearly all of it.

Storing AkA_k means storing kk columns of UU, kk columns of VV, and kk singular values: k(m+n+1)k(m + n + 1) numbers instead of mnmn. For a 96 × 96 image at rank 8 that's 1,544 numbers instead of 9,216.

A 96 × 96 grayscale image rebuilt from its largest k singular values. The plaid is a row pattern plus a column pattern, so 2 pieces draw it exactly. A single diagonal edge needs far more.

What compresses well

Low rank means the rows are built from a few shared patterns. Smooth gradients, flat regions, and anything lined up with the rows and columns compress well. The plaid is a row pattern plus a column pattern, so it has rank 2. A diagonal edge is the opposite: each row is the previous one shifted by a pixel, and no small set of shared patterns draws it, so its singular values fall off slowly.

Real image formats such as JPEG use other transforms that do better on photographs. The SVD's strength is generality: it finds the best low-rank summary of *any* table of numbers, and in machine learning that's usually what you want.

Ratings and recommendations

A table of who rated what is a matrix too. People's tastes aren't random: someone who loves one action film probably likes the next. So the ratings matrix is close to low rank, and its top pieces read as hidden factors, here roughly "likes films at all" and "action or romance".

Made-up ratings from 1 to 5. Six people rate three action films and two romances. Two pieces, roughly "likes films" and "action or romance", rebuild every rating to within three quarters of a star.

Recommender systems use this the other way around. Most of a real ratings table is empty, and fitting a low-rank matrix to the ratings that do exist fills in the blanks with predictions. That's matrix factorization, the method made famous by the Netflix Prize.

LoRA: fine-tuning with a low-rank update

A large language model has weight matrices with thousands of rows and columns. Fine-tuning it for a new task changes each weight matrix by some update ΔW\Delta W. LoRA (low-rank adaptation) bets that this update is close to low rank, and learns it directly as a product of two thin matrices:

W′=W+ΔW=W+BA,B∈Rd×r, A∈Rr×dW' = W + \Delta W = W + BA, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times d}

With rr much smaller than dd, that's 2dr2dr numbers to train instead of d2d^2. The original weights stay frozen, and the small update can be stored, swapped, or merged back into WW.

Full ΔW: d × dB: d × rA: r × d
8

A full update has d² = 16,777,216 weights. LoRA with rank 8 learns 2dr = 65,536, which is 0.39% as many.

Fine-tuning changes a weight matrix W by some ΔW. LoRA learns that change as a product BA of a tall, thin matrix and a short, wide one, so it has rank at most r.

The same truncation is also how PCA reduces dimensions: the PCA Calculator keeps the top kk components, which are the top kk right singular vectors of the centered data.