ZestCalc

Attention Calculator

Compute scaled dot-product attention, the core of every transformer: exact scores QKᵀ, the softmax weights, and each output as a weighted average of the values. Enter Q, K, and V, or tokens with projection weights.

Queries, keys, and values

One row per token. Q and K need the same number of columns, and K and V the same number of rows. Up to 8 tokens and 6 columns.

Examples

Up to 8 rows and 6 columns.

Use integers, decimals, or fractions such as 1/3. Paste rows from a spreadsheet or a Python list into any cell.

Attention

Updates as you type.

3 queries attend over 3 keys, with d_k = 2.

Attention weights

Each row adds up to 1. Values marked ≈ are rounded.

Query “the”

Scores
[102]\begin{bmatrix} 1 & 0 & 2 \end{bmatrix}
Scaled by √d_k
[2202]\begin{bmatrix} \frac{\sqrt{2}}{2} & 0 & \sqrt{2} \end{bmatrix}
Weights
≈[0.2840.140.576]\approx \begin{bmatrix} 0.284 & 0.14 & 0.576 \end{bmatrix}
Output
≈(0.86, 0.716)\approx \left(0.86,\ 0.716\right)

Query “the” attends most to key “sat”, with weight ≈0.576\approx 0.576.

Scores QKᵀ

102240222

Output AV

≈0.86≈0.716≈0.232≈0.8132/32/3

Scores and scaled scores are exact. Softmax uses e^x, so weights and outputs are rounded, except where they're known exactly.

What attention computes

Attention lets every token in a sequence gather information from the others. Each token has three vectors:

  • a query qq: what it's looking for
  • a key kk: what it offers to be found by
  • a value vv: what it hands over when it's found

Stack them as the rows of QQ, KK, and VV. Scaled dot-product attention, the operation at the heart of every transformer, is then three matrix steps:

A=softmax⁡ ⁣(QKTdk),output=AVA = \operatorname{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}}\right), \qquad \text{output} = AV
  1. Scores. QKTQK^{T} dots every query with every key, the operation the Matrix Multiplication Calculator performs with KK flipped by the Matrix Transpose Calculator. A large score means the query and key point the same way, the idea behind the Cosine Similarity Calculator, without dividing by the lengths.
  2. Weights. Divide by dk\sqrt{d_k}, where dkd_k is the width of the queries and keys. Then apply softmax to each row: esje^{s_j} divided by the sum of all of them. Every weight is positive and each row adds up to 1.
  3. Output. Multiply by VV. Row ii of the output is ∑jaijvj\sum_j a_{ij} v_j, a weighted average of the value vectors, so it always lands inside the shape the values span.

Why divide by √d_k

A dot product adds up dkd_k products. If the entries are random with spread 1, the sum has spread about dk\sqrt{d_k}, so wider vectors give larger scores. Softmax turns large differences into weights very close to 0 and 1: almost all the attention goes to a single key, and the gradients that training relies on almost vanish. Dividing by dk\sqrt{d_k} keeps the scores at spread about 1 whatever the width. The "Large scores" example shows how sharp softmax gets even after scaling.

Where Q, K, and V come from

In a transformer, the tokens arrive as rows of a matrix XX, one embedding per token, as on Data as Vectors and Matrices. Three learned weight matrices turn every token into its query, key, and value:

Q=XWQ,K=XWK,V=XWVQ = XW_Q, \qquad K = XW_K, \qquad V = XW_V

The "Tokens and weights" input does exactly this. The weights are learned by backpropagation like any other layer's: attention is built entirely from matrix products and a softmax, so gradients flow through it with the same transpose and outer-product rules.

The causal mask

A model that writes text one token at a time mustn't peek at the tokens after the current one. The causal mask sets every score above the diagonal to −∞-\infty before the softmax, so e−∞=0e^{-\infty} = 0 and those weights vanish. Query 1 can only see key 1, query 2 sees keys 1 and 2, and so on. GPT-style models use this mask, while models that read a whole text at once, such as BERT, don't.

Reading the result

  • The weights grid is the attention pattern: row ii shows where token ii looks. Diagrams of "what a transformer attends to" are this grid.
  • A tie spreads attention evenly. When a query scores every visible key the same, every weight is exactly 1/n1/n and the output is the plain average of the values. The calculator reports these exactly.
  • Exactness. Scores are exact fractions. Scaled scores are exact too, as roots when dkd_k isn't a perfect square. Softmax needs exe^x, so weights and outputs are rounded and marked ≈\approx.

Real models run many attention heads side by side, each with its own WQW_Q, WKW_K, and WVW_V, and glue their outputs together. Every head is the computation above, with dkd_k around 64 and sequences of thousands of tokens. A transformer layer is attention followed by the kind of layer on Neural Network Layers, stacked dozens of times.