Attention Calculator
Compute scaled dot-product attention, the core of every transformer: exact scores QKᵀ, the softmax weights, and each output as a weighted average of the values. Enter Q, K, and V, or tokens with projection weights.
Queries, keys, and values
One row per token. Q and K need the same number of columns, and K and V the same number of rows. Up to 8 tokens and 6 columns.
Up to 8 rows and 6 columns.
Use integers, decimals, or fractions such as 1/3. Paste rows from a spreadsheet or a Python list into any cell.
Attention
Updates as you type.
3 queries attend over 3 keys, with d_k = 2.
Attention weights
Each row adds up to 1. Values marked ≈ are rounded.
Query “the”
- Scores
- Scaled by √d_k
- Weights
- Output
Query “the” attends most to key “sat”, with weight .
Scores QKᵀ
Output AV
Scores and scaled scores are exact. Softmax uses e^x, so weights and outputs are rounded, except where they're known exactly.
What attention computes
Attention lets every token in a sequence gather information from the others. Each token has three vectors:
- a query : what it's looking for
- a key : what it offers to be found by
- a value : what it hands over when it's found
Stack them as the rows of , , and . Scaled dot-product attention, the operation at the heart of every transformer, is then three matrix steps:
- Scores. dots every query with every key, the operation the Matrix Multiplication Calculator performs with flipped by the Matrix Transpose Calculator. A large score means the query and key point the same way, the idea behind the Cosine Similarity Calculator, without dividing by the lengths.
- Weights. Divide by , where is the width of the queries and keys. Then apply softmax to each row: divided by the sum of all of them. Every weight is positive and each row adds up to 1.
- Output. Multiply by . Row of the output is , a weighted average of the value vectors, so it always lands inside the shape the values span.
Why divide by √d_k
A dot product adds up products. If the entries are random with spread 1, the sum has spread about , so wider vectors give larger scores. Softmax turns large differences into weights very close to 0 and 1: almost all the attention goes to a single key, and the gradients that training relies on almost vanish. Dividing by keeps the scores at spread about 1 whatever the width. The "Large scores" example shows how sharp softmax gets even after scaling.
Where Q, K, and V come from
In a transformer, the tokens arrive as rows of a matrix , one embedding per token, as on Data as Vectors and Matrices. Three learned weight matrices turn every token into its query, key, and value:
The "Tokens and weights" input does exactly this. The weights are learned by backpropagation like any other layer's: attention is built entirely from matrix products and a softmax, so gradients flow through it with the same transpose and outer-product rules.
The causal mask
A model that writes text one token at a time mustn't peek at the tokens after the current one. The causal mask sets every score above the diagonal to before the softmax, so and those weights vanish. Query 1 can only see key 1, query 2 sees keys 1 and 2, and so on. GPT-style models use this mask, while models that read a whole text at once, such as BERT, don't.
Reading the result
- The weights grid is the attention pattern: row shows where token looks. Diagrams of "what a transformer attends to" are this grid.
- A tie spreads attention evenly. When a query scores every visible key the same, every weight is exactly and the output is the plain average of the values. The calculator reports these exactly.
- Exactness. Scores are exact fractions. Scaled scores are exact too, as roots when isn't a perfect square. Softmax needs , so weights and outputs are rounded and marked .
Real models run many attention heads side by side, each with its own , , and , and glue their outputs together. Every head is the computation above, with around 64 and sequences of thousands of tokens. A transformer layer is attention followed by the kind of layer on Neural Network Layers, stacked dozens of times.
Related calculators
The linear algebra underneath, and more of the machine learning section.
Cosine Similarity Calculator
Compare up to eight vectors by the angle between them, with exact cosines, a similarity grid, and the closest match for each. Includes toy word vectors and king − man + woman.
Matrix Multiplication Calculator
Multiply two to four matrices with exact fractions, entry-by-entry explanations, and Python list import and export.
Matrix Transpose Calculator
Transpose matrices up to 12×12 in exact fractions, watch rows become columns, and check whether a matrix is symmetric or skew-symmetric.
Dot Product and Angle Calculator
Find the dot product, magnitudes, unit vectors, and the angle between two vectors, with exact roots like √14. Drag vectors and watch the sign change at 90°.