Area 01 · LLM · Concept 03 of 20

Self-Attention

Every token looks at the other tokens and decides which ones matter for its meaning.

01–10 · The basePhase 1 · Weeks 3–6≈ 1.5 hours

The idea

In "The bank raised its rates", the word "bank" means a lender, not a riverbank, because of the words around it. Self-attention is how the model does that. Every token produces three vectors: a query (what am I looking for?), a key (what do I contain?) and a value (what do I pass on?).

Each token compares its query with every earlier token’s key. High match scores mean "pay attention to this one". The scores are turned into weights that sum to 1, and the token’s new vector becomes a weighted mix of the other tokens’ values. Many heads run in parallel, so one head can track grammar while another tracks which noun a pronoun refers to.

The cost grows with the square of the sequence length, because every token compares itself with every other. That is why very long contexts are expensive and why tricks like the KV cache and efficient attention kernels matter so much.

Why it matters

Attention explains both the model’s strength (using context) and its limits: long prompts cost more, and details buried in the middle of a very long prompt can get less attention than those at the start or end.

Picture it

Words to know

  • Query, key, valueThree projections of each token used to compute attention.
  • Attention headOne independent attention computation. Layers have many.
  • Causal maskStops a token from attending to tokens after it, so the model cannot peek ahead.

Try it without code

Find the lost-in-the-middle effect

  1. Paste a long article into a chat app and hide one odd fact in the middle ("The office cat is called Pixel").
  2. Ask a question whose answer is that fact.
  3. Move the fact to the very start and ask again. Compare how reliably it is found.

Use it in code

Scaled dot-product attention in NumPy. This is the formula at the heart of every transformer.

python
import numpy as np

def attention(Q, K, V, causal=True):
    d = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d)                 # how well each query matches each key
    if causal:
        mask = np.triu(np.ones_like(scores), k=1)  # hide future tokens
        scores = np.where(mask == 1, -1e9, scores)
    weights = np.exp(scores - scores.max(-1, keepdims=True))
    weights /= weights.sum(-1, keepdims=True)      # softmax: each row sums to 1
    return weights @ V, weights

rng = np.random.default_rng(0)
x = rng.normal(size=(4, 8))                        # 4 tokens, 8 dims
Wq, Wk, Wv = (rng.normal(size=(8, 8)) for _ in range(3))
out, w = attention(x @ Wq, x @ Wk, x @ Wv)
print(np.round(w, 2))                              # row i = where token i looked
$ python attention.py
PRACTICE · TICK IT AFTER THIS

Run the snippet, then turn off the causal mask and explain in one sentence what changed in the weight matrix and why chat models need the mask.

Common mistakes

  • Reading attention weights as a full explanation of the model’s reasoning. They are one signal among many.
  • Stuffing the most important instruction in the middle of a huge prompt.

Go deeper