AI Careers

Implement Attention From Scratch: The Interview Version and Its Follow-Ups

Scaled dot-product attention fits in ten lines of NumPy. The interview is the follow-ups: why divide by the square root of d, how to make softmax stable, where the causal mask goes, what the KV cache saves, and why grouped-query attention exists. A complete walkthrough.

Why labs ask this

Asking a candidate to implement attention is the ML equivalent of asking them to reverse a linked list: small enough to finish, deep enough to separate people who have used transformers from people who understand them. It comes up in ML-leaning rounds at AI labs and at companies building on models, sometimes as live coding and sometimes as a whiteboard discussion.

The code itself is short. What the interviewer is really testing is the follow-ups, each of which probes a piece of understanding that matters in practice - numerical stability, masking, complexity, and the memory trade-offs that dominate inference cost. This post builds the implementation and then works through each follow-up in the order they usually arrive.

Scaled dot-product attention, stable softmax, causal mask

Attention computes, for every query position, a weighted average of value vectors, where the weights come from how well that query matches each key. In matrix form it is softmax of Q times K-transpose divided by the square root of d, times V.

Two details are where candidates lose points. First, softmax must be numerically stable: subtract the row maximum before exponentiating, or large logits overflow to infinity and you get NaNs. Mathematically the result is identical, because the constant cancels in the normalisation. Second, the causal mask goes on the logits before the softmax, as negative infinity, so masked positions get exactly zero weight. Masking after the softmax leaves rows that no longer sum to one.

import numpy as np

def softmax(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)      # stability: exp never overflows
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

def attention(q, k, v, causal=False):
    """q: (n, d)  k: (m, d)  v: (m, dv)  ->  (n, dv)"""
    d = q.shape[-1]
    scores = q @ k.T / np.sqrt(d)                # (n, m)
    if causal:
        n, m = scores.shape
        mask = np.triu(np.ones((n, m), dtype=bool), k=m - n + 1)
        scores = np.where(mask, -np.inf, scores)  # mask BEFORE softmax
    return softmax(scores) @ v

rng = np.random.default_rng(0)
x = rng.normal(size=(4, 8))
out = attention(x, x, x, causal=True)
assert out.shape == (4, 8)
assert np.allclose(out[0], x[0])                 # first token can only see itself

Follow-up one: why divide by the square root of d?

If the components of q and k are independent with mean zero and variance one, their dot product has variance d. As d grows, the logits spread out, the softmax saturates toward one-hot, and its gradient vanishes for every position except the winner. Dividing by the square root of d brings the variance back to one, keeping the softmax in a regime where it can still learn. You can demonstrate it in two lines by measuring the variance of random dot products at d equal to 64 and 1024.

The follow-up's follow-up is complexity. Computing the scores is O(n squared times d) time and O(n squared) memory for a sequence of length n. That quadratic memory is why long-context models rely on kernels like FlashAttention, which compute the same result in tiles without materialising the full score matrix. You do not need to implement it; you should know why it exists.

Follow-up two: multi-head attention

Multi-head attention projects the input into h smaller query, key and value spaces, runs attention in each independently, concatenates the results and projects back. Each head can learn a different notion of relevance at the same total cost as one wide head. The implementation is mostly reshaping, and the interviewer is watching whether you get the axes right.

def multi_head(x, Wq, Wk, Wv, Wo, h, causal=True):
    n, d_model = x.shape
    dh = d_model // h
    def split(t):                                 # (n, d_model) -> (h, n, dh)
        return t.reshape(n, h, dh).transpose(1, 0, 2)
    q, k, v = split(x @ Wq), split(x @ Wk), split(x @ Wv)
    heads = np.stack([attention(q[i], k[i], v[i], causal) for i in range(h)])
    concat = heads.transpose(1, 0, 2).reshape(n, d_model)
    return concat @ Wo

d_model, h = 16, 4
W = [rng.normal(size=(d_model, d_model)) / np.sqrt(d_model) for _ in range(4)]
y = multi_head(rng.normal(size=(5, d_model)), *W, h=h)
assert y.shape == (5, d_model)

Follow-up three: the KV cache and grouped-query attention

During generation a model produces one token at a time, and each new token attends to every previous one. Recomputing keys and values for the whole prefix at every step would make generation quadratic. The KV cache stores each layer's keys and values as they are produced, so each step computes projections only for the new token and attends over the cached ones.

The cost moves from compute to memory. Per token, the cache holds two tensors - keys and values - per layer, each of size number-of-KV-heads times head dimension. For a model with 80 layers, 64 heads of dimension 128 and 16-bit values, that is 2 x 80 x 64 x 128 x 2 bytes, about 2.6 MB per token, or roughly 10 GB for a single 4,000-token sequence.

That arithmetic is why grouped-query attention exists. Instead of one key and value head per query head, several query heads share one. With 8 KV heads instead of 64, the same model needs about 320 KB per token - an eight-fold saving that directly translates into more concurrent users per GPU. Multi-query attention is the extreme case of a single shared KV head. Ending your answer with that number, and what it means for serving, is what turns a correct implementation into a strong interview.

Keep reading

Related articles