Area 01 · LLM · Concept 02 of 20

Transformer Architecture

The network design behind GPT, Claude and Llama: stacked attention and feed-forward layers.

01–10 · The basePhase 1 · Weeks 3–6≈ 2 hours

The idea

A transformer turns each token ID into a vector (an embedding), adds information about the token’s position, and then passes the whole sequence through a stack of identical blocks. Each block has two parts: an attention layer, where tokens exchange information with each other, and a feed-forward layer, where each token’s vector is transformed on its own.

Modern chat models are decoder-only transformers. They read the sequence left to right and, at the top of the stack, produce a probability for every token in the vocabulary as the next token. The model picks one, appends it, and runs again. That loop is generation.

Size comes from three knobs: how many blocks (layers), how wide each vector is, and how many attention heads each layer has. A "70B" model has about 70 billion learned numbers spread across those layers. More parameters generally means more knowledge and better reasoning, at the cost of memory and speed.

Why it matters

You do not need to build one, but knowing the shape explains context limits, why output is slower than input, why the KV cache exists, and what quantization and LoRA are actually changing.

Picture it

Words to know

  • Layer / blockOne attention + feed-forward unit. Models stack dozens of them.
  • EmbeddingThe vector that represents a token inside the model.
  • LogitsRaw scores for every possible next token, before they become probabilities.
  • Decoder-onlyA transformer that only predicts the next token from the tokens before it.

Try it without code

Watch next-token prediction

  1. Open a chat app and ask it to continue the sentence "The capital of France is" with exactly one word.
  2. Then ask: "List the five most likely next words and roughly how likely each is."
  3. Notice the model is ranking options, not looking up a fact.

Use it in code

Load a small open model and print its top next-token guesses. This is the whole idea of generation in a few lines.

python
# pip install torch transformers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

name = "Qwen/Qwen2.5-0.5B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

ids = tok("The capital of France is", return_tensors="pt").input_ids
with torch.no_grad():
    logits = model(ids).logits[0, -1]          # scores for the next token
probs = torch.softmax(logits, dim=-1)
top = torch.topk(probs, 5)
for p, i in zip(top.values, top.indices):
    print(f"{tok.decode(i)!r:>12}  {p:.2%}")

print(model.config.num_hidden_layers, "layers,", model.config.hidden_size, "wide")
$ python next_token.py
PRACTICE · TICK IT AFTER THIS

Extend the script into a manual generation loop: pick the top token, append it, repeat 20 times, and print the sentence as it grows.

Common mistakes

  • Thinking the model stores facts in a database it looks up. Knowledge is spread across the weights.
  • Confusing encoder models (like BERT, used for embeddings) with decoder models (used for chat).

Go deeper