A question takes a second to answer. Everything that made the answer possible took months. Follow the sentence through the machine that produced it.
Collect, filter, deduplicate, tokenize.
begin with the open web
Petabytes in. A fraction kept.
Everything starts as text nobody wrote for a model: web pages, code repositories, scanned papers. A filter decides what is worth keeping, and how much it rejects depends on the recipe. A single crawl snapshot runs to petabytes before the filter touches it.
MinHash dedup, then language and quality filters
Near copies out. Mixture weights in.
The same page appears thousands of times across a crawl. MinHash signatures find near duplicates so each document is learned once. Then mixture weights decide how much code, math, and books the model sees relative to the open web.
byte-pair encoding merges frequent pairs
Bytes merge into pieces the model can count.
A tokenizer starts from raw bytes and repeatedly merges the most frequent adjacent pair. After a hundred thousand merges, common words are single tokens and rare words split into parts. Llama 3 uses a vocabulary of 128,256 tokens. The question at the top of this page becomes a handful of them.
Vectors, positions, attention, experts.
look up d = 4,096 vectors and rotate by position
A token becomes a vector. Position becomes an angle.
Each token id looks up a vector of a few thousand numbers. Position is not stored in that vector. Rotary embeddings rotate pairs of coordinates in the query and key projections by an angle proportional to position, so the score between two tokens depends on how far apart they are.
project Q, K, V [causal mask on]
Every token asks the tokens before it.
A query from one position scores against the keys of every earlier position. Softmax turns the scores into weights, and the weighted sum of values flows back into that position. Future positions are masked to minus infinity, so a token predicting the next word cannot read it.
Without the mask, each position can read its own answer. Training loss drops fast and the curve looks brilliant. The model then falls apart the moment it has to generate without the future in view.
Checked invariant NoFutureTokenLeakage. The failure is reachable in the abstract model; this toggle shows it, not a measurement.
add the residual stream, RMSNorm, into SwiGLU or MoE
Wide feed-forward layers. Only a few used per token.
After attention, each position passes through a feed-forward block. In a mixture-of-experts layer a small router scores many such blocks and sends the token to the top two. The model holds far more parameters than any single token touches, at the cost of routing and communication.
Six parts, one residual stream, repeated layer after layer on every forward pass. Drag the drawing to rotate it. Choose a part to read what it does.
Splits text into token ids and looks up each one as a dense vector in the residual stream.
Distribute, backpropagate, update, repeat.
next-token cross-entropy, then backpropagate
One model. Thousands of GPUs. One gradient.
No single GPU holds the model. Tensor parallelism splits a layer across the GPUs in a node, pipeline parallelism splits layers across nodes, and data parallelism gives each group its own batch. Gradients ring-reduce across the whole cluster before any weight changes.
A large activation overflows a 16-bit float. Infinity becomes NaN, NaN spreads through every weight it touches, and the loss goes vertical. Clipping bounds finite gradients but cannot repair a value that is already infinite. Recovery means rewinding to the last healthy checkpoint.
Checked invariant FiniteWeightsAndLoss. The failure is reachable in the abstract model; this toggle shows it, not a measurement.
clip the global gradient norm to 1.0 [grad clip on]
Gradients rescaled. Moments updated. Weights moved.
The global gradient norm is clipped to a ceiling, often 1.0. AdamW keeps running averages of the gradient and its square in 32-bit precision and uses them to scale each weight’s update. A learning-rate schedule warms up, holds, and decays over the run.
The same arithmetic repeats across eleven orders of magnitude of time. Drag the cursor along the log scale, then move the sliders to see what a run costs.
420 ms One global training step: forward, backward, and all-reduce.
repeat for 15T tokens, about 90 days, then anneal on math and code
next micro-batch forward pass [loop back to attention]A world simulator that does not yet take instructions.
Weights are serialized to disk. The result predicts the next token of almost any text whose shape it has seen. Ask it a question and it may continue with five more questions, because that is what questions are often followed by on the web.
Demonstrations, feedback, serving, the answer.
train on prompt and assistant conversations
Demonstrations teach the shape of an answer.
Thousands of prompt-and-response pairs, written or reviewed by people, teach the model to answer rather than continue. The loss is computed only on the assistant’s tokens. The prompt is context, not target.
Optimize hard against an imperfect reward model with nothing holding the policy near its reference, and it finds whatever the reward model over-rates: length, confidence, a favourite phrasing. Scores climb while answers degrade.
Checked invariant NoRewardExploit. The failure is reachable in the abstract model; this toggle shows it, not a measurement.
RLHF or GRPO with a KL penalty, β = 0.04 [KL leash on]
Sample several answers. Reward the better ones. Stay close to home.
For each prompt the policy samples a group of responses. Verifiers check code and math; preference models score the rest. Group-relative advantage pushes probability toward the better responses, and a KL penalty keeps the policy near the supervised model.
quantize and deploy to a paged KV-cache engine
Compute the prefix once. Then append.
At inference the prompt is processed in one parallel pass and its keys and values are cached. Each new token attends over the cache and appends its own entry. PagedAttention stores the cache in fixed-size pages so memory is not wasted on padding.
speculative draft, temperature and top-p sampling
A streamed reply. Evaluation still matters.
The question from the top of the page arrives. Tokens stream out one at a time, each sampled from a distribution over the vocabulary. Reaching this point proves the pipeline ran, not that the answer is grounded, calibrated, or safe. That takes separate evidence.
One question, six tokens, and a pipeline that took months to build before it could reply in a second.
Back to the questionThis is a common dense or mixture-of-experts decoder recipe, simplified. Real pipelines differ in order, scale, and detail. The drawings illustrate mechanisms; the sliders compute teaching examples, not measurements of any real model. Numbers such as 128,256 tokens, 16,384 GPUs, and 15 trillion tokens are published figures from the Llama 3 report used as scale anchors.
The three failure detours come from a 16-state abstract lifecycle model checked with TLC. Each named invariant is violated by a reachable state in that model. That check says nothing about whether any real model is correct or safe. TLC report · TLA+ module · state graph.
The scaling lab uses the parametric fit from Hoffmann et al., Training Compute-Optimal Large Language Models (2022): L(N, D) = 1.69 + 406.4 / N0.34 + 410.7 / D0.28. Compute uses the common C ≈ 6ND approximation. Memory assumes 2 bytes for weights, 2 for gradients, and 12 for Adam moments and a 32-bit master copy.
Sources: The Llama 3 Herd of Models · Attention Is All You Need · RoFormer · Switch Transformers · Chinchilla · PagedAttention · DeepSeekMath (GRPO) · PyTorch gradient clipping.
Drawings and page by Praneeth Paikray. The visual form owes a debt to Kevin T. Ngo’s statechart-art, whose MIT-licensed lettering face, KevinSans, sets the headings. Source for the drawings: card-scenes.js.
