Skip to content
Align your AI

Align Your AI · Part One

Contents

Chapter 2

How models are made

A customer opens a bank’s support chat and types: “Can I raise my card limit before a trip?” A second later an answer appears. In that second, a good deal happened. Software wrapped the question in the bank’s standing instructions and the earlier turns of the conversation, then formatted the whole thing the way the model expects. A server holding a compressed copy of the model’s learned values turned the text into numbers, ran them through the model, and picked the answer one word fragment at a time. Each of those steps was put there by someone, and each can change. This chapter explains them in the order they are built, so the failure chapters that follow make sense to a reader who has never trained a model.

What a model is

A large language model gets its behavior from billions of learned numbers called weights, or parameters. It reads text as tokens, chunks of words produced by a separate program called a tokenizer, and for every position it scores each possible next token.1 Whatever training taught the model, including when to refuse a request, is stored in those weights. There is no rule list inside that anyone can read.

A checkpoint is the weights saved as files. It usually comes with a configuration file, the tokenizer and a chat template. Anything that changes the weights, whether more training, combining two models, editing or compression, can change behavior in places no one thought to check.

Pretraining makes a base model

The first and most expensive stage is pretraining. The model is shown a vast amount of text, much of it gathered from the web, and learns to predict what comes next; the result is called a base model.2 A base model continues text. It does not reliably follow instructions or decline harmful requests.

Whatever is in the training data can be learned, including material someone planted on purpose. Chapter 8 covers data poisoning and other supply-chain problems. Filtering the data is one of the few safeguards that acts before any behavior exists.

Post-training makes an assistant

Post-training turns a base model into an assistant. In supervised fine-tuning, the model is trained on examples of the desired behavior, often written by people. In preference training, people compare pairs of the model’s answers and the model is tuned toward the ones they preferred. OpenAI’s 2022 InstructGPT paper described this recipe.3

The classic version is called reinforcement learning from human feedback (RLHF). A separate reward model is trained to predict people’s preferences, and the main model is then optimized to score well with it.4 A later method, direct preference optimization, reaches a similar result without the separate reward model.

Chat models are trained both to follow instructions and to behave safely, so refusing a harmful request is one learned behavior among many.5 It can be strengthened or worn away by later training like any other.

A newer stage scores the model’s output with a program instead of a person: unit tests for code, an answer check for mathematics. DeepSeek reported that this kind of reinforcement learning taught its R1 model to reason step by step without human-written examples of reasoning.6 Models trained this way write out a chain of thought before answering, a technique first shown by prompting models to spell out intermediate steps.7 That written reasoning is generated text like any other. It is useful evidence about what the model is doing, and Chapter 9 discusses how far it can be trusted.

Every one of these stages optimizes a stand-in for what people actually want. A reward model stands in for human judgment, a rating for quality, a test suite for correct code. Where the stand-in and the intent come apart, training follows the stand-in, which is the subject of Chapter 6. A company that tunes its prompts against an automated grader, or trains on its users’ thumbs-up clicks, is running the same kind of optimization, as OpenAI’s experience in Chapter 1 shows.

Adapting a model makes a new one

A company running its own copy of an open model can fine-tune it on its own data. Full fine-tuning updates every weight. A popular cheaper method, LoRA, freezes the original weights and trains small added matrices with far fewer parameters.8 QLoRA does the same on top of a compressed base model, so it fits on smaller hardware.9

The result, called an adapter, is a small file trained against one specific base model. Retraining the same adapter on a newer version of the base produces a different model, even when the recipe is identical.

Two other techniques build new models from existing ones. Merging combines several fine-tuned versions of the same base, for instance by adding together the changes each one made to the weights.10 Distillation trains a smaller “student” model to reproduce a larger “teacher” model’s outputs.11 Today it often takes the form of one model writing training examples for another. Either way, the new model can pick up traits that nobody listed, a problem Chapter 8 takes up.

Serving changes the model too

Before a model goes into service, companies often quantize it: store its weights at lower numerical precision to save memory and speed up answers. One widely used method compressed large models to a few bits per weight with little loss of accuracy on its benchmarks.12 Those benchmarks measured accuracy only. Whether the compressed model still refuses what it should is a separate question, which is why Chapter 7 treats compression as a change to be tested.

The inference engine is the software that runs the model. It makes its own choices about precision, about batching many users’ requests together and about which server handles which request, so the same weights can answer differently under another engine or another configuration. Anthropic’s serving bugs in Chapter 1 lived here. The engine also keeps a large working memory of earlier parts of each conversation, called the key-value cache, and modern serving systems are designed around managing it efficiently.13

The engine picks each next token from the model’s scores using a decoding rule, and the rule matters. A well-known study showed that decoding choices alone can make the same model’s text bland, incoherent or stuck in repetitive loops.14 Settings such as temperature add randomness, so one prompt can get different answers on different runs. That randomness is why any before-and-after comparison first needs to know how much answers vary with no change at all, a point Chapter 4 develops.

A conversation, however it looks on screen, reaches the model as a single sequence of tokens. A chat template turns the list of messages into that sequence, marking where each turn begins with special control tokens, and formats differ between models.2 The system prompt holds the deployer’s standing instructions. It sits in the same stream as the user’s words and any documents the system retrieves, and it outranks them only as far as training taught the model to let it. That is the opening attackers use in prompt injection, covered in Chapter 10.

The stack

Put together, what answers a user is a stack: the weights and any adapters, their precision, the inference engine and its sampling settings, the chat template and the system prompt, and, for an agent, the surrounding software and tools described in Chapter 5. Each layer can change on its own, and each change produces a system that has not yet been tested. The bank’s support chat from the opening of this chapter has at least five such layers. A team that writes them all down, with versions, knows what it is actually serving.

Looking inside a model

Most organizations will only ever see a model’s inputs and outputs. Researchers who hold the weights can do more. Activations are the intermediate values a model computes for a given input on the way to its answer. A probe is a small, separately trained classifier that reads a property from those activations.15 One study found that in the open chat models it tested, refusal is carried by a single direction in the activations: remove it and the model stops refusing, add it and the model refuses harmless requests.5

These methods need full access to the weights, which companies running their own models have and API customers do not. A probe is also trained on one particular set of weights, so after the model is retrained or fine-tuned it needs checking again before its readings can be trusted, as Chapter 7 discusses.

Further reading

  • The Illustrated Transformer, by Jay Alammar, explains the architecture behind these models one diagram at a time.16
  • Dive into Deep Learning, by Aston Zhang and co-authors, is a free book of runnable notebooks.17
  • Andrej Karpathy’s Neural Networks: Zero to Hero builds a GPT-style model and its tokenizer from scratch in code.1 Sebastian Raschka’s Build a Large Language Model (From Scratch) covers the same ground in book form.18
  • Stanford’s CS336, Language Modeling from Scratch, has students build a model end to end.19
  • Nathan Lambert’s RLHF Book covers post-training in depth.20

Notes

  1. Andrej Karpathy. “Neural Networks: Zero to Hero.” course page and video series, karpathy.ai, living page. https://karpathy.ai/zero-to-hero.html. From backpropagation to a GPT; one lecture builds the GPT tokenizer, which “translates between strings and tokens (text chunks)”. ↩︎
  2. Hugging Face. “Chat templates.” Transformers documentation, living page. https://huggingface.co/docs/transformers/chat_templating. Pretraining creates a base model that is then fine-tuned for chat; “The chat is still just a sequence of tokens”; different models use different formats and control tokens. ↩︎
  3. Ouyang, Wu, Jiang et al. “Training language models to follow instructions with human feedback.” arXiv:2203.02155, 2022-03-04. Supervised fine-tuning on labeler demonstrations, then RLHF on rankings of outputs (InstructGPT). ↩︎
  4. Rafailov, Sharma, Mitchell, Ermon, Manning, Finn. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” arXiv:2305.18290, 2023-05-29. RLHF fits a reward model then optimizes against it; DPO solves the same problem with a classification loss. ↩︎
  5. Arditi, Obeso, Syed et al. “Refusal in Language Models Is Mediated by a Single Direction.” arXiv:2406.11717, 2024-06-17. Across open chat models, erasing one direction stops refusal and adding it elicits refusal of harmless requests; chat models are “fine-tuned for both instruction-following and safety”. ↩︎
  6. DeepSeek-AI, Daya Guo, Dejian Yang et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” Nature volume 645, pages 633-638 (2025); arXiv:2501.12948, 2025-01-22. Reasoning incentivized by RL without human-labeled reasoning trajectories. ↩︎
  7. Wei, Wang, Schuurmans et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” arXiv:2201.11903, 2022-01-28. Prompting with intermediate reasoning steps improves large models’ reasoning. ↩︎
  8. Hu, Shen, Wallis et al. “LoRA: Low-Rank Adaptation of Large Language Models.” arXiv:2106.09685, 2021-06-17. Freezes pretrained weights and trains injected low-rank matrices. ↩︎
  9. Dettmers, Pagnoni, Holtzman, Zettlemoyer. “QLoRA: Efficient Finetuning of Quantized LLMs.” arXiv:2305.14314, 2023-05-23. Trains LoRA adapters through a frozen, quantized base. ↩︎
  10. Ilharco, Ribeiro, Wortsman et al. “Editing Models with Task Arithmetic.” arXiv:2212.04089, 2022-12-08. Weight differences from fine-tuning (“task vectors”) can be added and negated. ↩︎
  11. Hinton, Vinyals, Dean. “Distilling the Knowledge in a Neural Network.” arXiv:1503.02531, 2015-03-09. Compresses the knowledge of a large model or ensemble into a smaller one. ↩︎
  12. Frantar, Ashkboos, Hoefler, Alistarh. “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.” arXiv:2210.17323, 2022-10-31. One-shot weight quantization to a few bits per weight with negligible accuracy degradation on its benchmarks. ↩︎
  13. Kwon, Li, Zhuang et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” arXiv:2309.06180, 2023-09-12. The KV cache per request is large and changes size dynamically; vLLM manages and shares it. ↩︎
  14. Holtzman, Buys, Du, Forbes, Choi. “The Curious Case of Neural Text Degeneration.” arXiv:1904.09751, 2019-04-22. Decoding strategies alone change text quality from the same model; introduces nucleus sampling. ↩︎
  15. Alain, Bengio. “Understanding intermediate layers using linear classifier probes.” arXiv:1610.01644, 2016-10-05. Linear classifiers (“probes”) trained independently of the model, on its intermediate layers. ↩︎
  16. Jay Alammar. “The Illustrated Transformer.” jalammar.github.io, 2018-06-27, updated since. https://jalammar.github.io/illustrated-transformer/. A visual walk through the transformer. ↩︎
  17. Zhang, Lipton, Li, Smola. “Dive into Deep Learning.” arXiv:2106.11342, 2021-06-21; web edition at https://d2l.ai/. Open-source book drafted in notebooks with exposition, math and runnable code. ↩︎
  18. Sebastian Raschka. “Build a Large Language Model (From Scratch).” Manning, with the official code repository on GitHub. https://github.com/rasbt/LLMs-from-scratch. Code for developing, pretraining and fine-tuning a GPT-like model. ↩︎
  19. Tatsunori Hashimoto, Percy Liang (instructors). “CS336: Language Modeling from Scratch.” Stanford University course page. https://cs336.stanford.edu/. Students build a language model from tokenizer to alignment and reasoning RL. ↩︎
  20. Nathan Lambert. “Reinforcement Learning from Human Feedback” (the RLHF Book). arXiv:2504.12501, 2025-04-16, revised; free web edition at https://rlhfbook.com/. Instruction tuning, reward models, rejection sampling, RL and direct alignment algorithms. ↩︎