Glossary¶
Every term the book defines, in alphabetical order, with a plain definition and the chapter where it is first explained.
Table A.1: Glossary of terms used in this book.
| Term | Meaning | Chapter |
|---|---|---|
| Autoregressive | Using your past outputs as inputs for your next step. | 2 |
| Backpropagation | The mathematical process of calculating the gradient for every single weight in the model. | 13 |
| Batching | Grouping multiple training examples together and processing them at the same time. | 12 |
| Block Size | The maximum number of tokens the model can look at at one time (its context window). | 12 |
| Causal Mask | A filter that prevents tokens from seeing future tokens. | 6 |
| Checkpoint | A file containing the saved state (weights and configuration) of a model at a specific point in training. | 14 |
| Concatenation | Joining multiple vectors end-to-end to form a longer vector. | 7 |
| Cross-Entropy Loss | A mathematical way to measure how wrong the model's predictions are. Lower is better. | 11 |
| DataLoader | A PyTorch utility that automatically groups individual examples into batches and shuffles them. | 12 |
| Dataset | A PyTorch class that defines how to retrieve a single training example. | 12 |
| Dropout | A technique that randomly turns off some neurons during training to prevent memorizing the data. | 14 |
| Embedding | A lookup table that maps a token ID to its vector. | 5 |
| End to End | A process that takes raw input (text) and goes through every necessary step to produce the final output (a trained model and generated text) without manual intervention. | 17 |
| Feed-Forward | A small neural network that processes each token independently. | 8 |
| GELU | A smooth curve that replaces negative numbers with near-zero values. | 8 |
| Gradient | The direction we need to move our weights to increase the error. We move in the opposite direction to decrease it. | 13 |
| Greedy Decoding | Always picking the single highest-probability token. | 15 |
| Instruction tuning | A post-training step that teaches the model to answer questions and follow instructions. | 17 |
| Key (K) | A vector representing what a token contains. | 6 |
| Language Model | A system that predicts the next token in a sequence. | 2 |
| LayerNorm | A step that resets numbers to a safe size so training stays stable. | 8 |
| LM Head | The final linear layer that maps internal numbers back to vocabulary words. | 10 |
| Logits | The raw scores the model outputs for each possible next token. | 10 |
| Matrix Multiplication | Combining two tensors to transform data. | 3 |
| Multi-Head Attention | Running several attention operations simultaneously. | 7 |
| Optimizer | The algorithm that updates the model's weights. We use Adam, a popular and steady choice. | 13 |
| Package Manager | A tool that downloads and installs code libraries. | 1 |
| Parameters | The numbers inside the model that adjust during training. | 2 |
| Projection | A linear layer that blends the concatenated outputs. | 7 |
| PyTorch | A library for doing math on large arrays of numbers very quickly. | 3 |
| Query (Q) | A vector representing what a token is looking for. | 6 |
| Repository | A folder of code stored online. | 1 |
| Residual Connection | A shortcut that lets information skip a step, keeping the original signal intact. | 9 |
| RLHF | Reinforcement Learning from Human Feedback, a method to train models to behave politely and align with human preferences. | 17 |
| Sampling | Choosing the next token randomly, giving higher-probability tokens a better chance to be selected. | 15 |
| Self-Attention | A mechanism where tokens evaluate every other token in the sequence to gather context. | 6 |
| Shape | The dimensions of a tensor (like rows and columns). | 3 |
| Softmax | A function that turns any numbers into probabilities that sum to 1. | 3 |
| State Dict | A PyTorch dictionary that maps each layer of the model to its learned weights. | 14 |
| Temperature | A number that scales the logits before they are turned into probabilities, controlling the randomness of the output. | 16 |
| Tensor | A multi-dimensional array of numbers. | 3 |
| Terminal | A text-based window where you type commands. | 1 |
| Token | One small piece of text, here a single character. | 2 |
| Tokenizer | A function that turns text into tokens and then into numbers. | 4 |
| Top-k | A limit that restricts the model to only sample from the k most likely next tokens, ignoring all others. | 16 |
| Transformer Block | A reusable block of code containing attention, feed-forward, and normalization. | 9 |
| Value (V) | A vector holding the actual information to be shared. | 6 |
| Vector | A list of numbers that acts as a coordinate in a high-dimensional space. | 5 |
| Virtual Environment | An isolated toolbox for a single project's packages. | 1 |