Glossary¶
Every term the book defines, in alphabetical order, with a plain definition and the chapter where it is first explained.
Table A.1: Glossary of terms used in this book.
| Term | Meaning | Chapter |
|---|---|---|
| Autoregressive | Building an answer one piece at a time, feeding each new piece back in before predicting the next one. | 2 |
| Backpropagation | The mathematical process of calculating the gradient for every single weight in the model. | 13 |
| Batching | Grouping multiple training examples together and processing them at the same time. | 12 |
| Block Size | The maximum number of tokens the model can look at at one time (its context window). | 12 |
| Causal Mask | A filter that hides the tokens that come later, so a token sees only itself and the past. | 6 |
| Checkpoint | A file that saves a model at one point in training, with its learned weights and its configuration (how it is built). | 14 |
| Concatenation | Joining several lists of numbers end to end to make one longer list. | 7 |
| Cross-Entropy Loss | A mathematical way to measure how wrong the model's predictions are. Lower is better. | 11 |
| DataLoader | A PyTorch tool that automatically groups single examples into batches and shuffles them. | 12 |
| Dataset | A PyTorch tool that defines how to fetch one training example from the text. | 12 |
| Dropout | A technique that randomly turns off some neurons (small parts of the model) during training, so it cannot simply memorize the data. | 14 |
| Embedding | A lookup table that gives each token ID its own vector. | 5 |
| End to End | A process that runs every step on its own, from raw text to a trained model and its generated text. | 17 |
| Feed-Forward | The step after attention, where each token works on its own to process the information it gathered. | 8 |
| GELU | A smooth curve that replaces negative numbers with near-zero values. | 8 |
| Gradient | The direction we need to move our weights to increase the error. We move in the opposite direction to decrease it. | 13 |
| Greedy Decoding | Always picking the single highest-probability token. | 15 |
| Instruction tuning | An extra round of training, after the kind done in this book, that teaches the model to answer questions and follow instructions. | 17 |
| Key (K) | A list of numbers that says what a token has to offer. | 6 |
| Language Model | A system that predicts the next token in a sequence. | 2 |
| LayerNorm | A step that resets numbers to a safe size so training stays stable. | 8 |
| Learning Rate | How far the optimizer moves the weights on each step. If it is too small, training crawls, and if it is too large, it can overshoot. | 13 |
| LM Head | The model's last step, which turns each token's internal numbers into one score for every character in the vocabulary. | 10 |
| Logits | The raw scores the model outputs for each possible next token. | 10 |
| Matrix Multiplication | A way of multiplying two tensors that turns data from one shape into another. | 3 |
| Multi-Head Attention | Running several attention heads at the same time, so the same text is read in several different ways. | 7 |
| Optimizer | The algorithm that updates the model's weights. We use Adam, a popular and steady choice. | 13 |
| Package Manager | A tool that downloads and installs packages, the ready-made pieces of code a project uses. | 1 |
| Parameters | The numbers inside the model that adjust during training. | 2 |
| Projection | A final step that blends the heads' joined outputs into one list of numbers for each token. | 7 |
| PyTorch | A Python library (ready-made code) that does math on large sets of numbers very quickly. | 3 |
| Query (Q) | A list of numbers that says what a token is looking for. | 6 |
| Repository | A folder of code stored online. | 1 |
| Residual Connection | A shortcut that lets information skip a step, keeping the original signal intact. | 9 |
| RLHF | Reinforcement Learning from Human Feedback, a way to train a model on people's feedback so it answers politely and as people prefer. | 17 |
| Sampling | Choosing the next token randomly, giving higher-probability tokens a better chance to be selected. | 15 |
| Self-Attention | The step where each token looks at the tokens before it and decides how much each one matters to its own meaning. | 6 |
| Shape | How big a tensor is in each direction, such as how many rows and how many columns. | 3 |
| Softmax | A function that turns any numbers into probabilities that sum to 1. | 3 |
| State Dict | A PyTorch dictionary (a lookup table) that pairs each layer of the model with its learned weights. | 14 |
| Temperature | A number the scores (logits) are divided by before they become probabilities. A low temperature plays it safe, and a high one takes risks. | 16 |
| Tensor | A container of numbers, arranged as a list, a table, or a stack of tables. | 3 |
| Terminal | A text-based window where you type commands. | 1 |
| Token | One small piece of text, here a single character. | 2 |
| Tokenizer | A translator that cuts text into tokens and gives each token a number. | 4 |
| Top-k | A limit that lets the model sample only from the k most likely next tokens, where k is a number you choose (40 here). | 16 |
| Transformer Block | A reusable block of code containing attention, feed-forward, and normalization. | 9 |
| Value (V) | A list of numbers holding the information a token passes on when it is picked. | 6 |
| Vector | A list of numbers that works like coordinates on a map, giving a token its place among the others. | 5 |
| Virtual Environment | A separate toolbox for one project, so what you install for it stays out of your other projects. | 1 |