Teach Your Computer to Write
Build and Train an LLM from Zero
Also as PDF or EPUB. The code and exercises are on GitHub, and every chapter has a notebook you can run in Colab.
Book text CC BY-NC-ND 4.0 · code MIT licensed
The book used to be called Building an LLM from Zero. The text, the code and the address are the same.
Why This Book?
There are already a lot of good resources for learning about AI and large language models. This one tries to find a balance: enough theory to understand what you are building, paired with real, runnable code you can execute on the machine in front of you.
The entire book was written and tested on a five-year-old ThinkPad T14 Gen 1 with no GPU. Training completes in about 20-30 minutes. The subject here is the machine itself: what the numbers are doing, why the thing invents facts, and what it can and cannot learn. Prompting strategy, deployment and governance are real subjects, and separate ones; this book stays inside the model.
About the Book
A step-by-step tutorial for building a GPT-style language model entirely from scratch using Python and PyTorch, written for business students. By the end, you will have trained a model that generates Shakespeare-like text.
Every chapter has the same parts in the same order: the idea in plain language with a picture and an everyday comparison, a few numbered lines of code, what the computer printed when the code ran, one small change to try, what the idea means at work, and a 15-minute assignment.
Who Is This For?
- Business students. You use AI tools every week and want to know what is inside them. Every idea comes with a picture, an everyday comparison and an example from work.
- The instructors who teach them. Every chapter has a lecture deck timed for one class, a 15-minute assignment and a notebook that runs in a browser.
- Managers and analysts. You buy, approve or supervise AI tools. After this book you can ask a vendor better questions, because you have built the thing they are selling.
You do not need calculus, linear algebra or any machine learning background. Every script in the book is already written. You run it, read what each line does, and change one number at a time. If you have seen a little Python, the code will read faster.
For instructors: every chapter has a lecture deck timed for one class, with speaker notes, a 15-minute assignment and a notebook that runs in a browser. The decks are on the book's Lecture Slides page, and Teach With This Book shows how to run a course with them. The exercises and assignments are also in the code repository.
Quick Facts
| Chapters | 17 |
| Modules | 3 |
| Slide decks | 17 |
| Notebooks | 17 |
| Assignments | 17, 15 minutes each |
| Model size | 824,832 parameters |
| Dataset | Tiny Shakespeare |
| Training time | ~20-30 min (CPU) |
| Framework | PyTorch (CPU only) |
| License | MIT / CC BY-NC-ND 4.0 |
Table of Contents
Front Matter
| Preface: Why I Wrote This Book | |
| Introduction: Five Phases, One Loop |
Module 1: Foundations
| Ch 01 | Environment Setup |
| Ch 02 | What Is an LLM? |
| Ch 03 | Tensors and PyTorch |
| Ch 04 | Tokenization |
| Ch 05 | Embeddings |
Module 2: Building the Model
| Ch 06 | Self-Attention |
| Ch 07 | Multi-Head Attention |
| Ch 08 | Feed-Forward and Layer Norm |
| Ch 09 | The Transformer Block |
| Ch 10 | The Full GPT Architecture |
| Ch 11 | Causal Language Modeling |
Module 3: Training and Generation
| Ch 12 | Dataset and DataLoader |
| Ch 13 | The Training Loop |
| Ch 14 | Checkpointing |
| Ch 15 | Greedy and Sampling |
| Ch 16 | Temperature and Top-k |
| Ch 17 | Putting It All Together |
Appendices
| A | Glossary |
| B | Ten Papers That Built Generative AI |
ROMEO:
Praviour soul to shall that that are the and not,
Real output from the 824,832-parameter model you build, after about 25 minutes of training on a laptop CPU. It has learned the character names, the line breaks and the shape of the words; it has not learned to mean anything. Chapter 17 explains why.