Introduction: Five Phases, One Loop¶

Name: Building an LLM from Zero: Look Inside the Black Box
Author: Truong (Jack) Luu

Before you write a line of code, it helps to know where the thing you are about to build sits. Artificial intelligence is not one invention. It is five waves, each of which kept what worked and changed one idea. The model in this book belongs to the fourth wave, and it inherits almost everything from the second.

The five phases of artificial intelligence on a timeline Figure I.1: Five waves of artificial intelligence. From the second wave onward, every one of them learns the same way.

Where this book sits¶

The first phase, roughly 1950 to 1985, had people write down every rule by hand. If this, then that, thousands of times over. ELIZA, released in 1966, imitated a therapist by spotting a keyword in your sentence and turning it back into a question. It knew nothing, and people confided in it anyway. That is the oldest lesson in the field: we read understanding into anything that answers in fluent sentences.

The second phase, roughly 1985 to 2011, gave up on writing the rule. Show the machine ten thousand emails already labeled spam or not spam, and let it find the pattern itself. This is where the arithmetic of this book begins: turn everything into numbers, guess, measure how wrong the guess was, and nudge the numbers. Nothing since has changed that.

The third phase, 2012 to 2016, changed the scale and nothing else. Stack more layers, use far more data, run it on graphics cards. The result that settled the argument was an image model in 2012 that won its competition by a margin nobody could dispute, using the same loop.

The fourth phase, 2017 to 2023, changed the goal. Earlier models sorted an input into a category. These produce the next piece of it. Two ideas made that work: give every word a position in a space of numbers so that similar words sit near each other, and let the model weigh how much every other word matters before it commits to the next one. That second idea arrived in 2017 and is called attention. The rest of this book is those two ideas, built small enough to read.

The fifth phase, 2023 onward, barely changed the model at all. What changed is what we let it do: give it a goal and a set of tools, and let it plan a step, use a tool, check the result, and plan the next one. The engine underneath is still a next-word predictor. It is the one you are about to build.

Table I.1: The five phases, and what actually changed at each one.

Phase	Roughly	What changed	What the machine does
Rules	1950 to 1985	People write every rule	Matches patterns, follows conditions
Machine learning	1985 to 2011	Learn the rule from examples	Guess, measure the error, adjust
Deep learning	2012 to 2016	Far more layers and data	The same loop, much bigger
Generative AI	2017 to 2023	Produce the next piece, not a label	Embeddings and attention, same loop
Agentic AI	2023 onward	Give the model tools and a goal	The same model, inside a system

One loop, running for forty years¶

Strip away the vocabulary and every phase from the second onward runs the same three steps.

The training loop: guess, measure the error, adjust the weights Figure I.2: The loop that every model in this book runs, millions of times.

The machine guesses. Something measures how far the guess landed from the right answer. Something else adjusts the numbers so the next guess lands closer. Then it does it again, millions of times. You will build all three parts: the guess in Chapter 10, the measurement in Chapter 11, and the adjustment in Chapter 13.

Read that loop again and notice what is missing. No step in it asks whether the answer is true. The model is rewarded for producing text that fits the pattern of its training data, and a convincing invention fits that pattern exactly as well as a fact does. This is why language models state false things in clean, confident prose. It is not a defect that better engineering will remove. It is what the loop optimizes for, and you will see it directly in Chapter 17 when your own model writes Shakespeare that no one ever wrote.

A layer is a stack of small regressions¶

If you have fitted a line through a scatter plot, you have already trained a model. You picked a slope and an intercept, measured how far each point missed the line, and chose the values that made the misses smallest. That is the whole idea: a guess, a ruler, an adjustment.

A neural network is that, repeated. Each unit inside a layer multiplies its inputs by its own weights, adds them up, and adds one more number to shift the result. A slope for every input and an intercept, which is a regression. A layer runs a few hundred of those at once, as a single multiplication of two tables of numbers, and a network stacks the layers. Nothing in this book is harder than that; there is only a great deal of it. Chapter 3 covers the tables of numbers, and Chapter 8 builds the layer.

The four parts you are about to build¶

Follow one guess from a chatbot and you pass through four pieces, in this order.

The dictionary. Words are turned into lists of numbers, positioned so that words used alike sit near each other. This is the embedding, and it is Chapter 5.
The structure. Layers of numbers, each doing its small job and handing the result to the next. Attention lives here, in Chapters 6 through 10.
The ruler. A single number saying how far the guess landed from the right answer. This is the loss, and it is Chapter 11.
The messenger. The error is carried back through every layer, telling each weight which way to move. This is backpropagation, and it runs the training in Chapter 13.

Why build it by hand¶

A calculator is a fine thing if you already know arithmetic. If you never learned it, you will not notice when the calculator gives you a wrong answer, because you have nothing to check it against.

That is the argument for this book. You can already get a language model to write code, draft a memo, or explain itself. What you cannot do, without having built one, is tell when it is wrong, and say why. By the end you will have written every part of a working model, trained it on your own laptop, watched its error fall, and read the text it produces. After that, the polished paragraph on your screen stops being magic and becomes a machine you understand well enough to doubt.