Module 1 · Foundations
Chapter 2 of 17 Run it in your browser

Chapter 2: What Is an LLM?¶

Name: Teach Your Computer to Write: Build and Train an LLM from Zero
Author: Truong (Jack) Luu

The map highlights the whole chain as an overview Figure 2.1: What the whole system does, before we zoom in.

With our environment set up in the previous chapter, we turn to the model itself (Figure 2.1). A Language Model (LM) seems like a magic box that understands what you type, but under the hood it performs a specific, simple task.

In this chapter you will:

Learn the single rule that drives all language models.
See how models generate long answers one step at a time.
Understand how reading data replaces human teachers.

Words to Know

Token: one small piece of text, here a single character.
Language Model: a system that predicts the next token in a sequence.
Autoregressive: building an answer one piece at a time, feeding each new piece back in before predicting the next one.
Parameters: the numbers inside the model that adjust during training.

Theory: The Next-Token Engine¶

Imagine you are texting a friend and you type: "I am so hungry, I could eat a". Before you finish, your phone suggests: horse. Or pizza.

A text box with incomplete text points to a prediction box showing horse Figure 2.2: The core task of a language model is guessing what comes next.

Your phone is doing next-word prediction (Figure 2.2), guessing what word is most likely to come next based on everything you typed so far. A Large Language Model (LLM) does the same thing. Given a sequence of tokens, it predicts what comes next. That is the whole idea, and everything else in this book builds a machine that can do this well.

How It Talks Back¶

If the model only predicts the next token, how does it write an essay? It predicts one token, adds it to the text, and predicts again, a loop called autoregressive generation.

You type: "What is the capital of France?"
The model predicts the next token: "Paris".
That output is added to the input: "What is the capital of France? Paris".
It predicts the next token: " is".
And repeats until it decides to stop.

A diagram showing the output text being fed back into the model as input Figure 2.3: In autoregressive generation, each predicted output loops back to become the next input.

The model never thinks about the whole answer at once (Figure 2.3). It keeps predicting the next piece, over and over, building the sentence step by step.

What is a Token?¶

A token is the smallest unit the model works with, which could be a whole word, a piece of a word, or a single character.

In this book, we use character-level tokens, so every letter, space, and punctuation mark is one token. We use characters because the vocabulary is tiny (only 65 characters in our Shakespeare data) and you can understand it instantly. Real models use pieces of words, called subwords, instead of single characters, but the math is the same.

The Power of Self-Supervised Learning¶

To make a model "Large", we give it millions of parameters, which are the numbers inside the model that act like dials. Training turns these dials until the model produces good output.

But how does it learn without a teacher grading its work? Through self-supervised learning.

With language, the text itself is the answer key, so if your training text is "Hello world", the model gets these practice questions automatically:

See H, predict e.
See He, predict l.
See Hel, predict l.

A diagram showing text subsets predicting the next character Figure 2.4: The text itself provides thousands of built-in practice questions.

Because no human labels are needed, you can train a model on millions of pages of raw text (Figure 2.4). The model reads the data and learns the patterns.

Try It: The Numbers Game¶

Is it only predicting characters? We can prove it by looking at real data. If we see the letters "the ", what is the most likely next character in Shakespeare?

We wrote a tiny script to scan our training data and count every character that follows "the ".

Terminal

$ python src/examples/ch02_next_char.py
What follows 'the ' in Shakespeare?
's': 573 times
'w': 459 times
'c': 448 times
'p': 426 times
'm': 376 times

This is what the model learns to do, but instead of counting by hand, it uses math to work out how likely each next character is.

In Business

If a company builds an AI assistant to draft emails in its own house style, the concept is the same. The model trains on the company's archive of past writing. By learning what character or word typically follows another in that archive, the assistant learns the company's voice and terminology automatically.

Watch Out

It is easy to think the model "understands" the text, but it does not. It is a math engine calculating the most probable next token based on patterns in its training data.

Key Takeaways¶

A language model is a system that predicts the next token.
Autoregressive generation means the model uses its own outputs as inputs for the next step.
Tokens are the basic pieces of text, like characters or words.
Self-supervised learning uses the raw text itself as the answer key.
The whole system revolves around next-token prediction, and the first step in our map is the math and tools we use to build it.

Check Your Understanding¶

What does a language model predict?
Why is it called "autoregressive"?
How does self-supervised learning differ from having a teacher grade the work?

Assignment¶

Assignment 2: New word, new guess

About 15 minutes. You change one word and write no new code. Work in this chapter's Colab notebook or on your own laptop.

Run it. Run src/examples/ch02_next_char.py as it is. Copy the five letters and their counts.
Change one word. Find the line prefix = "the ". The prefix is the piece of text the script looks for. Change the to my and keep the space before the closing quotation mark, so the line reads prefix = "my ". Run the script again.
Compare. Which letter comes first now? Which letters are in both lists? Name one word Shakespeare could write after my that starts with the new first letter.
Explain it. In two or three sentences a manager could follow, say why the most likely next letter changed when you changed one word.

Hand in: both lists of five letters and counts, your answers to step 3, and your sentences.