Chapter 11: Causal Language Modeling¶
Figure 11.1: We define the model's goal, predicting the next token.
In Chapter 10 we pieced together the Transformer architecture. Our engine is ready, but an engine needs a task to complete (see Figure 11.1). In this chapter we provide our model with a goal, causal language modeling. Causal language modeling is predicting the next token using only prior tokens. We also learn how to judge its performance.
In this chapter you will:
- Shift the training sequence to create input and target pairs.
- Understand cross-entropy loss as a measure of surprise.
- See how generation is a loop of predicting and appending.
Words to Know
- Logits: The raw scores the model outputs before they are turned into probabilities.
- Softmax: A mathematical function that squashes a list of any numbers into positive percentages that add up to 100%.
- Cross-Entropy Loss: A mathematical way to measure how wrong the model's predictions are. Lower is better.
- Autoregressive: A process that uses its own past outputs as inputs for its next step.
Theory¶
The Training Trick: One Pass, Many Examples¶
So how do we train a model to predict the next token? Well, we could give it one token and have it predict what the next one is. Then we'd see if it's right and give it two tokens. This would take way too long.
With causal language modeling we can get T training examples all at once from one forward pass through our sequence of length T.
Figure 11.2: One sequence provides multiple training examples simultaneously.
The model predicts over the entire sequence (Figure 11.2). Since there is a causal mask (described in Chapter 6), position 2 has no visibility of position 3, and each position can only see prior context. Thus the model can make a legitimate guess for the next token at each position.
Input and Target: The Shifted Pair¶
To implement this efficiently, we take our sequence of text and create two slightly different copies:
- Input (
x): The sequence missing its last token. - Target (
y): The sequence missing its first token.
Basically this means that the character at each position i in the input, $x[i]$, is trying to predict the symbol at position i in the target, $y[i]$. So for our example sequence "HELLO", x would be "HELL" and y would be "ELLO". At position 0, "H" predicts "E", and at position 1, "E" (given "H" as previous context) predicts "L", etc.
Logits and Probabilities¶
When our model makes predictions, it outputs raw numbers called logits, which are neither a single character nor a clean percentage.
Figure 11.3: Softmax converts raw scores into valid percentages.
Logits can be any real number. Negative. Positive. Small. Large. To interpret them, we run them through a function called softmax (Figure 11.3). Softmax rescales all the logits so they are positive and add up to exactly 1.0 (or 100%), resulting in a probability distribution over our 65 possible characters.
Measuring Success: Cross-Entropy Loss¶
Now that we can calculate probabilities, we need some way to evaluate how our model is doing. This is where cross-entropy loss comes into play.
You can think of loss as how surprised the model is that a given character is next. If the true next character is "A" and the model gave "A" a 99% probability then it's not very surprised so the loss would be low. If it gave "A" a 1% probability then it's very surprised and the loss would be high.
If our model was totally random it would predict all of our 65 characters with an equal chance of about 1.5%. Using some math, we can find that the loss for this scenario is about 4.17. We want to minimize this during training.
With that we've completed our Next-Token Scores section of our map. Now that our model can predict and determine how incorrect it is we can start training.
Code¶
Figure 11.4: The model produces logits for the input, which are compared to the target to calculate loss.
The code below implements this (Figure 11.4).
| src/ch10_causal_lm.py (excerpt) | |
|---|---|
And how to run the full script:
$ python src/ch10_causal_lm.py
--- 1. Constructing input/target pairs ---
Sequence: [20, 17, 30, 30, 33, 1, 35, 53, 56, 30]
Input x: [20, 17, 30, 30, 33, 1, 35, 53, 56]
Target y: [17, 30, 30, 33, 1, 35, 53, 56, 30]
At each position i, x[i] predicts y[i].
--- 2. Computing cross-entropy loss ---
Logits shape : torch.Size([4, 20, 65])
Targets shape: torch.Size([4, 20])
Loss (random model): 4.3070
Expected loss for random: 4.1744
--- 3. Text generation ---
We extend a starting sequence one token at a time.
Generated IDs (first 10): [0, 54, 34, 5, 55, 18, 63, 52, 43, 10]
Output shape: torch.Size([1, 51])
After training (Chapter 13), this will produce real text!
What just happened:
- Lines 1 and 2 slice a short sequence of 10 tokens into an input
xof length 9 and a targetyof length 9 to demonstrate input/target pairs. - Line 4 passes a random batch of 4 sequences, 20 tokens each, into the untrained model to get logits for the loss computation step.
- Lines 7 to 10 calculate the cross-entropy loss, which is 4.3070, close to our expected random guessing loss of 4.1744.
- The output shows we generated 50 tokens. The machinery works, but the weights are still random, so the characters are too.
Shape Check:
The shapes below are from the loss computation step. Note that instead of using a single sequence of length 10 here we use 4 sequences of length 20. Shapes at each step are shown in Table 11.1.
Table 11.1: Tensor shapes for the cross-entropy loss calculation.
| Variable | Shape | Meaning |
|---|---|---|
token_ids |
[4, 20] |
A batch of 4 sequences, 20 tokens each. |
logits |
[4, 20, 65] |
A score for each of the 65 possible next characters, at every position. |
loss |
[] |
A single number, the average surprise. |
Try It¶
To see how confidence affects the loss, we can simulate different prediction scenarios manually.
Lines 14 and 15 calculate the loss when the model is confident and correct, which gives a much lower loss.
$ python src/examples/ch11_loss_demo.py
Scenario 1: Guessing blindly (50% / 50%)
Loss: 0.6931
Scenario 2: Confident and right (88% for 'A')
Loss: 0.1269
Scenario 3: Confident and wrong (88% for 'B')
Loss: 2.1269
Try It
Open src/examples/ch11_loss_demo.py and change logits_right to [[5.0, 0.0]] to make the model even more confident. Run the script and see how close to zero the loss gets.
In Business
Imagine you are trying to train an assistant to write text in your organization's house style. Causal language modeling is how you teach it to write like your organization does. It reads through thousands of past emails and reports (target sequences) and figures out which words are likely to come after other words in your specific context. The loss lets you know how similar its predictions are to your historical data.
Watch Out
PyTorch's F.cross_entropy() function expects the logits to already be flattened, with one row of scores for each token. F.cross_entropy() wants a 2D table of shape [Total Tokens, VOCAB_SIZE], not the model's 3D stack of tables [BATCH, TIME, VOCAB_SIZE]. That's why there's a logits.view(B * T_len, config.vocab_size) in that code. Forgetting that view is a common bug.
Key Takeaways¶
- A single forward pass on a sequence of length
TprovidesTseparate training examples, which makes training efficient. - The target sequence is the input sequence shifted one position into the future.
- The model outputs raw logits, which softmax converts into probabilities.
- Cross-entropy loss measures how "surprised" the model is by the correct answer. We want this number to be as low as possible.
- An untrained model guessing among 65 characters will have a loss of approximately 4.17.
Check Your Understanding¶
- If your sequence is "DATA", what is the input sequence
xand the target sequencey? - Why do we need softmax before we can interpret the model's output as percentages?
- If a model is perfectly confident and perfectly correct, what should its cross-entropy loss be?
Assignment¶
Assignment 11: One number, two lists
About 15 minutes. You change one number and write no new code. Work in this chapter's Colab notebook or on your own laptop.
- Run it. Run
src/ch10_causal_lm.pyas it is. Copy the first three lines of numbers it prints:Sequence,Input xandTarget y. - Change one number. Find the line
example_ids = torch.tensor([20, 17, 30, 30, 33, 1, 35, 53, 56, 30]). These ten numbers are the token IDs of one short sequence. Change the1in the middle to50and run the script again. - Compare. Your new number shows up in
Input xand inTarget y. Is it in the same place in both lists? Which number ofInput xsits directly above your new number inTarget y? Which end of the sequence is missing fromInput x, and which end is missing fromTarget y? - Explain it. In two or three sentences a manager could follow, say how a single piece of text supplies both the questions and the correct answers for training.
Hand in: both sets of three lines, your answers to step 3, and your sentences.
Further Reading¶
Train once, reuse everywhere. Labeled examples are expensive, and every new task used to need its own pile of them. BERT trained one Transformer on ordinary text by hiding words and asking it to fill the blanks, then adapted that single model to many tasks with a small amount of labeled data each. It reads in both directions at once, which the causal mask in Chapter 11 forbids. That mask is the difference between a model that fills blanks and one that writes forward.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding (arXiv:1810.04805). arXiv. https://doi.org/10.48550/arXiv.1810.04805
Next up: Chapter 12: Dataset and DataLoader