Skip to content
Module 2 ยท Building the Model
Chapter 11 of 17
Run it in your browser

Chapter 11: Causal Language Modeling

The map with Next-Token Scores highlighted Figure 11.1: We define the model's goal, predicting the next token.

In Chapter 10 we pieced together the Transformer architecture. Our engine is ready, but an engine needs a task to complete (see Figure 11.1). In this chapter we provide our model with a goal, causal language modeling. Causal language modeling is predicting the next token using only prior tokens. We also learn how to judge its performance.

In this chapter you will:

  • Shift the training sequence to create input and target pairs.
  • Understand cross-entropy loss as a measure of surprise.
  • See how generation is a loop of predicting and appending.

Words to Know

  • Logits: The raw scores the model outputs before they are turned into probabilities.
  • Softmax: A mathematical function that squashes a list of any numbers into positive percentages that add up to 100%.
  • Cross-Entropy Loss: A mathematical way to measure how wrong the model's predictions are. Lower is better.
  • Autoregressive: A process that uses its own past outputs as inputs for its next step.

Theory

The Training Trick: One Pass, Many Examples

So how do we train a model to predict the next token? Well, we could give it one token and have it predict what the next one is. Then we'd see if it's right and give it two tokens. This would take way too long.

With causal language modeling we can get T training examples all at once from one forward pass through our sequence of length T.

Sequence shifted to create targets Figure 11.2: One sequence provides multiple training examples simultaneously.

The model predicts over the entire sequence (Figure 11.2). Since there is a causal mask (described in Chapter 6), position 2 has no visibility of position 3, and each position can only see prior context. Thus the model can make a legitimate guess for the next token at each position.

Input and Target: The Shifted Pair

To implement this efficiently, we take our sequence of text and create two slightly different copies:

  1. Input (x): The sequence missing its last token.
  2. Target (y): The sequence missing its first token.

Basically this means that the character at each position i in the input, $x[i]$, is trying to predict the symbol at position i in the target, $y[i]$. So for our example sequence "HELLO", x would be "HELL" and y would be "ELLO". At position 0, "H" predicts "E", and at position 1, "E" (given "H" as previous context) predicts "L", etc.

Logits and Probabilities

When our model makes predictions, it outputs raw numbers called logits, which are neither a single character nor a clean percentage.

Logits turned into probabilities and loss Figure 11.3: Softmax converts raw scores into valid percentages.

Logits can be any real number. Negative. Positive. Small. Large. To interpret them, we run them through a function called softmax (Figure 11.3). Softmax rescales all the logits so they are positive and add up to exactly 1.0 (or 100%), resulting in a probability distribution over our 65 possible characters.

Measuring Success: Cross-Entropy Loss

Now that we can calculate probabilities, we need some way to evaluate how our model is doing. This is where cross-entropy loss comes into play.

You can think of loss as how surprised the model is that a given character is next. If the true next character is "A" and the model gave "A" a 99% probability then it's not very surprised so the loss would be low. If it gave "A" a 1% probability then it's very surprised and the loss would be high.

If our model was totally random it would predict all of our 65 characters with an equal chance of about 1.5%. Using some math, we can find that the loss for this scenario is about 4.17. We want to minimize this during training.

With that we've completed our Next-Token Scores section of our map. Now that our model can predict and determine how incorrect it is we can start training.

Code

Code flow: logits and targets into cross-entropy Figure 11.4: The model produces logits for the input, which are compared to the target to calculate loss.

The code below implements this (Figure 11.4).

src/ch10_causal_lm.py (excerpt)
    x = example_ids[:-1]
    y = example_ids[1:]

    logits = model(token_ids)

    # Calculate loss by comparing predictions to actual targets
    loss = F.cross_entropy(
        logits.view(B * T_len, config.vocab_size),
        targets.view(B * T_len)
    )

And how to run the full script:

Terminal
$ python src/ch10_causal_lm.py
--- 1. Constructing input/target pairs ---
Sequence: [20, 17, 30, 30, 33, 1, 35, 53, 56, 30]
Input  x: [20, 17, 30, 30, 33, 1, 35, 53, 56]
Target y: [17, 30, 30, 33, 1, 35, 53, 56, 30]
At each position i, x[i] predicts y[i].

--- 2. Computing cross-entropy loss ---
Logits shape : torch.Size([4, 20, 65])
Targets shape: torch.Size([4, 20])
Loss (random model): 4.3070
Expected loss for random: 4.1744

--- 3. Text generation ---
We extend a starting sequence one token at a time.

Generated IDs (first 10): [0, 54, 34, 5, 55, 18, 63, 52, 43, 10]
Output shape: torch.Size([1, 51])
After training (Chapter 13), this will produce real text!

What just happened:

  1. Lines 1 and 2 slice a short sequence of 10 tokens into an input x of length 9 and a target y of length 9 to demonstrate input/target pairs.
  2. Line 4 passes a random batch of 4 sequences, 20 tokens each, into the untrained model to get logits for the loss computation step.
  3. Lines 7 to 10 calculate the cross-entropy loss, which is 4.3070, close to our expected random guessing loss of 4.1744.
  4. The output shows we generated 50 tokens. The machinery works, but the weights are still random, so the characters are too.

Shape Check:

The shapes below are from the loss computation step. Note that instead of using a single sequence of length 10 here we use 4 sequences of length 20. Shapes at each step are shown in Table 11.1.

Table 11.1: Tensor shapes for the cross-entropy loss calculation.

Variable Shape Meaning
token_ids [4, 20] A batch of 4 sequences, 20 tokens each.
logits [4, 20, 65] A score for each of the 65 possible next characters, at every position.
loss [] A single number, the average surprise.

Try It

To see how confidence affects the loss, we can simulate different prediction scenarios manually.

src/examples/ch11_loss_demo.py
"""Calculate cross-entropy loss for different confidence levels."""
import torch
import torch.nn.functional as F

# Two possible words: 'A' (index 0) or 'B' (index 1)
target = torch.tensor([0]) # The correct answer is 'A'

print("Scenario 1: Guessing blindly (50% / 50%)")
logits_blind = torch.tensor([[0.0, 0.0]])
loss_blind = F.cross_entropy(logits_blind, target)
print(f"Loss: {loss_blind.item():.4f}")

print("\nScenario 2: Confident and right (88% for 'A')")
logits_right = torch.tensor([[2.0, 0.0]])
loss_right = F.cross_entropy(logits_right, target)
print(f"Loss: {loss_right.item():.4f}")

print("\nScenario 3: Confident and wrong (88% for 'B')")
logits_wrong = torch.tensor([[0.0, 2.0]])
loss_wrong = F.cross_entropy(logits_wrong, target)
print(f"Loss: {loss_wrong.item():.4f}")

Lines 14 and 15 calculate the loss when the model is confident and correct, which gives a much lower loss.

Terminal
$ python src/examples/ch11_loss_demo.py
Scenario 1: Guessing blindly (50% / 50%)
Loss: 0.6931

Scenario 2: Confident and right (88% for 'A')
Loss: 0.1269

Scenario 3: Confident and wrong (88% for 'B')
Loss: 2.1269

Try It

Open src/examples/ch11_loss_demo.py and change logits_right to [[5.0, 0.0]] to make the model even more confident. Run the script and see how close to zero the loss gets.

In Business

Imagine you are trying to train an assistant to write text in your organization's house style. Causal language modeling is how you teach it to write like your organization does. It reads through thousands of past emails and reports (target sequences) and figures out which words are likely to come after other words in your specific context. The loss lets you know how similar its predictions are to your historical data.

Watch Out

PyTorch's F.cross_entropy() function expects the logits to already be flattened, with one row of scores for each token. F.cross_entropy() wants a 2D table of shape [Total Tokens, VOCAB_SIZE], not the model's 3D stack of tables [BATCH, TIME, VOCAB_SIZE]. That's why there's a logits.view(B * T_len, config.vocab_size) in that code. Forgetting that view is a common bug.

Key Takeaways

  • A single forward pass on a sequence of length T provides T separate training examples, which makes training efficient.
  • The target sequence is the input sequence shifted one position into the future.
  • The model outputs raw logits, which softmax converts into probabilities.
  • Cross-entropy loss measures how "surprised" the model is by the correct answer. We want this number to be as low as possible.
  • An untrained model guessing among 65 characters will have a loss of approximately 4.17.

Check Your Understanding

  1. If your sequence is "DATA", what is the input sequence x and the target sequence y?
  2. Why do we need softmax before we can interpret the model's output as percentages?
  3. If a model is perfectly confident and perfectly correct, what should its cross-entropy loss be?

Assignment

Assignment 11: One number, two lists

About 15 minutes. You change one number and write no new code. Work in this chapter's Colab notebook or on your own laptop.

  1. Run it. Run src/ch10_causal_lm.py as it is. Copy the first three lines of numbers it prints: Sequence, Input x and Target y.
  2. Change one number. Find the line example_ids = torch.tensor([20, 17, 30, 30, 33, 1, 35, 53, 56, 30]). These ten numbers are the token IDs of one short sequence. Change the 1 in the middle to 50 and run the script again.
  3. Compare. Your new number shows up in Input x and in Target y. Is it in the same place in both lists? Which number of Input x sits directly above your new number in Target y? Which end of the sequence is missing from Input x, and which end is missing from Target y?
  4. Explain it. In two or three sentences a manager could follow, say how a single piece of text supplies both the questions and the correct answers for training.

Hand in: both sets of three lines, your answers to step 3, and your sentences.

Further Reading

Train once, reuse everywhere. Labeled examples are expensive, and every new task used to need its own pile of them. BERT trained one Transformer on ordinary text by hiding words and asking it to fill the blanks, then adapted that single model to many tasks with a small amount of labeled data each. It reads in both directions at once, which the causal mask in Chapter 11 forbids. That mask is the difference between a model that fills blanks and one that writes forward.

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding (arXiv:1810.04805). arXiv. https://doi.org/10.48550/arXiv.1810.04805

Next up: Chapter 12: Dataset and DataLoader