08 Module 2 · Building the Model
Chapter 8 of 17 Run it in your browser

Chapter 8: Feed-Forward and Norms

Figure 8.1: We just built attention; now we process that context.

While attention (from Chapter 7) lets a token gather information from its neighbors, gathering information is only half the job. The token still needs to process that information and decide what to do with it. In this chapter you will:

• Build a feed-forward network to process context.
• Use a GELU curve to help the model learn smoothly.
• Add Layer Normalization to keep the math stable.

Words to Know

• Feed-Forward: The step after attention, where each token works on its own to process the information it gathered.
• GELU: A smooth curve that replaces negative numbers with near-zero values.
• LayerNorm: A step that resets numbers to a safe size so training stays stable.

Theory

Thinking on Your Own

Attention is like listening to your team explain their ideas, while feed-forward is going back to your desk and thinking it over yourself.

Figure 8.2: The network expands to think, then compresses back to an answer.

The feed-forward layer is a small neural network applied to each token by itself (Figure 8.2). In our model, a token is a list of 128 numbers, which the network first expands to 512 numbers. This 4x expansion gives the model a wide workspace to spread out its calculations, and after processing the network compresses the answer back down to 128 numbers.

The Smooth GELU Curve

Inside the feed-forward network, we apply a mathematical curve called GELU.

Figure 8.3: GELU replaces a hard switch with a smooth dimmer.

A neural network needs a curve that bends (a non-linear curve) to learn complex patterns. An older choice is ReLU, which acts like a hard switch: negative numbers become exactly zero, while positive numbers stay the same.

GELU is a gradual dimmer switch (Figure 8.3). Strongly negative numbers become nearly zero, numbers near zero are gently reduced, and positive numbers pass through almost unchanged.

During training, we use math (calculus) to work out how to adjust the model's weights. A hard corner like ReLU can produce sudden jumps that make that math unstable, while GELU's smooth curve makes training much more stable and is what most modern models use.

Keeping Numbers Safe with LayerNorm

As numbers flow through many layers of a neural network, they can grow very large or shrink very small. Like compound interest, small multiplications add up quickly, and extremely large or small numbers break the training math.

Layer Normalization (LayerNorm) fixes this. It re-centers and re-scales the numbers for each token, so that after LayerNorm the values average to 0 and have a standard spread of 1. The run below prints these as mean and std.

Figure 8.4: LayerNorm centers messy numbers back around zero.

Think of it like resetting a runner's stopwatch after every lap (Figure 8.4). It does not change who is winning, but it keeps the numbers on the screen small and easy to read. In modern models, we apply LayerNorm before each major step, which gives the attention and feed-forward layers well-behaved numbers to work with.

On the map in Figure 8.1, the feed-forward network processes the context gathered by attention, and LayerNorm keeps the math stable before we pass the numbers to the next stage.

Code

class FeedForward(nn.Module):
    def __init__(self):
        super().__init__()
        C = config.n_embd
        # A small neural network applied to each token independently
        self.net = nn.Sequential(
            nn.Linear(C, 4 * C),
            nn.GELU(),
            nn.Linear(4 * C, C),
            nn.Dropout(config.dropout),
        )

    def forward(self, x):
        return self.net(x)

Run the script to see the feed-forward layer and LayerNorm in action.

Terminal

$ python src/ch07_feedforward.py
--- GELU activation ---
GELU is like ReLU (zeros out negatives) but with a smooth curve.

  Input : [-3.0, -1.0, -0.5, 0.0, 0.5, 1.0, 2.0, 3.0]
  GELU  : [-0.004, -0.159, -0.154, 0.0, 0.346, 0.841, 1.954, 2.996]
  ReLU  : [0.0, 0.0, 0.0, 0.0, 0.5, 1.0, 2.0, 3.0]

--- Layer Normalization ---
LayerNorm re-centers and re-scales values at each position.

Before LayerNorm: mean=5.23, std=9.13
After  LayerNorm: mean=0.0000, std=1.0039

What just happened:

1. Lines 7 and 9 build a feed-forward network that expands from 128 to 512, then back to 128.
2. Line 8 applies GELU, which gently reduces negative numbers instead of snapping them to zero.
3. We passed messy numbers through LayerNorm and saw them neatly centered at zero with a standard spread of 1.

Shape Check:

• Input to FeedForward: [Batch, Time, 128]
• Output of FeedForward: [Batch, Time, 128]

Try It

Try It

Open src/ch07_feedforward.py and change the LayerNorm input to have a massive spread: x_single = torch.randn(config.n_embd) * 1000 + 500. Run the script again. You will see that LayerNorm still tames it back to a mean of 0 and a standard spread of 1.

In Business

Imagine you are building an assistant to draft company emails in your house style. If the system is unstable, it might output gibberish. LayerNorm acts like a manager double-checking work between steps, so that the data never drifts too far off track before it is passed to the next team.

Key Takeaways

• The feed-forward layer processes each token independently to build meaning.
• It expands the token's data 4x to give itself room to think.
• GELU provides a smooth mathematical curve to make training stable.
• LayerNorm resets numbers to a safe size so deep networks do not break.

Check Your Understanding

1. Why does the feed-forward network expand its input size by 4?
2. What happens to a strongly negative number when it passes through GELU?
3. What is the average value of a token's numbers immediately after LayerNorm?

Assignment

Assignment 8: Dimmer or switch

About 15 minutes. You change one number and write no new code. Work in this chapter's Colab notebook or on your own laptop.

1. Run it. Run src/ch07_feedforward.py as it is. Copy the three lines it prints under GELU activation: the Input, GELU and ReLU lines.
2. Change one number. Find the line sample = torch.tensor([-3.0, -1.0, -0.5, 0.0, 0.5, 1.0, 2.0, 3.0]). These are the eight test numbers the script feeds to GELU and to ReLU. Change the first one, -3.0, to -2.0 and run the script again. The list now holds -2.0 and 2.0, two numbers of the same size with opposite signs.
3. Compare. What does GELU print for -2.0, and what does it print for 2.0? What does ReLU print for the same two numbers? Which of the two, GELU or ReLU, gives -3.0 and -2.0 exactly the same result?
4. Explain it. In two or three sentences a manager could follow, say what GELU does to a positive number and to a negative number of the same size, and why the chapter calls GELU a dimmer and ReLU a switch.

Hand in: both sets of three lines, your answers to step 3, and your sentences.

Further Reading

The line that keeps training stable. As numbers pass through a deep stack they drift, some growing, some shrinking, until learning stalls. Layer normalization rescales each token's vector back to a standard spread before the next step, using only that token's own numbers, so it works the same whatever the batch size. It is one line in Chapter 8 and one of the reasons deep Transformers train at all.

The smooth switch inside the feed-forward layer. A network needs a non-linear step, or every layer collapses into one. The common choice cut everything negative to zero, a hard switch. GELU fades instead of cutting, which gives the optimizer a gentler surface to work on and is the activation used in GPT-style models, including the one in Chapter 8.

Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization (arXiv:1607.06450). arXiv. https://doi.org/10.48550/arXiv.1607.06450

Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (GELUs) (arXiv:1606.08415). arXiv. https://doi.org/10.48550/arXiv.1606.08415

Next up: Chapter 9: The Transformer Block