Chapter 16: Temperature and Top-k¶
Figure 16.1: We tune how the text is generated.
Random sampling produces diversity, but sometimes at the cost of coherence. The model might select a rare character which violates grammar or produces an unrecognizable word. We’ll want some reins on the randomness. In this chapter you will:
- Use temperature to adjust how risky the model's choices are.
- Use top-k to cut the list of choices short, so the model cannot pick truly bad characters.
- Combine these controls for optimal text generation.
Words to Know
- Temperature: a number the scores (logits) are divided by before they become probabilities. A low temperature plays it safe, and a high one takes risks.
- Top-k: a limit that lets the model sample only from the k most likely next tokens, where k is a number you choose (40 here).
Theory¶
There are two knobs between the model's raw scores and the character it ultimately selects. Temperature alters the odds. Top-k pruning determines which characters are eligible. They are independent, and you will usually set both.
Figure 16.2: Temperature changes the shape of the odds, and top-k changes how many characters stay in the running.
Temperature¶
Temperature is essentially one mathematical operation applied to logits just before softmax. We divide each logit by one scalar value, and that alters the distribution over characters.
Figure 16.3: Low temperature sharpens the choice, and high temperature flattens it.
- Low temperature (e.g., 0.5): Dividing by a number smaller than 1 pushes the scores further apart, so the top choice becomes overwhelmingly favored and the model plays it safe.
- High temperature (e.g., 2.0): Dividing by a large number squishes the scores together. The probabilities flatten out, meaning unusual choices become more likely, and the model takes risks.
- T = 1.0: This is standard sampling, where the model uses its learned probabilities exactly as they are.
Top-k¶
There is still a probability at any given temperature that could result in the wrong character being chosen. That chance might be 0.001%. If you sample enough times you will eventually see a mistake.
By using top-k sampling we can cap our selection pool. If we set top_k = 40, we would take all 65 character choices, keep the 40 with the highest scores, and drop the bottom 25 by setting their probabilities to zero. By dropping these choices we are preventing any sampling from occurring on those lower end values.
Figure 16.4: Top-k keeps the highest-scoring characters and zeroes the rest. Drawn here with ten characters and k = 4, while the book's code uses 65 and k = 40.
These controls adjust the Next-Token Scores on the "where we are" map, right before generation loops back to produce the New Text.
In Business
When creating boilerplate text for compliance documentation with the house-style assistant, you may want to choose a lower temperature (0.5) setting so that the text it generates is always as close to the safe outputs as possible. However, if you want witty taglines to market your business, try using a higher temperature (0.9) setting with top-k sampling (40) to get creative, surprising results that still conform to grammatical norms.
Code¶
We apply temperature and top-k right before the softmax function in the generation loop.
Line 2 divides the scores by the temperature to adjust randomness. Line 7 discards any score below the top-k threshold by setting it to negative infinity.
Figure 16.5: The code applies temperature, then top-k, then softmax.
We implement top-k in PyTorch using masked_fill. We take the score of the 40th best token (threshold), and change any token with lower score to -inf. When we apply softmax to turn scores into probabilities, -inf becomes exactly 0.
The run below shows how different settings affect the generated text.
$ python src/ch15_generate_sampling.py
--- temp=0.5, top_k=40 ---
JULIET:
My must grace prove a son the come on my lord.
...
--- temp=0.8, top_k=40 ---
JULIET:
That will, my call thee for thee king win a be parder
The his in suppy ascented a not a say?
...
--- temp=1.0, top_k=40 ---
JULIET:
Look, do indo witned me devise thy grohed:
A give gring brothe whith ighers paison:
...
--- temp=1.5, top_k=40 ---
JULIET:
fattlyals give niet, thou stain, viful-wid thel,
Brob dayse mernion shing you fea,
...
--- temp=1.0, no top_k ---
JULIET:
Than mide smear of thou I am go vious, you me
now you scalf them frate ell, to 'till will she
...
--- Sweet spot ---
temperature=0.8 to 1.0 and top_k=40 usually gives the best results.
What just happened:
- At
temp=0.5, the text is coherent but relies heavily on common, safe words. - At
temp=0.8to1.0, the text feels more natural and varied. - At
temp=1.5, the text quickly fills with made-up words ("fattlyals"). - A temperature between 0.8 and 1.0, combined with
top_k=40, typically works best.
Shape Check¶
Table 16.1 lists the shapes of tensors used during top-k filtering.
Table 16.1: Tensors modified during the top-k sampling process.
| Tensor | Shape | What it means |
|---|---|---|
threshold |
[1, 1] |
The cutoff score (the 40th highest value). |
logits |
[1, 65] |
65 scores, where 25 of them have been set to -inf. |
Try It¶
Try It
Open the script src/examples/ch16_explore_temp.py and run it. It initializes with a very low temperature (0.1), and you will see it is essentially doing greedy decoding (picking the safe top character every time).
Key Takeaways¶
- Temperature adjusts the spread of the probabilities. Low temperature makes the model conservative, while high temperature makes it creative and risky.
- Top-k restricts the model from ever picking the worst-scoring tokens and so prevents bizarre errors.
- To combine them, first apply temperature, then apply top-k to filter out the bad options, and finally convert to probabilities with softmax.
- A combination of
temperature=0.8andtop_k=40is a widely used default for high-quality text generation.
Check Your Understanding¶
- If you set the temperature to 0.1, what happens to the gap between the highest and lowest scores?
- Why do we replace the eliminated scores in top-k with negative infinity (
-inf) instead of0? - Which step must happen first in the code: applying top-k or calculating softmax probabilities?
Assignment¶
Assignment 16: Turn up the heat
About 15 minutes. You change one number and write no new code. Work in this chapter's Colab notebook or on your own laptop.
- Run it. Run
src/examples/ch16_explore_temp.pyas it is. Copy the text the model writes afterJULIET:. - Change one number. Near the bottom of the file, find
temperature=0.1. Temperature is the number the scores are divided by before the model draws a character. Change0.1to2.0and run the script again. The lineGenerating with T=0.1...is a fixed label, so it will still say 0.1. - Compare. Which text keeps repeating the same words? Which one is full of made-up words? Which one is closer to real English?
- Explain it. In two or three sentences a manager could follow, say why neither setting would suit an assistant that writes to customers, and which way you would turn the dial from each one.
Hand in: both texts, your answers to step 3, and your sentences.
Further Reading¶
Where top-k sampling comes from. This paper is about writing stories from a prompt, and along the way it introduced the sampling rule you use in Chapter 16: keep only the k most likely next tokens and draw from those. It keeps the text varied without letting the model pick something absurd from the long tail.
Why always picking the likeliest word goes wrong. Always taking the highest-scoring next token, which Chapter 15 calls greedy decoding, produces flat and repetitive text, and this paper shows that this happens because human writing is not made of the most predictable word at every turn. Their alternative keeps the smallest set of tokens whose probabilities add up to a chosen share and samples from that. How you choose the next token matters as much as how well the model was trained.
Fan, A., Lewis, M., & Dauphin, Y. (2018). Hierarchical neural story generation (arXiv:1805.04833). arXiv. https://doi.org/10.48550/arXiv.1805.04833
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019). The curious case of neural text degeneration (arXiv:1904.09751). arXiv. https://doi.org/10.48550/arXiv.1904.09751
Next up: Chapter 17: Putting It All Together