Ten Papers That Built Generative AI

Every idea in this book was somebody's result first. These ten papers are the ones the rest stands on: each solved a problem that was blocking the field, and each changed what people built afterwards. They are in the order the story runs, from how a network learns at all to how a text predictor became something you can give instructions to.

You do not need the mathematics to follow them. Each entry says what was not working, what the paper did about it, and why it still matters, and then gives the reference so you can read the original. All ten are free to read.

Table B.1: The ten papers, in the order the ideas arrived.

#	Paper	Year	What it settled
1	Backpropagation	1986	Gave every weight in a deep network its share of the blame.
2	Long short-term memory	1997	Gave a sequence model a memory it could keep over long text.
3	Word vectors	2013	Turned word meaning into distance in a vector space.
4	Attention	2014	Let a model look back over its input and weigh what mattered.
5	The Transformer	2017	Kept attention, removed the step-by-step reading.
6	Pre-training and transfer	2018	Trained one model on plain text, then reused it everywhere.
7	Decoder-only language models	2019	Showed next-token prediction alone learns many tasks.
8	Scaling laws	2020	Made the pay-off from more data and compute predictable.
9	Few-shot prompting	2020	Taught a model a task from examples in the prompt.
10	Learning from human feedback	2022	Turned a text predictor into something that follows instructions.

Backpropagation (1986)

How a network learns anything at all. A network with layers in the middle had an obvious problem: when the answer came out wrong, nobody could say which of the middle weights was at fault. This paper gave the answer. Send the error backwards through the network and give each weight a share of the blame in proportion to how much it moved the result. Every model in this book learns that way, and so does every model in production today; loss.backward() in the training loop is this paper.

In this book: Chapter 13.

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. https://doi.org/10.1038/323533a0

Long short-term memory (1997)

Memory, before attention. Models that read a sentence one word at a time kept forgetting the beginning by the time they reached the end, because the learning signal faded as it travelled back through the steps. The fix was a cell with gates that decide what to keep, what to drop, and what to pass on. This ran almost every serious language system for twenty years. The question it answers, what should I still remember from earlier in the text, is the same question attention answers, by a completely different route.

In this book: Chapter 2.

Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

Word vectors (2013)

Meaning becomes geometry. Give every word its own ID number and the model learns nothing from the numbering: "king" sits as far from "queen" as it does from "toaster". This paper trained a deliberately cheap prediction task so that words used in similar company ended up with similar vectors, and did it fast enough to run on billions of words. Embeddings, the subject of Chapter 5, start here, and so does the vector search behind modern recommendation and retrieval.

In this book: Chapter 5.

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space (arXiv:1301.3781). arXiv. https://doi.org/10.48550/arXiv.1301.3781

Attention (2014)

The first attention. Translation models of the day read the whole source sentence, squeezed it into a single fixed vector, and wrote the translation from that. Long sentences did not survive the squeeze. The fix: let the model look back over every input word and decide, at each output word, which ones matter right now. That weighted look-back is attention. The Transformer three years later kept this idea and threw out everything around it.

In this book: Chapter 6.

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate (arXiv:1409.0473). arXiv. https://doi.org/10.48550/arXiv.1409.0473

The Transformer (2017)

The architecture this book builds. Reading a sequence one step at a time is slow, because step 500 cannot start until step 499 has finished, and distant words stay hard to connect. This paper removed the step-by-step reading entirely and kept only attention, plus a note of each token's position. Every token can then be processed at once, which is what made training on very large amounts of text practical. The model you build in Chapters 6 to 10 is this design, made small.

In this book: Chapter 6, Chapter 7, Chapter 9.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need (arXiv:1706.03762). arXiv. https://doi.org/10.48550/arXiv.1706.03762

Pre-training and transfer (2018)

Train once, reuse everywhere. Labeled examples are expensive, and every new task used to need its own pile of them. BERT trained one Transformer on ordinary text by hiding words and asking it to fill the blanks, then adapted that single model to many tasks with a small amount of labeled data each. It reads in both directions at once, which is exactly what the causal mask in Chapter 11 forbids: masking is what separates a model that fills blanks from one that writes forward.

In this book: Chapter 11.

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding (arXiv:1810.04805). arXiv. https://doi.org/10.48550/arXiv.1810.04805

Decoder-only language models (2019)

One model, many jobs, no retraining. The question was whether a model trained only to predict the next word would pick up skills nobody trained it for. Trained on a large, varied sweep of web pages, it began answering questions, summarizing and translating with no task-specific training at all, simply because the prompt made the task clear. That settled the architecture question for text generation: a decoder that predicts the next token, which is the model in Chapter 10.

In this book: Chapter 2, Chapter 10.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners [Technical report]. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

Scaling laws (2020)

Why the field started building bigger. Before this, deciding how large to make a model, how much text to train it on, and how much compute to spend was guesswork. The paper measured all three and found the error falls along smooth, predictable curves across a very wide range of sizes. That turned model building into a budgeting exercise, and it is the reason the industry spent the following years scaling up. It also explains the ceiling on the model you train here: a few hundred thousand parameters and a few hundred thousand characters of Shakespeare buy a certain quality of text, and no more.

In this book: Chapter 12, Chapter 17.

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models (arXiv:2001.08361). arXiv. https://doi.org/10.48550/arXiv.2001.08361

Few-shot prompting (2020)

Where prompting came from. Even a pre-trained model normally had to be fine-tuned, with fresh labeled examples and a training run, before it could do a new job. At 175 billion parameters the authors found something different: write two or three examples into the prompt and the model follows the pattern, with no weights changed at all. That behavior is what people now call prompting, and it is why a language model became something you talk to rather than something you retrain.

In this book: Chapter 17.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., ... Amodei, D. (2020). Language models are few-shot learners (arXiv:2005.14165). arXiv. https://doi.org/10.48550/arXiv.2005.14165

Learning from human feedback (2022)

The step that turns a text predictor into an assistant. A model trained to continue text is not the same thing as a model that does what you ask; it will happily continue your question with more questions. The authors collected human demonstrations of good answers and human rankings of competing answers, and fine-tuned on that feedback. The result matters for how you read this book: a 1.3 billion parameter model trained this way was preferred by people to the 175 billion parameter model it came from. Capability and helpfulness are different problems, and this book builds the first one.

In this book: Chapter 17.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback (arXiv:2203.02155). arXiv. https://doi.org/10.48550/arXiv.2203.02155

The rest of the reading

These are the papers behind particular pieces of the model you built. Each one also appears at the end of the chapter that uses it.

Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization (arXiv:1607.06450). arXiv. https://doi.org/10.48550/arXiv.1607.06450

Fan, A., Lewis, M., & Dauphin, Y. (2018). Hierarchical neural story generation (arXiv:1805.04833). arXiv. https://doi.org/10.48550/arXiv.1805.04833

He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep residual learning for image recognition (arXiv:1512.03385). arXiv. https://doi.org/10.48550/arXiv.1512.03385

Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (GELUs) (arXiv:1606.08415). arXiv. https://doi.org/10.48550/arXiv.1606.08415

Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019). The curious case of neural text degeneration (arXiv:1904.09751). arXiv. https://doi.org/10.48550/arXiv.1904.09751

Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization (arXiv:1412.6980). arXiv. https://doi.org/10.48550/arXiv.1412.6980

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90. https://doi.org/10.1145/3065386

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., ... Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library (arXiv:1912.01703). arXiv. https://doi.org/10.48550/arXiv.1912.01703

Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386–408. https://doi.org/10.1037/h0042519

Sennrich, R., Haddow, B., & Birch, A. (2015). Neural machine translation of rare words with subword units (arXiv:1508.07909). arXiv. https://doi.org/10.48550/arXiv.1508.07909

Weizenbaum, J. (1966). ELIZA: A computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45. https://doi.org/10.1145/365153.365168