Understanding the Evolution of ChatGPT: Part 2 — GPT-2 and GPT-3

The article discusses the evolution of ChatGPT, focusing on GPT-2 and GPT-3, which were designed to bypass the finetuning stage and make language models truly intelligent. GPT-2 and GPT-3 share similar architectures and were developed with a common philosophy, exploring task-agnostic learning, the scaling hypothesis, and in-context learning. The paradigm shift from "pre-training plus finetuning" to "pre-training only" was motivated by the zero-shot behaviors of GPT-1, which showed that pre-training could improve its zero-shot capability. Finetuning has limitations, including the need for large datasets for each new task and the risk of exploiting spurious correlations in the finetuning data. Task-agnostic learning, the scaling hypothesis, and in-context learning are key elements that influenced the design of GPT-2 and GPT-3. GPT-2 was designed to test whether a larger model pre-trained on a larger dataset could be directly used to solve downstream tasks, while GPT-3 tested whether in-context learning could bring improvements over GPT-2 when further scaled up. GPT-2 achieved good results on many tasks, but still performed worse than state-of-the-art models on some tasks, motivating the development of GPT-3. GPT-3 adopted a similar model architecture to GPT-2 and was trained on even larger datasets, achieving strong performance on many NLP datasets. The development of GPT-2 and GPT-3 paves the way for new research directions in NLP and the broader ML community, focusing on understanding emergent capabilities and developing new training paradigms.