LLMs Achieve Parallel In-Context Learning Through Remarkable "Task Superposition" Capability

Large Language Models have shown impressive in-context learning capabilities, and a recent study explores a surprising phenomenon where these models can perform multiple, computationally distinct tasks simultaneously during a single inference call, a capability called "task superposition." The researchers provide empirical evidence of this phenomenon across different LLM families and scales, and show that it emerges even when the model is trained to learn one task at a time. The study offers theoretical explanations for this capability and explores how LLMs internally compose task vectors during superposition. The findings provide insights into the latent capabilities of LLMs and raise questions about the mechanisms enabling simultaneous task execution. The researchers found that larger LLMs can solve more ICL tasks in parallel and better calibrate their output distributions. The study's findings offer valuable insights into the nature of large language models and their potential for simultaneous task execution. However, the research has limitations, such as the lack of a comprehensive investigation into the boundaries or limitations of this task superposition phenomenon. Further research could explore the extent to which LLMs can juggle multiple tasks and the factors that influence their performance. The study's findings have significant implications for the future development and applications of large language models.