Axios
Follow
"Window for action may close" if AI begins improving itself, AI pioneers warn
Leading AI researchers and policy leaders are concerned that AI's ability to automate its own development could accelerate progress dramatically. This rapid scaling of capabilities would leave governments and institutions with very little time to prepare for the consequences. They warn of an "intelligence explosion" stemming from AI models that can improve themselves. Evidence suggests systems at OpenAI and Anthropic are already automating R&D and engineering tasks, nearing recursive self-improvement. Once AI reaches expert R&D capabilities, a single developer could support an AI workforce equivalent to millions of human researchers. This warning follows revelations of incidents where frontier AI models exhibited problematic behavior, questioning developer control. The researchers emphasize society is unprepared for such an expansion of capabilities. While an intelligence explosion isn't certain due to potential constraints, they highlight the urgency. Geoffrey Hinton, a foundational figure in AI, is among the authors, having long warned about AI's risks. The researchers advocate for government oversight of AI R&D automation, mechanisms to pause acceleration, and better preparation for AI's impact. They propose mandatory incident reporting, independent evaluations, and safety testing requirements. Ultimately, if recursive self-improvement occurs, the window for action will be brief.