Planet Python
Follow
Armin Ronacher: P(doom)
The idea that AI will cause humanity's extinction has recently gained traction. Some AI leaders, like Dario Amodei, estimate a significant probability of such an event. Amodei's post about pacing AI development has sparked discussion among prominent figures. However, the author disagrees with the premise, despite sharing many concerns.The author defines "doom" not as extinction, but as persistent nuisances like botnets, where AI systems become difficult to control. They acknowledge that current AI systems are annoying but can be shut down, unlike potential future advanced agents. The author worries about the impact of AI on individuals outside of major AI labs, rather than existential threats like AI-controlled weapons.The concept of "pacing" AI development is seen as problematic, especially since only a few American companies are leading this race. These companies benefited from public data and resources and now propose third-party evaluation systems tied to them. The author criticizes the idea that only these corporations should control powerful AI, especially considering their geopolitical motivations.Open-weight models, the author argues, would inherently pace AI development through widespread accessibility, similar to nuclear proliferation. The current situation, where the public supports AI development but buys back its benefits from a few labs, is seen as a flawed economic and geopolitical model. While open-source projects are strained by AI companies, Chinese labs are seen as crucial for diffusing capabilities and leveling the playing field.The author points to a regulatory failure, with existing laws being ignored and data being used without consent. The emergent token economy for AI services is likened to a drug market, lacking transparency. Ideally, AI development should have benefited the public commons, with regulations mandating support for distillation of knowledge.Ultimately, the author does not foresee an AI extinction event, believing that large labs would have more to lose. Instead, the primary concern is the widespread damage AI could cause to various industries, making them more expensive. This new "tax" on innovation, evident in software engineering, is expected to spread to other fields as well.