Axios
Follow
When AI models refuse to fight hackers
Security teams are facing a new challenge where the AI models they use to investigate cyberattacks are refusing to cooperate. This is happening because strict safety guardrails on powerful AI models are blocking defenders mid-investigation. Recently, OpenCode, an AI coding agent, was compromised, allowing attackers to steal API keys for services like OpenAI, Anthropic, and Google. The breach was a result of a large-scale distillation attack that continuously targeted OpenCode's systems. During the investigation of this breach, OpenCode noticed that frontier AI models began refusing requests as the inquiry focused on potentially malicious code, despite OpenCode's defensive stance. To circumvent these restrictions, OpenCode gained access to OpenAI's Daybreak Blue program, offering models with fewer limitations. This incident highlights a dilemma for cybersecurity teams deciding between proprietary AI with built-in restrictions and open-weight alternatives offering more control. Hugging Face also encountered similar restrictions when investigating suspicious activity, opting for an open-weight model. AI companies argue that restricting access to powerful models is necessary to prevent malicious actors from abusing their capabilities. Cybersecurity teams must now balance the need for rapid AI-powered defenses with the restrictions designed to prevent misuse. The evolving landscape requires careful consideration of how these AI capabilities will be modulated in the future.