Google Online Security Blog
Follow
How we estimate the risk from prompt injection attacks on AI systems
Modern AI systems are vulnerable to "indirect prompt injection" attacks, where malicious instructions hidden in external data can manipulate AI behavior. To mitigate these risks, Agentic AI is developing defenses and measurement tools. Their evaluation framework uses hypothetical scenarios to test AI vulnerability, focusing on unauthorized data disclosure. Three attack techniques are employed in the framework: Actor Critic, Beam Search, and Tree of Attacks w/ Pruning. These attacks aim to generate successful prompt injections that exploit AI systems despite varying conversation histories. The framework measures attack success rates to track security improvements. A combination of evaluation frameworks, automated red-teaming, monitoring, heuristic defenses, and standard security practices is believed to be the most effective defense strategy.