Google Online Security Blog
Follow
AI threats in the wild: The current state of prompt injections on the web
Google's Threat Intelligence teams are actively monitoring Indirect Prompt Injection (IPI) attacks, a major security concern for AI systems. They investigated real-world IPI exploitation by scanning the public web using Common Crawl, a large web archive. Their research aimed to identify how threat actors are using IPI. The team developed a multi-stage approach, including pattern matching, classification with Gemini, and manual review, to filter out false positives. The analysis revealed attempts at manipulating AI, including harmless pranks, SEO, and malicious activities. Malicious attempts included data exfiltration and destruction, though these were generally unsophisticated. The findings suggest that IPI attack sophistication is low but increasing, with a 32% rise in malicious attempts observed. Google anticipates a rise in both the scale and sophistication of IPI attacks in the future. They are investing in model hardening and red teams, including external researcher participation through a vulnerability reward program. Google's real-time data processing capabilities allow them to identify and neutralize threats. They have provided resources for further exploration of their GenAI security research.