Google Online Security Blog
Follow
Mitigating prompt injection attacks with a layered defense strategy
Generative AI's rise introduces indirect prompt injections, where malicious instructions hide in external data sources like emails. These attacks can manipulate AI systems, exfiltrate data, and execute rogue actions, demanding robust security measures. Google employs a layered defense for Gemini, hardening models and using machine learning to detect malicious instructions. Prompt injection content classifiers filter harmful data, while security thought reinforcement steers the AI to ignore adversarial commands. Markdown sanitization and suspicious URL redaction prevent data exfiltration via images and unsafe links. A user confirmation framework requires explicit consent for risky actions, adding a human element to the security. End-user security mitigation notifications inform users about stopped attacks and provide learning resources. Google collaborates with researchers and shares threat intelligence to strengthen AI security. The company uses red teaming, BugSWAT events, and frameworks like SAIF to test and improve defenses. Future Gemini models will feature enhanced resilience and additional prompt injection defenses. Google's approach is detailed in various resources for those seeking to understand AI security threats and mitigation strategies.