Schneier on Security
Follow
Stealing AI Reasoning Traces
Leading LLM providers now hide their models' step-by-step reasoning to protect their intellectual property. This reasoning, also known as chain-of-thought, is returned to the client as encrypted text for subsequent requests. The research identifies an architectural vulnerability where these encrypted blocks are interchangeable across sessions, users, and models from the same provider. Exploiting this, a scalable decryption jailbreak is developed. By injecting an encrypted trace from a strong model into a weaker one, the weaker model is forced to output the trace in plaintext. This bypasses anti-distillation mechanisms, allowing adversaries to extract proprietary reasoning. The vulnerability also enables large-scale private data extraction, as demonstrated by recovering PII and credentials from public repositories. Hazardous information, even when the final output is safe, can also be revealed. Furthermore, attackers can execute invisible prompt injections by hiding malicious payloads within these encrypted blocks. The research proposes cryptographic and system-level mitigations to address these security concerns.