CVE-2025-23298: Getting Remote Code Execution in NVIDIA Merlin
The Trend Micro Zero Day Initiative discovered a critical vulnerability in NVIDIA's Transformers4Rec library, allowing remote code execution with root privileges. This vulnerability, CVE-2025-23298, stems from unsafe deserialization when loading model checkpoints using Python's pickle module. Transformers4Rec, part of the Merlin ecosystem, is widely used for recommendation tasks and integrates with Hugging Face Transformers. The flaw lies in the load_model_trainer_states_from_checkpoint function, which directly uses torch.load() without safety parameters, exposing it to malicious pickle files. Pickle's __reduce__ method allows arbitrary code execution during deserialization. The attack surface is significant due to common model sharing and the trust placed in checkpoint files, especially as these processes often run with elevated privileges. A malicious checkpoint could execute system commands before model weights are loaded. The real-world impact includes remote code execution, privilege escalation, data exfiltration, and supply chain attacks. NVIDIA has patched the vulnerability by implementing a custom loading mechanism that restricts deserialization to approved classes. This incident underscores the pervasive security challenges in the ML/AI ecosystem due to reliance on pickle. Developers are urged to avoid pickle for untrusted data, use weights_only=True, restrict trusted classes, and consider secure serialization formats like Safetensors. Organizations should audit model provenance, implement signing, and sandbox model loading. The ML community needs to move away from pickle and prioritize security in framework design.
load_model_trainer_states_from_checkpointfunction, which directly usestorch.load()without safety parameters, exposing it to malicious pickle files. Pickle's__reduce__method allows arbitrary code execution during deserialization. The attack surface is significant due to common model sharing and the trust placed in checkpoint files, especially as these processes often run with elevated privileges. A malicious checkpoint could execute system commands before model weights are loaded. The real-world impact includes remote code execution, privilege escalation, data exfiltration, and supply chain attacks. NVIDIA has patched the vulnerability by implementing a custom loading mechanism that restricts deserialization to approved classes. This incident underscores the pervasive security challenges in the ML/AI ecosystem due to reliance on pickle. Developers are urged to avoid pickle for untrusted data, useweights_only=True, restrict trusted classes, and consider secure serialization formats like Safetensors. Organizations should audit model provenance, implement signing, and sandbox model loading. The ML community needs to move away from pickle and prioritize security in framework design.