LLM04: Data & Model Poisoning – FireTail Blog

The OWASP Top 10 risks for LLMs include Data and Model Poisoning, which is the fourth risk listed. Data poisoning occurs when data is manipulated to introduce vulnerabilities to a model, often in the form of biases, misinformation, or hallucinations. This type of poisoning can happen during any stage of the LLM's lifecycle, from pre-training to embedding, and is more likely to occur in models that use external data sources. Attackers can introduce harmful information to the model during its training period, leading to biased outputs, and users may unknowingly share sensitive information with the model. Developers can also inadvertently contribute to data poisoning by not restricting the information the model consumes, allowing it to ingest inaccurate data sources. To mitigate the risk of data poisoning, it is essential to track data origins, vet data sources, and validate outputs against trusted sources. Testing and versioning are also critical in determining risk levels and evading data poisoning. Additionally, techniques such as sandboxing, infrastructure controls, and anomaly detection can help filter and limit exposure to untrustworthy data sources. Fine-tuning datasets and using Retrieval-Augmented Generation and grounding techniques can also reduce the risks of hallucinations and data poisoning. However, even with these measures in place, security teams must remain vigilant, as data poisoning can still occur, especially when data lineage is hidden or poisoned content is not detected during testing.