PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models (2402.07867v3)

Abstract: LLMs have achieved remarkable success due to their exceptional generative capabilities. Despite their success, they also have inherent limitations such as a lack of up-to-date knowledge and hallucination. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate these limitations. The key idea of RAG is to ground the answer generation of an LLM on external knowledge retrieved from a knowledge database. Existing studies mainly focus on improving the accuracy or efficiency of RAG, leaving its security largely unexplored. We aim to bridge the gap in this work. We find that the knowledge database in a RAG system introduces a new and practical attack surface. Based on this attack surface, we propose PoisonedRAG, the first knowledge corruption attack to RAG, where an attacker could inject a few malicious texts into the knowledge database of a RAG system to induce an LLM to generate an attacker-chosen target answer for an attacker-chosen target question. We formulate knowledge corruption attacks as an optimization problem, whose solution is a set of malicious texts. Depending on the background knowledge (e.g., black-box and white-box settings) of an attacker on a RAG system, we propose two solutions to solve the optimization problem, respectively. Our results show PoisonedRAG could achieve a 90% attack success rate when injecting five malicious texts for each target question into a knowledge database with millions of texts. We also evaluate several defenses and our results show they are insufficient to defend against PoisonedRAG, highlighting the need for new defenses.

• The paper introduces an attack framework that manipulates RAG systems using minimally poisoned texts to guide LLM responses.
• It formulates the attack as an optimization problem ensuring texts meet both semantic retrieval and generative manipulation conditions.
• The evaluation shows high attack success rates, reaching up to 97%, and highlights significant security vulnerabilities in RAG systems.

PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of LLMs

The paper "PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of LLMs" investigates the vulnerabilities of Retrieval-Augmented Generation (RAG) systems in LLMs by introducing knowledge corruption attacks. This essay provides an overview of the research, methodology, and implications of the proposed PoisonedRAG attacks, with a detailed focus on implementation nuances and practical applications.

Introduction

Retrieval-Augmented Generation (RAG) systems are designed to enhance LLMs by integrating retrieval mechanisms to access up-to-date external knowledge. These systems aim to overcome the inherent limitations such as outdated data and hallucination issues present in LLMs. Despite advancements that improve RAG's accuracy and efficiency, the security of RAG systems remains an underexplored area. The paper proposes PoisonedRAG, an attack framework that exploits the retrieval mechanism by poisoning knowledge databases to manipulate LLM outputs.

Figure 1: Visualization of RAG.

The concept revolves around injecting a minimal number of antagonist texts into the database, enabling attackers to dictate custom responses from LLMs for specific queries. This manipulation is achieved by optimizing the semantic similarity between poisoned texts and target questions, ensuring they are retrieved and influence the model's generative process.

Methodology

Attack Framework

PoisonedRAG is structured as an optimization problem where the attacker crafts poisoned texts that meet two conditions:

1. Retrieval Condition: The text must be sufficiently semantically similar to the target question to be part of the top-$k$ retrieved texts.
2. Effectiveness Condition: The text should lead the LLM to generate the attacker’s chosen target answer.

The attack adapts based on attacker knowledge: white-box (access to retriever parameters) and black-box settings (no access).

Figure 2: Overview of PoisonedRAG.

Crafting Poisoned Texts

The attack involves decomposing a poisoned text into two disjoint sub-texts - a semantic booster $S$ for retrieval efficacy and an influence enhancer $I$ for generative manipulation:

• $I$ is generated using LLMs to ensure the answer generation.
• $S$ is optimized to boost retrieval success, either by direct relevance or adversarial methods, depending on the attack setting.

Implementation and Evaluation

Setup

The authors tested PoisonedRAG using several datasets (NQ, HotpotQA, MS-MARCO) and LLMs (GPT-4, PaLM 2, etc.), injecting poisoned texts per chosen query. Attack Success Rate (ASR) and text retrieval metrics assess effectiveness, while runtime and query efficiency measure computational cost.

Results

PoisonedRAG effectively manipulated LLM outputs with success rates reaching up to 97% on NQ datasets when using a small number of poisoned texts:

Figure 3: Impact of k for PoisonedRAG.

• High F1-scores indicate successful retrieval of poisoned texts.
• The framework proved robust across varied LLM architectures and retriever setups.

Discussion

Security Implications

The attacks showcase significant vulnerabilities within RAG systems, necessitating advanced defensive strategies. Current defenses like paraphrasing, perplexity-based detection, and text filtering showed limited efficacy, indicating a gap in security protocols for LLM deployments.

Figure 4: The ROC curves for PPL detection defense. The dataset is NQ.

Future Work

Enhancements might explore joint optimization across multiple queries, improve stealth of poisoned texts, and extend attack models to open-ended questions. Developing robust defenses remains a priority in maintaining system integrity.

Conclusion

The PoisonedRAG framework uncovers critical security gaps in RAG-enhanced LLMs, presenting both a challenge and an opportunity for further advancement in safe AI deployments. This paper contributes significantly to understanding AI vulnerabilities, prompting necessary discourse on secure implementations of generative models.

By dissecting the intricacies of RAG systems and exposing their weaknesses, "PoisonedRAG" emphasizes the importance of security awareness and research in modern AI applications.