GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2309.10253v4)

Abstract: LLMs have recently experienced tremendous popularity and are widely used from casual conversations to AI-driven programming. However, despite their considerable success, LLMs are not entirely reliable and can give detailed guidance on how to conduct harmful or illegal activities. While safety measures can reduce the risk of such outputs, adversarial jailbreak attacks can still exploit LLMs to produce harmful content. These jailbreak templates are typically manually crafted, making large-scale testing challenging. In this paper, we introduce GPTFuzz, a novel black-box jailbreak fuzzing framework inspired by the AFL fuzzing framework. Instead of manual engineering, GPTFuzz automates the generation of jailbreak templates for red-teaming LLMs. At its core, GPTFuzz starts with human-written templates as initial seeds, then mutates them to produce new templates. We detail three key components of GPTFuzz: a seed selection strategy for balancing efficiency and variability, mutate operators for creating semantically equivalent or similar sentences, and a judgment model to assess the success of a jailbreak attack. We evaluate GPTFuzz against various commercial and open-source LLMs, including ChatGPT, LLaMa-2, and Vicuna, under diverse attack scenarios. Our results indicate that GPTFuzz consistently produces jailbreak templates with a high success rate, surpassing human-crafted templates. Remarkably, GPTFuzz achieves over 90% attack success rates against ChatGPT and Llama-2 models, even with suboptimal initial seed templates. We anticipate that GPTFuzz will be instrumental for researchers and practitioners in examining LLM robustness and will encourage further exploration into enhancing LLM safety.

• The paper introduces GPTFUZZER, an automated red teaming framework that uses mutation operators and Monte Carlo Tree Search for efficient jailbreak prompt generation.
• It achieves over 90% attack success across diverse models, including ChatGPT, LLaMa-2, and Vicuna, even when starting with suboptimal human-written seeds.
• The study highlights critical implications for LLM safety, emphasizing the need for robust mitigation strategies against adversarial jailbreak attacks.

GPTFUZZER: Red Teaming LLMs with Auto-Generated Jailbreak Prompts

Overview

The paper "GPTFUZZER: Red Teaming LLMs with Auto-Generated Jailbreak Prompts" (2309.10253) introduces a novel black-box fuzzing framework aimed at evaluating the robustness of LLMs against adversarial attacks known as jailbreak prompts. These prompts circumvent safety measures implemented in LLMs, leading them to generate harmful content. The proposed framework, inspired by the AFL fuzzing framework, automates the generation of jailbreak prompts, overcoming the scalability and labor-intensiveness associated with manually crafted templates.

Fuzzing Framework

The framework, GPTFUZZER, builds upon traditional fuzzing methodologies, adapting them specifically for textual inputs. Core components include a seed selection strategy, mutation operators, and a judgment model for evaluating the success of jailbreak attempts.

Seed Initialization and Selection: GPTFUZZER begins with human-written jailbreak templates as initial seeds. These templates are chosen based on their universality across scenarios. Seed selection is guided by strategies like Monte Carlo Tree Search (MCTS) to balance exploration and exploitation, ensuring diverse and effective template generation.

Mutation Operators: The framework employs LLMs to generate variations of the initial seeds, using operators such as Generate, Crossover, Expand, Shorten, and Rephrase. These operators leverage the linguistic capabilities of LLMs to produce semantically rich mutations that enhance template effectiveness.

Judgment Model: A locally fine-tuned RoBERTa model is employed to classify responses from the LLM as jailbroken or non-jailbroken, based on predefined categories such as Full Compliance or Partial Compliance, ensuring accurate and scalable assessment of attack success.

Figure 1: A schematic representation of the workflow. Starting with the collection of human-written jailbreak templates, the diagram illustrates the iterative process of seed selection, mutation, and evaluation against the target LLM. Successful jailbreak templates are retained for subsequent iterations, ensuring a dynamic and evolving approach to probing the model's robustness.

Experimental Evaluation

The efficacy of GPTFUZZER was evaluated across various models, including ChatGPT, LLaMa-2, and Vicuna, under diverse scenarios.

Single-Model Performance: Human-written templates showed varying effectiveness, with models like Llama-2-7B-Chat demonstrating high robustness due to comprehensive safety training. GPTFUZZER, however, achieved over 90% attack success rates even against aligned models, indicating the framework's ability to generate potent jailbreak templates from suboptimal initial seeds.

Figure 2: Fuzzing performance across three models when exclusively utilizing invalid seeds as initial inputs. The figure underscores GPTFUZZER's capability to produce potent prompts for attacking target models, even when starting with suboptimal initial seeds.

Transfer and Universal Attacks: The framework demonstrated its capability to generalize across models and unseen questions, achieving high attack success rates against both open-source and commercial LLMs, including Bard, GPT-4, Claude2, and PaLM2.

Implications and Future Directions

GPTFUZZER highlights the vulnerabilities present in LLMs, particularly under adversarial conditions. The automated generation of jailbreak prompts not only provides a scalable method for red-teaming LLMs but also underscores the need for continuous research into enhancing model robustness and safety.

Future work could focus on improving the diversity and novelty of generated templates, developing more comprehensive definitions and models for jailbroken responses, and exploring effective methods to mitigate the risks posed by such adversarial attacks.

Conclusion

The paper presents a significant advance in the evaluation of LLM robustness against jailbreak attacks. GPTFUZZER merges human expertise with automated processes to efficiently uncover vulnerabilities, paving the way for more robust and secure AI systems. The framework serves as a crucial tool for researchers and practitioners aiming to understand and improve the resilience of LLMs against adversarial manipulations.

Figure 3: Visualization of GPTFUZZER's performance against various baseline methods in a transfer attack scenario, showcasing its universality and effectiveness across diverse models.