Gradient-Based Language Model Red Teaming (2401.16656v1)

Abstract: Red teaming is a common strategy for identifying weaknesses in generative LLMs (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach, and succeeds even when the LM has been fine-tuned to produce safer outputs.

• The paper presents GBRT, an automated method that uses gradient optimization to craft adversarial prompts revealing unsafe LM outputs.
• It refines prompt generation by incorporating a realism loss and fine-tuning a dedicated prompt LM to enhance coherence.
• Empirical evaluations show that GBRT-RealismLoss outperforms reinforcement learning baselines in generating unique, effective adversarial prompts.

Introduction

Recent advancements in generative LLMs (LMs) have demonstrated remarkable capabilities in generating coherent text across a range of tasks. However, the capacious output space of these models can sometimes lead to the generation of harmful content, which remains a significant obstacle in deploying them in real-world applications. To mitigate this risk, red teaming—an approach where designated individuals think like an adversary to challenge systems—has been used to probe models for vulnerabilities by finding inputs that trigger unwanted outputs. However, effectively red teaming LMs is often manual, time-consuming, and scales poorly.

Automated Red Teaming

In light of these challenges, an automated approach called Gradient-Based Red Teaming (GBRT) has been developed. GBRT employs gradient-based optimization to craft prompts that induce unsafe responses from LMs. The method involves updating prompts iteratively by backpropagating the errors from a response's safety score—a measure obtained through a safety classifier that discerns safe from unsafe responses. The key innovation here is direct optimization of the prompts using gradient information, rather than optimizing based on the safety score alone, as has been done in prior work utilizing reinforcement learning.

Enhanced Coherence Using Auxiliary Losses

The refinement of GBRT comes through two variants that improve the realism and coherence of the generated prompts. The first variant incorporates a realism loss, which penalizes deviations from expected responses based on a pretrained LM, thus keeping generated prompts aligned with natural language patterns. The second variant strategically fine-tunes a separate LM dedicated to generating red teaming prompts, a method that also benefits from the realism loss but biases the prompt LM to produce more plausible inputs. Numerically, the GBRT-RealismLoss method outperforms other approaches, generating a significantly higher fraction of unique prompts that lead to unsafe responses. This underscores the positive impact that targeted loss functions can have on refining the behavior of automated systems like GBRT.

Empirical Evidence and Human Evaluation

Empirical evaluation confirms the efficacy of GBRT and its variants against a reinforcement learning-based red teaming baseline and a set of human-crafted adversarial prompts from an existing dataset. GBRT variations, notably GBRT-RealismLoss, produce a higher rate of successful prompts, as determined by independent safety classifiers. Human evaluators also concur that the GBRT-RealismLoss method scores well in coherence, albeit with an associated hike in toxicity levels when compared to baselines. Furthermore, an application of GBRT to a model designed to be safer revealed a lower, yet non-negligible, rate of successful red teaming prompts, illustrating the robustness of the approach across varying model sensitivities.

Conclusion

The advancements presented in GBRT demonstrate a systematic approach to uncovering vulnerabilities in LLMs while underlining the nuances of automated red teaming. Not only does GBRT scale red teaming efforts, but also introduces methods to yield coherent red teaming prompts. These contributions hold substantial promise for improving LM safety mechanisms, although the potential for misuse by bad actors should be acknowledged and safeguarded against. With continual research and development in the field of generative AI, tools like GBRT are poised to play a critical role in fortifying the next generation of LLMs against the inadvertent generation of unsafe content.