The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance (2401.03729v3)

Abstract: LLMs are regularly being used to label data across many domains and for myriad tasks. By simply asking the LLM for an answer, or ``prompting,'' practitioners are able to use LLMs to quickly get a response for an arbitrary task. This prompting is done through a series of decisions by the practitioner, from simple wording of the prompt, to requesting the output in a certain data format, to jailbreaking in the case of prompts that address more sensitive topics. In this work, we ask: do variations in the way a prompt is constructed change the ultimate decision of the LLM? We answer this using a series of prompt variations across a variety of text classification tasks. We find that even the smallest of perturbations, such as adding a space at the end of a prompt, can cause the LLM to change its answer. Further, we find that requesting responses in XML and commonly used jailbreaks can have cataclysmic effects on the data labeled by LLMs.

• The paper demonstrates that minor prompt alterations can change up to 10% of predictions, highlighting significant sensitivity in LLM performance.
• The study systematically evaluates 24 prompt variations across 11 tasks using models like ChatGPT and Llama 2 to assess impacts on accuracy and prediction similarity.
• The findings imply that while larger models are more robust, refined prompt engineering is crucial for ensuring reliable LLM performance in real-world applications.

The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect LLM Performance

Introduction

The paper "The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect LLM Performance" explores the sensitivity of LLMs to variations in prompt construction. Despite the growing use of LLMs in text classification tasks, little attention has been paid to how susceptible these models are to minor alterations in prompts. The authors systematically evaluate the impact of prompt variations across 11 text classification tasks, revealing surprising levels of variability in model performance due to even trivial changes.

Methodology

The authors conduct their experiments using OpenAI's ChatGPT and Llama 2 models of varying sizes (7B, 13B, and 70B). They explore 24 types of prompt variations categorized into Output Formats, Perturbations, Jailbreaks, and Tipping. The experiments are designed to assess how these prompt variations influence model predictions and accuracy. Each task involves 11,000 samples, and the variations include different output formats like Python list, JSON, and YAML, minor perturbations such as adding spaces or thank you notes, and commonly used jailbreak techniques.

Results

Sensitivity to Prompt Variations

The findings indicate that minor prompt variations significantly impact model predictions. For instance, specifying an output format leads to changes in a minimum of 10% of predictions. Notably, even adding spaces or common greetings to prompts alters a considerable number of predictions. The paper observes that larger models tend to be more robust to these variations, highlighting the propensity of smaller models to rely more on spurious correlations.

Figure 1: Number of predictions that change (out of 11,000) compared to No Specified Format style. Red bars correspond to the number of invalid responses provided by the model.

Impact on Model Accuracy

Table \ref{tab:multi-model-accuracy-scores} from the paper showcases how different prompt strategies affect model accuracy. While output formats significantly impact accuracy, no single format consistently outperforms others across all tasks. JSON and Python list formats typically yield higher accuracy, but models show variability in performance based on the complexity and nature of tasks.

Prediction Similarity

The paper employs multidimensional scaling (MDS) to visualize prediction similarity across variations. Clusters formed by perturbations indicate semantic similarity in outputs, while jailbreaks, due to their directive nature, result in a wider distribution of predictions.

Annotator Agreement Correlation

Interestingly, there is a counterintuitive weak negative correlation between the entropy of annotator predictions (a measure of annotator disagreement) and instances that change predictions when prompt variations are applied. This suggests that the changes in predictions do not purely arise from the intrinsic difficulty of the task as perceived by human annotators.

Implications and Future Work

The results imply that practitioners should be wary of over-reliance on LLMs for sensitive applications where minor changes in prompt construction could lead to significant variability in model outputs. Identifying stable prompt constructs and understanding their relationship to model performance is crucial for deploying LLMs reliably.

Future work could explore developing LLMs resilient to such perturbations and investigating the interplay between prompt construction and model architecture to enhance robustness. This research opens avenues for refining prompt engineering practices, which could mitigate the impact of prompt sensitivity in LLM deployment.

Conclusion

The paper highlights a critical aspect of LLM performance variability under prompt variations, calling for a deeper understanding of prompt constructions. By documenting the butterfly effect of minor alterations, it provides valuable insights for improving the stability and reliability of LLM applications across real-world tasks.