Reward Modeling for Mitigating Toxicity in Transformer-based Language Models (2202.09662v6)

Abstract: Transformer-based LLMs are able to generate fluent text and be efficiently adapted across various natural language generation tasks. However, LLMs that are pretrained on large unlabeled web text corpora have been shown to suffer from degenerating toxic content and social bias behaviors, consequently hindering their safe deployment. Various detoxification methods were proposed to mitigate the LLM's toxicity; however, these methods struggled to detoxify LLMs when conditioned on prompts that contain specific social identities related to gender, race, or religion. In this study, we propose Reinforce-Detoxify; A reinforcement learning-based method for mitigating toxicity in LLMs. We address the challenge of safety in LLMs and propose a new reward model that is able to detect toxic content and mitigate unintended bias towards social identities in toxicity prediction. The experiments demonstrate that the Reinforce-Detoxify method for LLM detoxification outperforms existing detoxification approaches in automatic evaluation metrics, indicating the ability of our approach in LLM detoxification and less prone to unintended bias toward social identities in generated content.