Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment
(2401.12474)Abstract
Considerable efforts have been invested in augmenting the role-playing proficiency of open-source LLMs by emulating proprietary counterparts. Nevertheless, we posit that LLMs inherently harbor role-play capabilities, owing to the extensive knowledge of characters and potential dialogues ingrained in their vast training corpora. Thus, in this study, we introduce Ditto, a self-alignment method for role-play. Ditto capitalizes on character knowledge, encouraging an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension. This method creates a role-play training set comprising 4,000 characters, surpassing the scale of currently available datasets by tenfold regarding the number of roles. Subsequently, we fine-tune the LLM using this self-generated dataset to augment its role-playing capabilities. Upon evaluating our meticulously constructed and reproducible role-play benchmark and the roleplay subset of MT-Bench, Ditto, in various parameter scales, consistently maintains a consistent role identity and provides accurate role-specific knowledge in multi-turn role-play conversations. Notably, it outperforms all open-source role-play baselines, showcasing performance levels comparable to advanced proprietary chatbots. Furthermore, we present the first comprehensive cross-supervision alignment experiment in the role-play domain, revealing that the intrinsic capabilities of LLMs confine the knowledge within role-play. Meanwhile, the role-play styles can be easily acquired with the guidance of smaller models. We open-source related resources at https://github.com/OFA-Sys/Ditto.
Overview
-
The paper investigates the role-playing abilities of open-source LLMs using a method called ITTOg based on self-alignment.
-
ITTOg enables standard LLMs to adopt role-play character attributes and dialogue styles by aligning responses with character-specific knowledge.
-
The WIKI ROLE dataset was created to train LLMs, featuring a broader range of characters compared to existing datasets.
-
Empirical evidence from the paper shows that LLMs can maintain consistent role identity and have accurate role-specific knowledge after self-alignment.
-
The research demonstrates LLMs' intrinsic role-play capabilities and presents a new approach to evaluation, leveraging LLMs themselves as judges.
Introduction
LLMs have been traditionally seen as task-oriented tools, focusing on understanding intent, following instructions, and problem-solving. However, their role-playing capabilities have largely been inherited from proprietary counterparts that possess more nuanced conversational and emotive expressions. In the paper by Lu et al. from Alibaba Inc., the authors explore the hypothesis that open-source LLMs intrinsically possess role-play attributes that can be harnessed using a method called ITTOg, based on self-alignment.
Self-Alignment for Role-Play
ITTOg operates under the assumption that LLMs, due to their exposure to a massive corpus of dialogues and character knowledge, can simulate role-play dialogues effectively. The technique relies on character knowledge to enable a standard instruction-following LLM to perform as a role-play agent. The approach is scalable; the researchers generated a dataset, WIKI ROLE, with a character scope significantly larger than existing datasets. ITTOg's process comprises collecting character attributes, generating role-specific and contrastive queries, then aligning the LLM responses to fit the character's profile.
Empirical Validation
The benchmarking of ITTOg on various parameter scales shows that the self-aligned LLMs maintain consistent role identity and display accurate role-specific knowledge, rivaling even sophisticated proprietary models. By evaluating the models through a meticulously constructed role-play benchmark and the roleplay subset of the MT-Bench, the authors assert the robustness of their approach. They demonstrate that LLMs can act as judges for their performance, a novel strategy that makes evaluation both reproducible and efficient.
Insights from Role-Play Dissection
The authors also present the first comprehensive cross-supervision alignment experiment in the role-play domain. Findings suggest that while consistent role identity is easily acquired, robust knowledge representation within role-play relies on the intrinsic capabilities of the seed LLMs. The research confirms that the knowledge and conversation styles required for role-playing are already within LLMs and can be fine-tuned to high performance levels. Contributions from ITTOg provide a deeper understanding of LLM's role-play potential, laying out the groundwork for further exploration in role-play AI applications.
Overall, the paper by Lu et al. signifies a substantial advance in harnessing open-source LLMs for role-play, eliminating dependence on proprietary models and broadening the scope of applications for LLMs in interactive scenarios.
Create an account to read this summary for free: