Evaluating and Inducing Personality in Pre-trained Language Models (2206.07550v3)

Abstract: Standardized and quantified evaluation of machine behaviors is a crux of understanding LLMs. In this study, we draw inspiration from psychometric studies by leveraging human personality theory as a tool for studying machine behaviors. Originating as a philosophical quest for human behaviors, the study of personality delves into how individuals differ in thinking, feeling, and behaving. Toward building and understanding human-like social machines, we are motivated to ask: Can we assess machine behaviors by leveraging human psychometric tests in a principled and quantitative manner? If so, can we induce a specific personality in LLMs? To answer these questions, we introduce the Machine Personality Inventory (MPI) tool for studying machine behaviors; MPI follows standardized personality tests, built upon the Big Five Personality Factors (Big Five) theory and personality assessment inventories. By systematically evaluating LLMs with MPI, we provide the first piece of evidence demonstrating the efficacy of MPI in studying LLMs behaviors. We further devise a Personality Prompting (P^2) method to induce LLMs with specific personalities in a controllable way, capable of producing diverse and verifiable behaviors. We hope this work sheds light on future studies by adopting personality as the essential indicator for various downstream tasks, and could further motivate research into equally intriguing human-like machine behaviors.

• The paper introduces a Machine Personality Inventory (MPI) that quantifies Big Five traits in LLMs, mirroring human psychometric evaluations.
• The study presents a Personality Prompting (P²) method to induce specific personality traits in models such as GPT-3.5 and Alpaca.
• Results indicate that LLMs can mimic stable, human-like personality profiles, which paves the way for tailoring AI behavior in practical applications.

Evaluating and Inducing Personality in Pre-trained LLMs

The paper "Evaluating and Inducing Personality in Pre-trained LLMs" (2206.07550) explores the introduction of structured personality traits in LLMs inspired by psychometric evaluations commonly used in human psychology. By leveraging these tools, the authors aim to bring a quantitative, standardized framework to assess and even induce specific personality traits within LLMs.

Introduction to Machine Personality

The research focuses on whether LLMs can exhibit behaviors analogous to human personalities, specifically through the lens of the Big Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism). The paper proposes a novel Machine Personality Inventory (MPI) tailored for LLMs to quantify these traits. The MPI draws from psychometric principles employed in human personality assessment, thus allowing an insight into whether these LLMs can simulate consistent personality-like behaviors.

Methodology

The researchers introduce the Personality Prompting (P²) method to control and induce specific traits in LLMs. The MPI consists of structured personality tests applied to various popular LLMs, such as GPT-3.5 and Alpaca. Using a combination of multiple-choice questions and tailored sentences inspired by human psychometric inventories, the models' responses were scored to determine personality tendencies. This scoring involved assessing the LLMs' tendencies across the Big Five dimensions.

Results and Observations

The evaluation demonstrated that GPT-3.5 and Alpaca closely matched human-like behavior metrics in terms of personality imitation. They showcased stable personality traits, similar to human population statistics along the Big Five dimensions. This validation supports the assertion that LLMs can be systematically evaluated and categorized based on personality, drawing parallel conclusions seen in human psychometry.

Moreover, the P² method successfully induced specific personality traits within LLMs. The models could be guided to produce behaviors aligned with targeted personality traits using controlled prompting methods. This was verified against the designed MPI and vignette tests that simulated real-world scenarios, which were later validated by human evaluators.

Implementation and Practical Implications

Implementing such a framework for personality assessment and induction in LLMs requires thorough understanding and customization of the MPI suited to the LLM's architecture. The paper outlines the potential for personality induction using a sequence of carefully designed prompts that leverage known psychological heuristic keywords.

Deployment of personality-inducing methods such as P² in practical applications can tailor chatbots or digital assistants to exhibit desired personality traits enhancing user interaction. However, this would require extensive calibration and validation against established psychometric properties.

Discussion and Future Prospects

This innovative exploration into LLMs' personality indicates new pathways for research into AI agents exhibiting human-like traits in a controllable and quantifiable way. By tuning LLMs' personalities, future research can explore more complex societal and ethical inquiries, such as trustworthiness in digital communication and emotional intelligence.

While the research establishes a foundational methodology for inducing personalities into LLMs, it also raises potentially critical issues regarding misuse and safety, as AI behaviors could be engineered to manipulate emotional responses in users. Future research should address these ethical implications, ensuring that personality indusion in LLMs is aligned with broader societal values.

Conclusion

The paper breaks new ground in integrating psychometric evaluations into AI research, highlighting the ability to induce and evaluate personality-like traits in LLMs. This interdisciplinary approach between AI and psychology ultimately lays a foundation for more nuanced understanding and integration of AI agents within human sociocultural contexts. As the field advances, further explorations into ethical considerations and application potentials will define how these models can be harmoniously integrated into daily human interactions.