Revisiting the Reliability of Psychological Scales on Large Language Models
(2305.19926)Abstract
Recent research has extended beyond assessing the performance of LLMs to examining their characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. The administration of personality tests to LLMs has emerged as a noteworthy area in this context. However, the suitability of employing psychological scales, initially devised for humans, on LLMs is a matter of ongoing debate. Our study aims to determine the reliability of applying personality assessments to LLMs, explicitly investigating whether LLMs demonstrate consistent personality traits. Analyzing responses under 2,500 settings reveals that gpt-3.5-turbo shows consistency in responses to the Big Five Inventory, indicating a high degree of reliability. Furthermore, our research explores the potential of gpt-3.5-turbo to emulate diverse personalities and represent various groups, which is a capability increasingly sought after in social sciences for substituting human participants with LLMs to reduce costs. Our findings reveal that LLMs have the potential to represent different personalities with specific prompt instructions. By shedding light on the personalization of LLMs, our study endeavors to pave the way for future explorations in this field. We have made our experimental results and the corresponding code openly accessible via https://github.com/CUHK-ARISE/LLMPersonality.
Overview
-
This paper investigates the reliability of the Big Five Inventory, a psychological scale, when administered to gpt-3.5-turbo, a large language model.
-
The study explores the model's response consistency across 2,500 settings influenced by factors such as item rephrasing and language diversity.
-
A comprehensive framework was designed for this research, analyzing how various components like instruction phrasing and choice presentation affect LLM responses.
-
Findings indicate that gpt-3.5-turbo can accurately represent diverse personality traits, suggesting LLMs’ potential in simulating human-like personality responses.
Evaluating Personality Assessments in LLMs: Insights from the Big Five Inventory on gpt-3.5-turbo
Introduction
The integration of LLMs into a wide array of applications underscores the imperative to understand their behavioral characteristics. A novel area of exploration involves administering psychological scales, initially designed for humans, to LLMs. Amid ongoing debates concerning the suitability of such methodologies, our focused examination reveals notable findings on the application of personality assessments, especially the Big Five Inventory, on gpt-3.5-turbo. This study systematically investigates the reliability of these scales under diverse conditions and explores the model's potential to replicate diverse personality traits effectively.
Examining the Reliability
The reliability of LLM responses to psychological scales is critically examined across various influencing factors: instruction nuances, item rephrasing, language diversity, choice labeling, and choice sequence. Our investigation, spanning 2,500 settings, confirms gpt-3.5-turbo's reliability on the Big Five Inventory, establishing consistency in responses despite the complexities introduced by these variables. Such findings challenge the notion of LLMs' inability to maintain stable personality traits under varied prompts and conditions.
Framework Design and Implementation
This study's methodological robustness stems from its comprehensive framework that dissects the components influencing LLM responses: from the phrasing of instructions and item specificity to the use of multiple languages and the presentation of choices. Notably, this approach uncovered gpt-3.5-turbo’s consistent performance across the spectrum of tests, thereby supporting the model's reliability in psychological assessment contexts.
The Mechanism behind Personality Representation
The exploration extends to how instructional contexts or adjustments can shape the personality portrayals of LLMs. Techniques ranging from environmental cueing to direct personality assignment and character embodiment were employed to assess gpt-3.5-turbo's adaptability. Our findings illustrate the model’s capacity to accurately represent a broad array of personalities, responding distinctly to each manipulation method employed.
Discussion on Methodological Insights and Limitations
The study acknowledges potential limitations arising from modifications to the original scales and the sole focus on the gpt-3.5-turbo model due to resource constraints. Despite these limitations, the research presents a detailed examination of LLMs’ response reliability to psychological scales, contributing a novel perspective to the discourse on the applicability and interpretation of such assessments in non-human intelligences.
Conclusion and Future Directives
Our work underscores gpt-3.5-turbo's ability to demonstrate stable and distinct personality traits as assessed by the Big Five Inventory, affirming the potential of LLMs in simulating human-like personality responses. This research paves the way for future studies to explore broader applications of psychological scales on various LLMs, potentially enhancing the development of AI systems that are not only more relatable but can also accurately mirror the human psychological diversity in digital interactions.
Create an account to read this summary for free: