Illuminating the Black Box: A Psychometric Investigation into the Multifaceted Nature of Large Language Models (2312.14202v1)

Abstract: This study explores the idea of AI Personality or AInality suggesting that LLMs exhibit patterns similar to human personalities. Assuming that LLMs share these patterns with humans, we investigate using human-centered psychometric tests such as the Myers-Briggs Type Indicator (MBTI), Big Five Inventory (BFI), and Short Dark Triad (SD3) to identify and confirm LLM personality types. By introducing role-play prompts, we demonstrate the adaptability of LLMs, showing their ability to switch dynamically between different personality types. Using projective tests, such as the Washington University Sentence Completion Test (WUSCT), we uncover hidden aspects of LLM personalities that are not easily accessible through direct questioning. Projective tests allowed for a deep exploration of LLMs cognitive processes and thought patterns and gave us a multidimensional view of AInality. Our machine learning analysis revealed that LLMs exhibit distinct AInality traits and manifest diverse personality types, demonstrating dynamic shifts in response to external instructions. This study pioneers the application of projective tests on LLMs, shedding light on their diverse and adaptable AInality traits.

• The paper introduces AInality as measurable, human-like traits in LLMs through a rigorous psychometric and machine learning framework.
• It employs tests like MBTI, BFI, SD3, and WUSCT alongside advanced prompt engineering to reveal consistent and identifiable personality profiles.
• The study demonstrates that LLMs exhibit adaptable personalities, suggesting opportunities for customized user-AI interactions via prompt modulation.

Exploring AInality in LLMs Through Psychometric Testing

This paper introduces the concept of "AInality" to describe the human-like personality traits exhibited by LLMs. The authors explore whether LLMs possess consistent and identifiable personality traits that can be measured using traditional psychometric tests (2312.14202). The paper employs a combination of prompt engineering and machine learning techniques to investigate the AInality of LLMs, examining its consistency, discernibility, and potential for modulation.

Methodology

The research methodology is structured around the application of psychometric tests, prompt engineering techniques, and machine learning models to analyze the personality traits of LLMs. The paper uses four psychometric tests: the Myers-Briggs Type Indicator (MBTI), the Big Five Inventory (BFI), the Short Dark Triad (SD3), and the Washington University Sentence Completion Test (WUSCT). Prompt engineering techniques, including few-shot prompting, chain-of-thought (CoT) prompting, role-play prompting, and generated knowledge prompting, are used to elicit responses from LLMs. The collected data is then analyzed using machine learning models such as Random Forest, Logistic Regression, SVM, Naive Bayes, and Neural Networks.

Experimental Results and Analysis

The paper's initial experiments aimed to determine whether LLMs exhibit personality traits. The authors found notable variations in the MBTI test results for each user, indicating distinct AInalities exhibited by the LLMs. The BFI test results for ChatGPT and Bard showed mixed alignment with their MBTI types, suggesting that LLM AInality may be more complex than either test can fully capture.

Figure 1: ChatGPT's Big Five Inventory test results, illustrating scores for Extraversion, Emotional Stability, Agreeableness, Conscientiousness, and Intellect.

The SD3 test results indicated that ChatGPT had relatively low scores for Machiavellianism and Psychopathy but a moderate score for Narcissism, while Bard had moderate scores for Machiavellianism and Narcissism and a low score for Psychopathy.

To assess the consistency of AInality, the researchers employed role-play techniques, instructing LLMs to adopt specific MBTI personality types and respond to questionnaires accordingly. LLMs demonstrated a strong understanding of each of the sixteen personality types and the capability to choose reactions in line with the specified personality type.

To identify AInality, the authors gathered MBTI test results from Bard and ChatGPT and utilized machine learning models for classification. The models, including Random Forest, Logistic Regression, and SVM, achieved high accuracy in predicting the LLMs, suggesting that LLMs exhibit consistent AInality traits, even when engaging in role-play scenarios that involve assuming different personalities.

The Washington University Sentence Completion Test (WUSCT) was used to uncover deeper aspects of LLM personality traits. The analysis of the WUSCT responses allowed the researchers to gain insights into the emotional, psychological, and AInality aspects of LLMs, revealing layers that might not be apparent through conventional self-reporting tests.

Implications and Future Directions

The paper's findings have several implications for the development and application of LLMs. The authors demonstrate that LLMs can exhibit diverse and adaptable AInality traits, which can be modulated through prompt engineering. This opens up possibilities for customizing LLMs according to users' unique personality types, enhancing user experiences, and improving human-AI interactions.

The paper also highlights the potential of using projective tests, such as the WUSCT, to gain deeper insights into the AInality structures of LLMs. The authors suggest that future research should focus on developing new AInality psychometric tests specifically tailored for LLMs, targeting their persona characteristics. This could lead to a more comprehensive understanding of LLM behavior and cognition, as well as the development of more user-centric AI systems.

Conclusion

This paper provides a comprehensive psychometric investigation into the multifaceted nature of LLMs, introducing the concept of AInality and demonstrating its discernibility, consistency, characterization, and malleability through prompt engineering (2312.14202). The use of machine learning analysis and the application of projective tests such as the WUSCT represent a significant contribution to the understanding of LLM psychological traits and the potential for creating more personalized and effective AI interactions.