Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models (2307.16180v1)

Abstract: The field of LLMs has made significant progress, and their knowledge storage capacity is approaching that of human beings. Furthermore, advanced techniques, such as prompt learning and reinforcement learning, are being employed to address ethical concerns and hallucination problems associated with LLMs, bringing them closer to aligning with human values. This situation naturally raises the question of whether LLMs with human-like abilities possess a human-like personality? In this paper, we aim to investigate the feasibility of using the Myers-Briggs Type Indicator (MBTI), a widespread human personality assessment tool, as an evaluation metric for LLMs. Specifically, extensive experiments will be conducted to explore: 1) the personality types of different LLMs, 2) the possibility of changing the personality types by prompt engineering, and 3) How does the training dataset affect the model's personality. Although the MBTI is not a rigorous assessment, it can still reflect the similarity between LLMs and human personality. In practice, the MBTI has the potential to serve as a rough indicator. Our codes are available at https://github.com/HarderThenHarder/transformers_tasks/tree/main/LLM/llms_mbti.

• The paper finds that LLMs exhibit distinct MBTI personality types, such as ENTJ for ChatGPT and INTJ for GPT-4, demonstrating inherent personality-like traits.
• The research shows that prompt engineering, through explicit and implicit cues, can effectively modify the MBTI profiles of LLMs.
• The study reveals that training data significantly impacts MBTI outcomes, influencing reasoning and adaptability through varied corpora.

Investigating Personality in LLMs through MBTI

Introduction

The paper "Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for LLMs" explores the applicability of the Myers-Briggs Type Indicator (MBTI) for assessing the personality-like traits of LLMs. By leveraging the MBTI, a prevalent human personality assessment tool, the research investigates whether LLMs, similar to humans, manifest personality traits and how these traits can be influenced by model-specific factors such as training data and prompt engineering.

(Figure 1)

Figure 1: Personality Test of Human and LLMs. LLMs with human-like capabilities can exhibit human-like personalities.

Methodology

The research focuses on several key areas: determining the inherent personality types of different LLMs, examining the potential for altering these personality types through prompt engineering, and assessing the influence of training datasets on the personality traits of LLMs. The methodology involves conducting extensive experiments using a selection of well-known LLMs, including ChatGPT and GPT-4, to evaluate their MBTI profiles.

The evaluation process utilizes the MBTI test, consisting of 93 multiple-choice questions. Answers are generated by analyzing the probability values of the final tokens for each option. The dichotomies assessed include extraversion vs. introversion, sensing vs. intuition, thinking vs. feeling, and judging vs. perceiving, resulting in 16 possible MBTI personality types.

Results

LLM Personalities

The paper reveals that LLMs do exhibit distinct personality-like MBTI types. For instance, ChatGPT identifies as ENTJ, characterized by assertiveness and leadership skills, while GPT-4 is labeled as INTJ, known for its expertise and analytical thinking. However, the consistency of personality traits across different models can vary.

Influencing Personality

Through prompt engineering, the paper explores the potential to alter the MBTI profiles of LLMs. Explicit prompts, like role-playing scenarios, and implicit prompts, such as few-shot learning examples, demonstrate varying levels of influence. Model instruction tuning significantly impacts a model's ability to adjust its perceived personality, with well-instructed models like ChatGPT showing greater adaptability.

Training Data Impact

The type of training corpus significantly affects the model’s MBTI outcome, particularly in thinking/feeling (T/F) and judging/perceiving (J/P) dimensions. For example, training on examination corpora enhances models' reasoning capabilities, thereby increasing their T-value, while question-answer corpora elevate adaptability and flexibility, impacting P-values.

Figure 2: Specific scores for each dichotomy among different LLMs.

Discussion

The research highlights the MBTI's utility as a rough indicator for evaluating LLMs, despite its limitations as a pseudo-scientific tool. The T/F and J/P dimensions provide valuable insights into models' reasoning and planning capabilities. These findings suggest that LLMs might benefit from tailored training in specific dichotomies that align with desired application outcomes, such as advanced reasoning or task planning.

Conclusion

While acknowledging the MBTI's limitations, the paper asserts that LLMs possess personality-like traits comparable to those outlined by human psychological assessments. The MBTI serves as a valuable tool for uncovering these traits, with practical implications for enhancing model adaptability and alignment with human values. Future work will likely explore additional training corpora and tasks, focusing on areas like commonsense comprehension and reasoning to refine LLM personality assessments.

By understanding and potentially guiding the personality attributes of LLMs, researchers and practitioners can better align these models with human-centric applications, potentially facilitating more seamless human-AI interactions and collaborations.