OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. Several publications on ChatGPT evaluation test its effectiveness on well-known NLP tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT's capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool's usefulness to society and how the learning and validation procedures for such systems should be established.
The paper evaluates ChatGPT's understanding across 25 varied analytical NLP tasks ranging from semantics to pragmatics.
Researchers used automated testing to produce over 49,000 responses from ChatGPT, assessing them against SOTA models.
ChatGPT showed an average quality drop of 25% compared to SOTA, with larger drops in more complex tasks, especially those needing pragmatic understanding.
Personalization of responses led to improved outcomes, but the study also found biases within ChatGPT's responses.
The paper concludes that despite ChatGPT's broad abilities, it requires further development to match specialized SOTA models and alleviate biases.
OpenAI's Generative Pre-trained Transformer (ChatGPT) is an AI system designed to provide detailed and precise answers across various domains. Several studies have tested ChatGPT's effectiveness on well-established NLP tasks. However, most have not leveraged automated evaluation and were limited in scope. Researchers from Wrocław University of Science and Technology in Poland delved into extensive testing of ChatGPT's range and depth of understanding using a diverse set of analytical NLP tasks.
The study evaluated ChatGPT on 25 diverse analytical NLP tasks across the spectrum of semantics and pragmatics. This involved tasks like word sense disambiguation, question answering, sentiment analysis, and pragmatic issues like emotion recognition. ChatGPT was automated and prompted to produce over 49,000 responses, which were compared against state-of-the-art (SOTA) solutions.
The findings revealed a varying performance by ChatGPT, with an average loss in quality of approximately 25% when compared to SOTA models. Notably, the more difficult the task (as indicated by lower SOTA performance), the higher the performance drop in ChatGPT's results, particularly for tasks requiring pragmatic understanding, such as emotion recognition.
The research also tested the model's ability to personalize responses to selected subjective tasks, which resulted in better predictions tailored to individual user preferences. However, additional qualitative analysis uncovered biases within ChatGPT, likely due to rules imposed on human trainers by OpenAI. This highlights the intrinsic challenge in balancing neutrality with contextual accuracy.
The results reveal that while ChatGPT shows significant abilities in a broad range of NLP tasks, it is not yet on par with specialized SOTA solutions. It holds promise as an AI tool that could potentially support various applications in society, provided it further hones its learning and validation procedures.
The outcomes from this study provide valuable insights into the capabilities and areas for improvement of language models like ChatGPT. It suggests a need for continued research into making such models more robust, unbiased, and contextually sensitive to further their applicability and usefulness in real-world scenarios.