Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Science

Abstract: Instruction-tuned LLMs have exhibited impressive language understanding and the capacity to generate responses that follow specific prompts. However, due to the computational demands associated with training these models, their applications often adopt a zero-shot setting. In this paper, we evaluate the zero-shot performance of two publicly accessible LLMs, ChatGPT and OpenAssistant, in the context of six Computational Social Science classification tasks, while also investigating the effects of various prompting strategies. Our experiments investigate the impact of prompt complexity, including the effect of incorporating label definitions into the prompt; use of synonyms for label names; and the influence of integrating past memories during foundation model training. The findings indicate that in a zero-shot setting, current LLMs are unable to match the performance of smaller, fine-tuned baseline transformer models (such as BERT-large). Additionally, we find that different prompting strategies can significantly affect classification accuracy, with variations in accuracy and F1 scores exceeding 10\%.

• The paper demonstrates that carefully designed prompt strategies can significantly enhance the zero-shot performance of LLMs compared to fine-tuned models.
• LLMs like GPT achieved impressive accuracy on tasks such as sarcasm detection and complaint recognition by using techniques like front-loading task descriptions and synonym replacements.
• The study highlights potential data leakage issues from training datasets and stresses the importance of robust evaluation in practical zero-shot classification applications.

Navigating Prompt Complexity for Zero-Shot Classification: A Study of LLMs in Computational Social Science

This paper investigates the performance of LLMs, specifically ChatGPT and OpenAssistant, in the zero-shot classification setting within the field of computational social science. The study focuses on evaluating the efficacy of these models using various prompting strategies on six different classification tasks without task-specific fine-tuning. The analysis also compares LLMs against smaller, fine-tuned models like BERT-large to ascertain their relative performance and capabilities.

Research Questions and Methodology

The paper addresses three primary research questions. Firstly, the authors explore the level of zero-shot performance that LLMs can achieve and compare it to fine-tuned models on similar tasks. Secondly, the study examines which prompting strategies are most effective in improving the performance of LLMs in this context. Thirdly, the authors consider potential data leakage issues related to the training data of the LLMs.

To address these questions, the authors conduct experiments using GPT-3.5-turbo (GPT) and OpenAssistant-LLaMA (LLaMA-OA) models across six tasks. They utilize different prompt strategies, including basic instructions, task and label descriptions (T/L Desc), few-sample examples, and memory recall prompts. Additionally, they test replacing original labels with synonyms to assess the robustness of the models' performances.

Key Findings

Zero-Shot Performance

The study shows that the task-specific fine-tuned BERT-large models generally outperform the LLMs in a zero-shot setting. However, LLMs like GPT, using carefully crafted prompts, can achieve impressive accuracy on complex classification tasks such as sarcasm detection and complaint recognition. GPT tended to exceed other models in zero-shot classification when optimal prompt strategies were applied, such as front-loading the prompt with task descriptions.

Prompting Strategies

The efficacy of different prompting strategies was significant. The simple prompts often outperformed more complex ones, notably showing that excessive detail might dilute model performance by diverting focus away from key instructions. LLMs benefited from carefully chosen synonyms for label names, which occasionally improved performance by preventing overfitting to specific token patterns seen in the model’s training corpus.

Data Leakage and Model Robustness

While the training datasets for LLMs such as GPT and LLaMA-OA are not fully transparent, the elicitation of exact dataset details from the models suggests some exposure to these datasets during training. This could influence the zero-shot performances observed, though the extent and nature of such potential data leakage remain unspecified.

Practical and Theoretical Implications

The paper highlights the challenges and potential of using LLMs in practical applications without task-specific training data. It underscores the importance of selecting prompt strategies carefully to maximize performance. This has implications for deploying LLMs in areas requiring scalable solutions with minimal specific annotations, such as automatic data annotation for large-scale analyses.

Moreover, this study indicates a direction for researchers seeking to employ LLMs for classification tasks in computational social science. As LLMs continue to evolve with more sophisticated architectures and training datasets, their capabilities in zero-shot learning and natural language understanding will likely progress, offering more robust out-of-the-box solutions for complex NLP tasks.

Conclusion

The paper "Navigating Prompt Complexity for Zero-Shot Classification" provides valuable insights into leveraging LLMs like GPT and OpenAssistant for zero-shot classification tasks in computational social science. While current performance levels suggest that fine-tuned models still hold an advantage in accuracy, the strategic design of prompt inputs can significantly enhance LLM performance, offering a viable alternative for applications with limited labeled data. Future work could explore integrating LLMs with other AI systems to improve robustness and the scope of zero-shot tasks further.