NLP at UC Santa Cruz at SemEval-2024 Task 5: Legal Answer Validation using Few-Shot Multi-Choice QA (2404.03150v1)
Abstract: This paper presents our submission to the SemEval 2024 Task 5: The Legal Argument Reasoning Task in Civil Procedure. We present two approaches to solving the task of legal answer validation, given an introduction to the case, a question and an answer candidate. Firstly, we fine-tuned pre-trained BERT-based models and found that models trained on domain knowledge perform better. Secondly, we performed few-shot prompting on GPT models and found that reformulating the answer validation task to be a multiple-choice QA task remarkably improves the performance of the model. Our best submission is a BERT-based model that achieved the 7th place out of 20.
- Longformer: The Long-Document Transformer.
- The Legal Argument Reasoning Task in Civil Procedure.
- Language Models are Few-Shot Learners.
- Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4.
- LEGAL-BERT: The Muppets straight out of Law School. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904, Online. Association for Computational Linguistics.
- LeXFfiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15513–15535, Toronto, Canada. Association for Computational Linguistics.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
- Glannon, J. W. (2019). The Glannon Guide to Civil Procedure. Wolters Kluwer, New York, NY, 4 edition.
- A Free Format Legal Question Answering System. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 107–113, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- OpenAI (2023a). GPT-3.5 Model Documentation. https://platform.openai.com/docs/models/gpt-3-5-turbo. Accessed: 2024-02-05.
- OpenAI (2023b). GPT-4 Model Documentation. https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo. Accessed: 2024-02-05.
- Boosting methods for multi-class imbalanced data classification: an experimental review. Journal of Big Data, 7:1–47.
- A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT.
- When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.