Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (CxG). For example, the caused-motion construction (CMC, She sneezed the foam off her cappuccino'') demonstrates that constructions must carry meaning, otherwise the fact thatsneeze'' in this context causes movement cannot be explained. We form the hypothesis that this remains challenging even for state-of-the-art LLMs, for which we devise a test based on substituting the verb with a prototypical motion verb. To be able to perform this test at statistically significant scale, in the absence of adequate CxG corpora, we develop a novel pipeline of NLP-assisted collection of linguistically annotated text. We show how dependency parsing and GPT-3.5 can be used to significantly reduce annotation cost and thus enable the annotation of rare phenomena at scale. We then evaluate GPT, Gemini, Llama2 and Mistral models for their understanding of the CMC using the newly collected corpus. We find that all models struggle with understanding the motion component that the CMC adds to a sentence.
The paper presents a method for constructing a corpus on the Caused Motion Construction (CMC) and evaluating LLMs' understanding of this rare linguistic phenomenon.
A novel pipeline combining human expertise and GPT-3.5 is introduced for efficient data collection and annotation of CMC instances.
An evaluation of several LLMs, including GPT, Gemini, Llama2, and Mistral models, shows significant gaps in their ability to interpret the CMC.
The study offers a cost-efficient hybrid human-LLM pipeline for collecting rare linguistic phenomena and outlines future directions for improving LLMs' comprehension.
Rare linguistic phenomena often elude the grasp of LLMs, presenting a unique challenge to both computational linguistics and the development of more nuanced, understanding AI systems. This paper presents a methodological innovation for both constructing a corpus centered on a rare linguistic structure known as the Caused Motion Construction (CMC) and evaluating various LLMs' capability to comprehend this structure. Through a hybrid pipeline combining human linguistic expertise, NLP tools, and the advanced capabilities of GPT-3.5, this study not only proposes a cost-efficient approach to annotating rare linguistic phenomena at scale but also critically assesses the current state-of-the-art LLMs against the backdrop of grammatical constructions that require a deeper semantic understanding.
The paper introduces a novel pipeline for data collection that significantly reduces the annotation burden typically associated with rare linguistic phenomena. This is particularly relevant for the CMC, which involves verbs that are conventionally intransitive, taking on a transitive role and implying motion or displacement as a result of the action.
Key to this methodology is an initial filtering phase using dependency parsing to identify potential CMC instances, followed by a refinement phase through GPT-3.5, which classifies these instances with an instructional prompt specifically tailored to identify the CMC. The dual-stage filtration considerably concentrates the density of CMC instances, and the final dataset comprises both manually verified instances and a larger, semi-automatically annotated corpus. The process involves intricate prompt design and engineering to optimize both the accuracy and the cost-efficiency of the LLM-assisted classification.
The primary focus of the evaluation is to ascertain whether various LLMs, including GPT, Gemini, Llama2, and Mistral models, can accurately interpret the CMC, which lies at the intersection of syntax and semantics. A specialized evaluation setup presents sentences containing the CMC to these models and questions whether the direct object in each sentence is indeed moving, an implicit understanding necessary to correctly interpret CMC instances. The accuracy rates reported reveal substantial gaps in the models' understanding, highlighting a critical area for future work on model training and development.
This study makes several significant contributions:
The methodology and findings underscore the inherent challenges and potential pathways for advancing the understanding capabilities of LLMs. Future research directions include extending this hybrid annotation and evaluation framework to other rare linguistic phenomena and exploring advancements in LLM architectures and training paradigms to enhance their grasp of complex linguistic constructions.
This paper sheds light on the intricacies involved in handling and understanding rare linguistic phenomena by contemporary LLMs. Its methodology and findings contribute valuable insights to the computational linguistics community, providing a clear direction for future research aimed at enhancing the semantic comprehension of LLMs.