Abstract
The NLP research community has devoted increased attention to languages beyond English, resulting in considerable improvements for multilingual NLP. However, these improvements only apply to a small subset of the world's languages. Aiming to extend this, an increasing number of papers aspires to enhance generalizable multilingual performance across languages. To this end, linguistic typology is commonly used to motivate language selection, on the basis that a broad typological sample ought to imply generalization across a broad range of languages. These selections are often described as being 'typologically diverse'. In this work, we systematically investigate NLP research that includes claims regarding 'typological diversity'. We find there are no set definitions or criteria for such claims. We introduce metrics to approximate the diversity of language selection along several axes and find that the results vary considerably across papers. Crucially, we show that skewed language selection can lead to overestimated multilingual performance. We recommend future work to include an operationalization of 'typological diversity' that empirically justifies the diversity of language samples.
Overview
-
The paper critiques the vague use of 'typological diversity' in NLP research and stresses the importance of a clear definition.
-
A set of metrics is introduced to evaluate language diversity, highlighting limitations in NLP studies that claim linguistic diversity.
-
The authors find that increasing the number of languages studied doesn't necessarily equate to higher typological diversity.
-
Recommendations include the use of metrics like MPSD for better generalizability and reflecting upon the sociocultural implications of expanding NLP applications.
Introduction
The NLP research community predominantly focuses on English, while multilingual NLP has been a secondary concern. Recently, there has been a shift to encompass more languages, with an emphasis on evaluating multilingual model performance across a suitably diverse sample of the world's languages. Considering linguistic diversity is thought to imply robust generalizability, yet the lack of a clear definition of 'typological diversity' in NLP research has been problematic. This paper provides a vital critique of how 'typological diversity' claims are substantiated in NLP literature and offers metrics for assessing the diversity of language selection along several axes.
Survey Methodology
The paper presents a comprehensive analysis of the use of 'typological diversity' in NLP research. To conduct their investigation, the authors formulated a set of metrics to evaluate and define 'typological diversity' across a series of studies. They surveyed NLP literature for claims of typological diversity and scrutinized the language sample justification provided in these studies, if any. Inter-annotator agreement was employed to ensure consistency in scoping the claims about linguistic diversity and dataset introduction. The systematic review covered well-known conferences and journals, and multiple justifications for typological diversity claims were annotated and discussed.
Analysis of Language Diversity
In assessing language diversity, the authors revealed significant variation in 'typological diversity' claims across papers. They proposed using mean pairwise syntactic distance (MPSD) and typological feature inclusion as approximate metrics. The data suggested a discrepancy between multilingual model evaluation and real-world linguistic diversity due to skewed language selections, which often lead to overestimated multilingual performance. The analysis demonstrated that simply adding more languages to a study does not necessarily increase its typological diversity; rather, researchers should consider language selection more carefully to improve the generalizability of their findings.
Recommendations and Implications
The authors advocate for future research to incorporate a defined operational method for typological diversity to avoid skewed generalizations. They recommend documenting language selection and employing measures such as MPSD or typological feature inclusion, enhancing our understanding of linguistic challenges in multilingual NLP modelling. Additionally, it is noted that the developed metrics and tools are approximations, given the incomplete linguistic resources and understanding. The authors proceed with an ethical perspective, emphasizing that expanding NLP applications to numerous languages is not an inherently positive aim without considering the sociocultural impact.
In conclusion, the paper highlights the issue of unsubstantiated 'typological diversity' claims in multilingual NLP literature and the potential pitfalls in assuming generalizability when such diversity is assumed but not empirically justified. It emphasizes the importance of principled reporting on linguistic diversity and the need to refine methodologies for claiming typological diversity. Not only does this have implications for the accuracy of multilingual model performance, but it also affects our understanding of the actual diversity present in the data that such models are tested against.
Create an account to read this summary for free: