MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer

Abstract: The main goal behind state-of-the-art pre-trained multilingual models such as multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer. However, due to limited model capacity, their transfer performance is the weakest exactly on such low-resource languages and languages unseen during pre-training. We propose MAD-X, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations. In addition, we introduce a novel invertible adapter architecture and a strong baseline method for adapting a pre-trained multilingual model to a new language. MAD-X outperforms the state of the art in cross-lingual transfer across a representative set of typologically diverse languages on named entity recognition and causal commonsense reasoning, and achieves competitive results on question answering. Our code and adapters are available at AdapterHub.ml

• The paper introduces MAD-X, a modular adapter framework that significantly improves cross-lingual transfer in low-resource languages by integrating language, task, and invertible adapters.
• The paper details a novel training strategy that employs masked language modeling for language adapters and task-specific fine-tuning for enhanced cross-task adaptability.
• Experiments show MAD-X outperforms models like multilingual BERT and XLM-R, achieving over a 5-point F1 score improvement on the WikiANN NER dataset.

MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer

The paper presents MAD-X, a modular adapter-based framework developed to enhance cross-lingual transfer capabilities in multilingual NLP models. The research primarily focuses on overcoming the limitations of existing multilingual models like multilingual BERT and XLM-R, which exhibit reduced performance when transferring knowledge to low-resource languages or languages not present during pretraining.

Framework Overview

MAD-X introduces a novel approach by incorporating three types of adapters: language adapters, task adapters, and invertible adapters. This modular architecture allows for efficient and targeted adaptation to various tasks and languages, significantly improving transfer performance while minimizing additional parameter overhead.

1. Language Adapters: These adapters are trained using masked language modeling (MLM) on unlabeled data from the target language. They capture language-specific characteristics and can be seamlessly interchanged to facilitate cross-lingual transfer.
2. Task Adapters: Task-specific adapters are employed during downstream fine-tuning. They are trained to encapsulate information pertinent to a particular task, irrespective of language, consequently enhancing task adaptability across diverse linguistic contexts.
3. Invertible Adapters: The introduction of invertible adapters addresses the vocabulary mismatch issue prevalent in multilingual models, especially when adapting to completely new languages. Using Non-linear Independent Component Estimation (NICE), these adapters enable reversible language-specific transformations, optimizing both input and output embeddings.

Experimental Evaluation

The framework was evaluated on three NLP tasks: Named Entity Recognition (NER), Question Answering (QA), and Causal Commonsense Reasoning (CCR). The experiments spanned a typologically diverse set of languages, including those not covered by existing state-of-the-art models. Key findings include:

• Performance Gains: MAD-X consistently outperformed baseline models such as XLM-R and multilingual BERT, particularly in scenarios involving transfer to low-resource and unseen languages. On the WikiANN NER dataset, MAD-X achieved an average F1 score improvement of over 5 points compared to XLM-R.
• Sample Efficiency: The modular design allows for the training of language adapters with relatively fewer iterations on low-resource languages, demonstrating the framework's sample efficiency.
• Model Agnosticism: The experiments showed that MAD-X can be effectively integrated with different pretrained models, including XLM-R at different scales and multilingual BERT. This flexibility highlights its adaptability in utilizing various foundational architectures.

Implications and Future Research

MAD-X's efficient parameter usage offers a promising solution for multilingual model scalability, addressing the constraints posed by the limited capacity of current models. The framework's ability to facilitate robust cross-lingual transfer across diverse tasks and languages could significantly broaden the scope of NLP applications, especially in regions with underrepresented languages.

Future research could explore expanding MAD-X's applicability to more complex tasks and refining the adapter architectures for better handling of languages with unique syntactic or cultural characteristics. Additionally, leveraging related languages' adapters could be a potential area for further enhancing transfer performance to truly low-resource languages.

Overall, the MAD-X framework represents a significant advancement in the field of NLP, offering a scalable and adaptable approach to improving cross-lingual transfer capabilities.