How do Large Language Models Handle Multilingualism? (2402.18815v3)

Abstract: LLMs have demonstrated impressive capabilities across diverse languages. This study explores how LLMs handle multilingualism. Based on observed language ratio shifts among layers and the relationships between network structures and certain capabilities, we hypothesize the LLM's multilingual workflow ($\texttt{MWork}$): LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures, respectively. In the final layers, LLMs generate responses aligned with the original language of the query. To verify $\texttt{MWork}$, we introduce Parallel Language-specific Neuron Detection ($\texttt{PLND}$) to identify activated neurons for inputs in different languages without any labeled data. Using $\texttt{PLND}$, we validate $\texttt{MWork}$ through extensive experiments involving the deactivation of language-specific neurons across various layers and structures. Moreover, $\texttt{MWork}$ allows fine-tuning of language-specific neurons with a small dataset, enhancing multilingual abilities in a specific language without compromising others. This approach results in an average improvement of $3.6\%$ for high-resource languages and $2.3\%$ for low-resource languages across all tasks with just $400$ documents.

• The paper introduces a three-stage multilingual workflow (MWork) that converts inputs to English, processes tasks, and generates responses in the original language.
• It validates the workflow using Parallel Language-specific Neuron Detection (PLND), showing that deactivating language-specific neurons impairs performance in non-English languages.
• Fine-tuning these neurons with limited contextual examples improves multilingual capabilities by 3.6% for high-resource and 2.3% for low-resource languages.

LLMs handle multilingualism through a structured process that leverages different layers of their architecture for understanding, processing, and generating text in multiple languages. This process, as detailed in "How do LLMs Handle Multilingualism?" (How do Large Language Models Handle Multilingualism?, 29 Feb 2024), involves a multilingual workflow (MWork) and is validated using Parallel Language-specific Neuron Detection (PLND).

Multilingual Workflow (MWork) in LLMs

The MWork framework posits that LLMs manage multilingual inputs via three distinct stages, each localized to specific layers within the model:

• Understanding (Initial Layers): The initial layers are responsible for converting multilingual inputs into a unified representation, effectively translating them into English for subsequent processing. This translation facilitates a common ground for task-solving, irrespective of the input language.
• Task-Solving (Intermediate Layers): The intermediate layers primarily operate in English, utilizing self-attention and feed-forward networks. Self-attention mechanisms are employed for reasoning, while feed-forward networks integrate multilingual knowledge to enrich the factual content. This stage is pivotal for the LLM's ability to "think" and derive solutions.
• Response Generation (Final Layers): The final layers generate responses in the original language of the query. This involves translating the English-centric thought process back into the user's language, ensuring coherent and contextually relevant outputs.

Parallel Language-specific Neuron Detection (PLND)

To empirically validate the MWork framework, the paper employs PLND, a novel method for identifying and quantifying the significance of individual neurons in relation to the input language, without relying on explicit task labels. PLND involves feeding a free text corpus of a specific language into the model and isolating the neurons that consistently activate.

The PLND method is mathematically defined for both Feed-Forward and Self-Attention layers. For the Feed-Forward Layer in Llama2, the importance of a neuron is quantified as the difference in the output when the specific neuron of $W_{up}$ is either activated or deactivated, calculated efficiently in parallel using a diagonal mask matrix. Similarly, for the Self-Attention Layer, the importance is calculated by measuring the difference in the attention weight when a specific neuron in $W_Q$ or $W_K$ is deactivated, also enabling parallel computation.

Empirical Validation Through Neuron Deactivation

The paper provides empirical evidence by deactivating language-specific neurons in different layers and observing the impact on performance:

• Understanding Layer: Deactivating language-specific neurons in the understanding layer significantly impairs performance in non-English languages while maintaining English performance. This observation supports the hypothesis that these layers are crucial for processing and translating non-English inputs.
• Task-Solving Layer: Deactivating language-specific neurons in the task-solving layers reduces performance across all languages, including English. This result corroborates the idea that the task-solving process heavily depends on English. Disabling the self-attention structure impairs the ability to solve tasks across all languages, whereas deactivating language-specific neurons within the feed-forward structure predominantly affects non-English languages.
• Generation Layer: Deactivating language-specific neurons in the generation layer affects the model's ability to generate outputs in non-English languages, as expected.

Furthermore, the paper reveals that languages from the same family tend to exhibit a higher degree of overlap in their language-specific neurons. English neurons show limited overlap with other languages, underscoring the predominant role of English-specific neurons within the model.

Fine-tuning Language-Specific Neurons

The paper demonstrates that fine-tuning language-specific neurons with a small number of contextual examples can enhance the multilingual capabilities of LLMs. This targeted fine-tuning results in performance improvements, particularly in multilingual understanding and generation. The results show an average improvement of $3.6\%$ for high-resource languages and $2.3\%$ for low-resource languages across all tasks with just $400$ documents.

In summary, LLMs process multilingual inputs by converting them into a unified representation (often English) in the initial layers, leveraging English for task-solving in the intermediate layers, and generating responses in the original language in the final layers. Techniques like PLND enable the identification and manipulation of language-specific neurons, offering insights into the multilingual capabilities of LLMs and enabling targeted fine-tuning for enhanced performance.