Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer | Metaglossia: The Translation World | Scoop.it

"Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer


Mondal, Soumen Kumar ; Sen, Sayambhu ; Singhania, Abhishek ; Jyothi, Preethi


Abstract
Multilingual large language models (LLMs) aim towards robust natural language understanding across diverse languages, yet their performance significantly degrades on low-resource languages. This work explores whether existing techniques to identify language-specific neurons can be leveraged to enhance cross-lingual task performance of lowresource languages. We conduct detailed experiments covering existing language-specific neuron identification techniques (such as Language Activation Probability Entropy and activation probability-based thresholding) and neuron-specific LoRA fine-tuning with models like Llama 3.1 and Mistral Nemo. We find that such neuron-specific interventions are insufficient to yield cross-lingual improvements on downstream tasks (XNLI, XQuAD) in lowresource languages. This study highlights the challenges in achieving cross-lingual generalization and provides critical insights for multilingual LLMs.


Publication:
eprint arXiv:2503.17456
Pub Date: March 2025 DOI:
10.48550/arXiv.2503.17456


arXiv: arXiv:2503.17456  Bibcode:  Keywords:
Computation and Language; Artificial Intelligence; Machine Learning
E-Print Comments: Accepted (oral) at NAACL 2025 (InsightsNLP)"
https://ui.adsabs.harvard.edu/abs/2025arXiv250317456M/abstract
#Metaglossia
#metaglossia_mundus