Lightweight Adaptation of Neural Language Models via Subspace Embedding (2308.08688v1)

Abstract: Traditional neural word embeddings are usually dependent on a richer diversity of vocabulary. However, the LLMs recline to cover major vocabularies via the word embedding parameters, in particular, for multilingual LLMs that generally cover a significant part of their overall learning parameters. In this work, we present a new compact embedding structure to reduce the memory footprint of the pre-trained LLMs with a sacrifice of up to 4% absolute accuracy. The embeddings vectors reconstruction follows a set of subspace embeddings and an assignment procedure via the contextual relationship among tokens from pre-trained LLMs. The subspace embedding structure calibrates to masked LLMs, to evaluate our compact embedding structure on similarity and textual entailment tasks, sentence and paraphrase tasks. Our experimental evaluation shows that the subspace embeddings achieve compression rates beyond 99.8% in comparison with the original embeddings for the LLMs on XNLI and GLUE benchmark suites.