What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models

Abstract: Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale LLMs. However, these techniques have received limited attention in the context of low-resource LLMs, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data LLM, AfriBERTa. Through a battery of experiments, we assess the effects of compression on performance across several metrics beyond accuracy. Our study provides evidence that compression techniques significantly improve the efficiency and effectiveness of small-data LLMs, confirming that the prevailing beliefs regarding the effects of compression on large, heavily parameterized models hold true for less-parameterized, small-data models.