Measuring Gender Bias in West Slavic Language Models (2304.05783v3)

Abstract: Pre-trained LLMs have been known to perpetuate biases from the underlying datasets to downstream tasks. However, these findings are predominantly based on monolingual LLMs for English, whereas there are few investigative studies of biases encoded in LLMs for languages beyond English. In this paper, we fill this gap by analysing gender bias in West Slavic LLMs. We introduce the first template-based dataset in Czech, Polish, and Slovak for measuring gender bias towards male, female and non-binary subjects. We complete the sentences using both mono- and multilingual LLMs and assess their suitability for the masked LLMling objective. Next, we measure gender bias encoded in West Slavic LLMs by quantifying the toxicity and genderness of the generated words. We find that these LLMs produce hurtful completions that depend on the subject's gender. Perhaps surprisingly, Czech, Slovak, and Polish LLMs produce more hurtful completions with men as subjects, which, upon inspection, we find is due to completions being related to violence, death, and sickness.