Evaluating Gender Bias in the Translation of Gender-Neutral Languages into English (2311.08836v2)
Abstract: Machine Translation (MT) continues to improve in quality and adoption, yet the inadvertent perpetuation of gender bias remains a significant concern. Despite numerous studies into gender bias in translations from gender-neutral languages such as Turkish into more strongly gendered languages like English, there are no benchmarks for evaluating this phenomenon or for assessing mitigation strategies. To address this gap, we introduce GATE X-E, an extension to the GATE (Rarrick et al., 2023) corpus, that consists of human translations from Turkish, Hungarian, Finnish, and Persian into English. Each translation is accompanied by feminine, masculine, and neutral variants for each possible gender interpretation. The dataset, which contains between 1250 and 1850 instances for each of the four language pairs, features natural sentences with a wide range of sentence lengths and domains, challenging translation rewriters on various linguistic phenomena. Additionally, we present an English gender rewriting solution built on GPT-3.5 Turbo and use GATE X-E to evaluate it. We open source our contributions to encourage further research on gender debiasing.
- Examining covert gender bias: A case study in turkish and english machine translation models.
- Tira Nur Fitria. 2021. Gender bias in translation using google translate: Problems and solution. Language Circle: Journal of Language and Literature, 15(2).
- Sourojit Ghosh and Aylin Caliskan. 2023. Chatgpt perpetuates gender bias in machine translation and ignores non-gendered pronouns: Findings across bengali and five other low-resource languages.
- Melvin Johnson. 2020. A scalable approach to reducing gender bias in google translate.
- James Kuczmarski. 2018. Reducing gender bias in google translate.
- OpenAI. 2022. Introducing chatgpt.
- Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318.
- Good, but not always fair: An evaluation of gender bias for three commercial machine translation systems.
- Assessing gender bias in machine translation – a case study with google translate.
- Gate: A challenge set for gender-ambiguous translation examples. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’23, page 845–854, New York, NY, USA. Association for Computing Machinery.
- Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics.
- They, them, theirs: Rewriting with gender-neutral english.
- No language left behind: Scaling human-centered machine translation.
- NeuTral Rewriter: A rule-based and neural approach to automatic rewriting into gender neutral alternatives. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8940–8948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.