Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
124 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CulturePark: Boosting Cross-cultural Understanding in Large Language Models (2405.15145v3)

Published 24 May 2024 in cs.AI, cs.CL, and cs.MA

Abstract: Cultural bias is pervasive in many LLMs, largely due to the deficiency of data representative of different cultures. Typically, cultural datasets and benchmarks are constructed either by extracting subsets of existing datasets or by aggregating from platforms such as Wikipedia and social media. However, these approaches are highly dependent on real-world data and human annotations, making them costly and difficult to scale. Inspired by cognitive theories on social communication, this paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. CulturePark simulates cross-cultural human communication with LLM-based agents playing roles in different cultures. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. Using CulturePark, we generated 41,000 cultural samples to fine-tune eight culture-specific LLMs. We evaluated these models across three downstream tasks: content moderation, cultural alignment, and cultural education. Results show that for content moderation, our GPT-3.5-based models either match or outperform GPT-4 on datasets. Regarding cultural alignment, our models surpass GPT-4 on Hofstede's VSM 13 framework. Furthermore, for cultural education of human participants, our models demonstrate superior outcomes in both learning efficacy and user experience compared to GPT-4. CulturePark proves an important step in addressing cultural bias and advancing the democratization of AI, highlighting the critical role of culturally inclusive data in model training. Code is released at https://github.com/Scarelette/CulturePark.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (80)
  1. Hofstede. Vsm13. https://geerthofstede.com/research-and-vsm/vsm-2013/, 2013.
  2. What is culture. A compilation of quotations. GlobalPAD Core Concepts, 1(22):1–21, 2012.
  3. Prompt distillation for efficient llm-based recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 1348–1357, 2023.
  4. Recommender systems in the era of large language models (llms). arXiv preprint arXiv:2307.02046, 2023.
  5. Social skill training with large language models. arXiv preprint arXiv:2404.04204, 2024.
  6. Rehearsal: Simulating conflict to teach conflict resolution. arXiv preprint arXiv:2309.12309, 2023.
  7. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv preprint arXiv:2309.08591, 2023.
  8. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. arxiv. Preprint posted online on March, 31, 2023.
  9. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. arXiv preprint arXiv:2309.12342, 2023.
  10. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14456, 2023.
  11. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. arXiv preprint arXiv:2310.12481, 2023.
  12. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785, 2022.
  13. Unintended impacts of llm alignment on global representation. arXiv preprint arXiv:2402.15018, 2024.
  14. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870, 2023.
  15. Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in llms. arXiv preprint arXiv:2310.07251, 2023.
  16. Sabiá: Portuguese large language models. In Brazilian Conference on Intelligent Systems, pages 226–240. Springer, 2023.
  17. Harmonizing global voices: Culturally-aware models for enhanced content moderation. arXiv preprint arXiv:2312.02401, 2023.
  18. Seallms–large language models for southeast asia. arXiv preprint arXiv:2312.00738, 2023.
  19. Typhoon: Thai large language models. arXiv preprint arXiv:2312.13951, 2023.
  20. Persianllama: Towards building first persian large language model. arXiv preprint arXiv:2312.15713, 2023.
  21. Taiwan llm: Bridging the linguistic divide with a culturally aligned language model. arXiv preprint arXiv:2311.17487, 2023.
  22. Culturellm: Incorporating cultural differences into large language models. arXiv preprint arXiv:2402.10946, 2024.
  23. Margarita Limón. On the cognitive conflict as an instructional strategy for conceptual change: A critical appraisal. Learning and instruction, 11(4-5):357–380, 2001.
  24. Cognitive conflict and goal conflict effects on task performance. Organizational behavior and human performance, 19(2):378–391, 1977.
  25. Social cognition. Mcgraw-Hill Book Company, 1991.
  26. Michael Minkov Geert Hofstede, Gert Jan Hofstede. Cultures and Organizations: Software of the Mind, Third Edition. McGraw Hill Professional, https://books.google.co.uk/books?id=7bYWmwEACAAJ, 2010.
  27. World Values Survey. World values survey. https://www.worldvaluessurvey.org/wvs.jsp, 2022.
  28. Cas Mudde. The 2012 stein rokkan lecture: Three decades of popu list radical right parties in western europe: so what? In The Populist Radical Right, pages 545–558. Routledge, 2016.
  29. Normsage: Multi-lingual multi-cultural norm discovery from conversations on-the-fly. arXiv preprint arXiv:2210.08604, 2022.
  30. Worldvaluesbench: A large-scale benchmark dataset for multi-cultural value awareness of language models. arXiv preprint arXiv:2404.16308, 2024.
  31. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404.12464, 2024.
  32. Massively multi-cultural knowledge acquisition & lm benchmarking. arXiv preprint arXiv:2402.09369, 2024.
  33. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907–1917, 2023.
  34. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. arXiv preprint arXiv:2404.15238, 2024.
  35. Mala-500: Massive language adaptation of large language models. arXiv preprint arXiv:2401.13303, 2024.
  36. Pew Global Attitudes Survey. Pew global attitudes survey. https://www.pewresearch.org/, 2022.
  37. OpenAI. text-embedding-3-small. https://platform.openai.com/docs/guides/embeddings, 2024.
  38. OpenAI. Chatgpt. https://chat.openai.com/, 2023.
  39. OpenAI. Gpt-4 technical report, 2023.
  40. Google. Gemini. https://deepmind.google/technologies/gemini/#introduction, 2023.
  41. Situated learning and education. Educational researcher, 25(4):5–11, 1996.
  42. Situated learning: Legitimate peripheral participation. Cambridge university press, 1991.
  43. Jeff Bilmes. Submodularity in machine learning and artificial intelligence. arXiv preprint arXiv:2202.00132, 2022.
  44. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
  45. Overview of osact5 shared task on arabic offensive language and hate speech detection. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 162–166, 2022.
  46. A multi-platform arabic news comment dataset for offensive language detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6203–6212, 2020.
  47. Developing a multilingual annotated corpus of misogyny and aggression. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pages 158–168, Marseille, France, May 2020. European Language Resources Association (ELRA).
  48. Hate speech detection in the bengali language: A dataset and its baseline evaluation. In Proceedings of International Joint Conference on Advances in Computational Intelligence: IJCACI 2020, pages 457–468. Springer, 2021.
  49. aimansnigdha. Bangla-abusive-comment-dataset. https://github.com/aimansnigdha/Bangla-Abusive-Comment-Dataset, 2018.
  50. Detect camouflaged spam content via stoneskipping: Graph and text joint embedding for chinese character variation representation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP2019). ACM, 2019.
  51. Towards identifying social bias in dialog systems: Frame, datasets, and benchmarks. arXiv preprint arXiv:2202.08011, 2022.
  52. Overview of the germeval 2018 shared task on the identification of offensive language. 2018.
  53. Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis. In Michael Beißwenger, Michael Wojatzki, and Torsten Zesch, editors, Proceedings of NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication, volume 17 of Bochumer Linguistische Arbeitsberichte, pages 6–9, Bochum, sep 2016.
  54. HASOC. Hasoc2020. https://hasocfire.github.io/hasoc/2020/index.html, 2020.
  55. Multilingual hatecheck: Functional tests for multilingual hate speech detection models. arXiv preprint arXiv:2206.09917, 2022.
  56. BEEP! Korean corpus of online news comments for toxic speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 25–31, Online, July 2020. Association for Computational Linguistics.
  57. daanVeer. Korean hatespeech dataset. https://github.com/daanVeer/HateSpeech_dataset, 2020.
  58. I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6193–6202, 2020.
  59. A large-scale comprehensive abusiveness detection dataset with multifaceted labels from reddit. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 552–561, 2021.
  60. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93, 2016.
  61. Offensive comments in the brazilian web: a dataset and baseline results. 2017.
  62. HateBR: A large expert annotated corpus of Brazilian Instagram comments for offensive language and hate speech detection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7174–7183, Marseille, France, June 2022. European Language Resources Association.
  63. Toxic language detection in social media for brazilian portuguese: New dataset and multilingual analysis. arXiv preprint arXiv:2010.04543, 2020.
  64. Overview of the task on automatic misogyny identification at ibereval 2018. Ibereval@ sepln, 2150:214–228, 2018.
  65. Overview of mex-a3t at ibereval 2018: Authorship and aggressiveness analysis in mexican spanish tweets. In Notebook papers of 3rd sepln workshop on evaluation of human language technologies for iberian languages (ibereval), seville, spain, volume 6, 2018.
  66. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th international workshop on semantic evaluation, pages 54–63, 2019.
  67. Angel Felipe Magnossao de Paula and Ipek Baris Schlicht. Ai-upv at iberlef-2021 detoxis task: Toxicity detection in immigration-related web news comments using transformers and statistical models. arXiv preprint arXiv:2111.04530, 2021.
  68. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). arXiv preprint arXiv:2006.07235, 2020.
  69. Kaggle. turkish offensive language detection. https://www.kaggle.com/datasets/toygarr/turkish-offensive-language-detection, 2022.
  70. Detecting abusive instagram comments in turkish using convolutional neural network and machine learning methods. Expert Systems with Applications, 174:114802, 2021.
  71. Turkish Spam V01. UCI Machine Learning Repository, 2019. DOI: https://doi.org/10.24432/C5WG7F.
  72. Çağrı Çöltekin. A corpus of turkish offensive language on social media. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6174–6184, Marseille, France, 2020.
  73. F Husain. Osact4 shared task on offensive language detection: Intensive preprocessing-based approach. arxiv 2020. arXiv preprint arXiv:2005.07297, 2020.
  74. Dataset of arabic spam and ham tweets. Data in Brief, 52(10990):4, 2024.
  75. Tackling cyber-aggression: Identification and fine-grained categorization of aggressive texts on social media using weighted ensemble of transformers. Neurocomputing, 490:462–481, 2022.
  76. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv 2307.09705, 2023.
  77. K-MHaS: A multi-label hate speech detection dataset in Korean online news comment. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3530–3538, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
  78. Offendes: A new corpus in spanish for offensive language research. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1096–1108, 2021.
  79. Detecting and monitoring hate speech in twitter. Sensors, 19(21):4654, 2019.
  80. Kaggle. 5k turkish tweets with incivil content. https://www.kaggle.com/datasets/kbulutozler/5k-turkish-tweets-with-incivil-content, 2021.
Citations (1)
View on Semantic Scholar

Summary

  • The paper presents CulturePark, a multi-agent system that generates culturally diverse data to fine-tune LLMs and mitigate Western-centric bias.
  • It employs self-calibration and self-guidance prompts to maintain high-quality dialogue and achieve cultural alignment across eight different cultures.
  • Experimental results show that the CulturePark models outperform GPT-4 in content moderation and cultural education, highlighting improved cross-cultural understanding.

CulturePark: Advancing Cross-Cultural Understanding in LLMs

Introduction

Cultural bias in LLMs is a well-documented issue, primarily arising from the predominance of English-language data that reflects Western cultural values. Traditional approaches to creating cultural datasets and benchmarks involve either selecting subsets of existing datasets or aggregating data from platforms such as Wikipedia and social media, both methods being resource-intensive and hard to scale. This paper introduces CulturePark, an innovative framework inspired by cognitive theories to address cultural bias in LLMs through simulated cross-cultural dialogues.

CulturePark Framework

CulturePark is an LLM-powered multi-agent communication platform designed to simulate cross-cultural communication between agents representing different cultures. The system operationalizes two primary roles: the main contact, an English-speaking agent, and several cultural delegates from various cultural backgrounds. By facilitating multi-turn dialogues among these agents, CulturePark generates a diverse and detailed dataset encapsulating human beliefs, customs, and norms from different cultures.

The platform employs improved prompting techniques to maintain high-quality dialogues. Self-calibration prompts ensure cultural bias reduction by aligning agent responses with culturally appropriate attitudes derived from seed data. Moreover, communication styles are diversified through self-guidance prompts and natural free chat to mitigate redundancy in responses and inspire novel, comprehensive ideas.

Data Generation and Refinement

CulturePark generated 41,000 cultural samples from an initial dataset of 4,100 seed inputs sourced from the World Values Survey (WVS) and Global Attitudes surveys (GAS). To prepare this data for fine-tuning cultural-specific LLMs, a multi-step data refinement process was employed. The process involves:

  1. Extracting cultural opinions: Opinions from cross-cultural dialogues are extracted via GPT-4.
  2. Fact verification: Extracted opinions are validated for factual accuracy and relevance.
  3. Removing redundancy: Semantic clustering techniques are used to enhance data diversity by removing similar data points.

Experimental Evaluation

Content Moderation

The fine-tuned models were evaluated on content moderation tasks across eight cultures: Arabic, Bengali, Chinese, German, Korean, Portuguese, Spanish, and Turkish. Impressively, the CulturePark-generated models either matched or outperformed GPT-4 on 41 datasets, indicating the effectiveness of the generated data in enhancing model performance on cultural-specific tasks.

Cultural Alignment

Utilizing Hofstede's VSM 13 framework, the culturally fine-tuned models exhibited superior cultural alignment compared to GPT-4. Evaluations on Hofstede’s cultural dimensions revealed that these models achieved significantly lower Euclidean distances, demonstrating enhanced alignment with the cultural values and norms specific to the evaluated cultures.

Situated Learning for Cultural Education

For cultural education, a human paper involving 24 participants validated the efficacy of the fine-tuned models. Participants interacting with CulturePark fine-tuned models showed better learning outcomes and higher satisfaction scores compared to those interacting with GPT-4. This indicates that CulturePark models provide more relevant and detailed cultural insights, thus aiding better academic and practical understanding of different cultures.

Implications and Future Directions

The implications of CulturePark are broad and significant:

  • Practical Implications: CulturePark promotes more inclusive and equitable AI applications by providing culturally sensitive and representative outputs, thereby reducing stereotypes and social biases. This can enhance user trust and adoption of AI systems in diverse cultural contexts.
  • Theoretical Implications: The framework contributes to the body of research on cultural representation in AI and natural language processing, offering a scalable, cost-effective method for generating culturally diverse data. It highlights the importance of multi-agent communication in uncovering and understanding deep-seated cultural norms and beliefs.

Future research could explore integrating more cultures and fine-tuning techniques using other open-source models like Llama2, broadening the scope and impact of CulturePark. Additionally, investigating the interplay between general model capabilities and cultural fine-tuning could yield insights into improving LLMs' general reasoning abilities while maintaining cultural specificity.

Conclusion

CulturePark stands as a promising approach to mitigating cultural bias in LLMs. By leveraging a multi-agent, cognitive conflict-inspired framework, it successfully generates high-quality, diverse cultural data and fine-tunes models that outperform state-of-the-art LLMs in multiple cultural contexts. The potential for enhanced cross-cultural understanding and AI democratization marks CulturePark as a valuable contribution to the field of natural language processing and AI ethics.