Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
124 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MetaHate: A Dataset for Unifying Efforts on Hate Speech Detection (2401.06526v1)

Published 12 Jan 2024 in cs.CL and cs.SI

Abstract: Hate speech represents a pervasive and detrimental form of online discourse, often manifested through an array of slurs, from hateful tweets to defamatory posts. As such speech proliferates, it connects people globally and poses significant social, psychological, and occasionally physical threats to targeted individuals and communities. Current computational linguistic approaches for tackling this phenomenon rely on labelled social media datasets for training. For unifying efforts, our study advances in the critical need for a comprehensive meta-collection, advocating for an extensive dataset to help counteract this problem effectively. We scrutinized over 60 datasets, selectively integrating those pertinent into MetaHate. This paper offers a detailed examination of existing collections, highlighting their strengths and limitations. Our findings contribute to a deeper understanding of the existing datasets, paving the way for training more robust and adaptable models. These enhanced models are essential for effectively combating the dynamic and complex nature of hate speech in the digital realm.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Pinpointing Fine-Grained Relationships between Hateful Tweets and Replies. Proceedings of the AAAI 2022, 36(10): 10418–10426.
  2. SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, 54–63. ACL.
  3. The Pushshift Reddit Dataset. Proceedings of the ICWSM 2020, 14: 830–839.
  4. Developing a Multilingual Annotated Corpus of Misogyny and Aggression. In Proceedings of the TRAC 2020, 158–168. ELRA.
  5. Using the Reddit Corpus for Cyberbully Detection, 180–189. Springer. ISBN 9783319754178.
  6. HateMM: A Multi-Modal Dataset for Hate Video Classification. Proceedings of the ICWSM 2023, 17: 1014–1023.
  7. Automated Hate Speech Detection and the Problem of Offensive Language. Proceedings of the ICWSM 2017, 11(1): 512–515.
  8. Hate Speech Dataset from a White Supremacy Forum. In Proceedings of the ALW2 2018. ACL.
  9. Hate Lingo: A Target-Based Linguistic Analysis of Hate Speech in Social Media. Proceedings of the ICWSM 2018, 12(1).
  10. Peer to Peer Hate: Hate Speech Instigators and Their Targets. Proceedings of the ICWSM 2018, 12(1).
  11. Overview of the Evalita 2018 Task on Automatic Misogyny Identification (AMI), 59–66. Accademia University Press. ISBN 9788831978699.
  12. A Hierarchically-Labeled Portuguese Hate Speech Dataset. In Proceedings of the ALW 2019, 94–104. ACL.
  13. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior. Proceedings of the ICWSM 2018, 12(1).
  14. MeTooMA: Multi-Aspect Annotations of Tweets Related to the MeToo Movement. Proceedings of the ICWSM 2020, 14: 209–216.
  15. A Large Labeled Corpus for Online Harassment Research. In Proceedings of the ACM WebSci 2017. ACM.
  16. Hate Towards the Political Opponent: A Twitter Corpus Study of the 2020 US Elections on the Basis of Offensive Speech and Stance Detection. In Proceedings of the WASSA 2021, 171–180. ACL.
  17. Hateful Comment Detection and Hate Target Type Prediction for Video Comments. In Proceedings of the CIKM 2023, CIKM ’23, 3923–3927. ACM. ISBN 9798400701245.
  18. Auditing Elon Musk’s Impact on Hate Speech and Bots. In Proceedings of the ICWSM 2023, 1133–1137. AAAI.
  19. When does a compliment become sexist? Analysis and classification of ambivalent sexism using twitter data. In Proceedings of the Second Workshop on NLP + CSS, 7–16. ACL.
  20. A systematic review on hate speech among children and adolescents: definitions, prevalence, and overlap with related phenomena. Trauma, violence, & abuse, 24(4): 2598–2615.
  21. Introducing the Gab Hate Corpus: defining and applying hate-based rhetoric to social media posts at scale. Language Resources and Evaluation, 56(1): 79–108.
  22. Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application.
  23. Aggression-annotated Corpus of Hindi-English Code-mixed Data. In Proceedings of the LREC 2018. Miyazaki, Japan: ELRA.
  24. Towards a Comprehensive Taxonomy and Large-Scale Annotated Corpus for Online Slur Usage. In Proceedings of the WOAH 2020, 138–149. Online: ACL.
  25. The FRENK Datasets of Socially Unacceptable Discourse in Slovene and English. In Text, Speech, and Dialogue, 103–114. Springer. ISBN 978-3-030-27947-9.
  26. Overview of the HASOC Track at FIRE 2020: Hate Speech and Offensive Language Identification in Tamil, Malayalam, Hindi, English and German. In Proceedings of the FIRE 2020, FIRE 2020. ACM.
  27. Overview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European Languages. In Proceedings of the FIRE 2019, FIRE ’19. ACM.
  28. Spread of Hate Speech in Online Social Media. In Proceedings of the ACM WebSci 2019, WebSci ’19. ACM.
  29. HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection. In Proceedings of the AAAI 2020.
  30. Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages and Conversational Hate Speech. In Proceedings of the FIRE 2021, FIRE 2021. ACM.
  31. A curated dataset for hate speech detection on social media text. Data in Brief, 46: 108832.
  32. Crowdsourcing a Word-Emotion Association Lexicon. Computational Intelligence, 29(3): 436–465.
  33. ETHOS: a multi-label hate speech detection dataset. Complex & Intelligent Systems, 8(6): 4663–4678.
  34. A Measurement Study of Hate Speech in Social Media. In Proceedings of the ACM HT, HT ’17, 85–94. ACM. ISBN 9781450347082.
  35. Nations, U. 2023. What is hate speech? Accessed: 15/11/2023.
  36. Multilingual and Multi-Aspect Hate Speech Analysis. In Proceedings of the EMNLP-IJCNLP 2019, 4675–4684. ACL.
  37. Toxicity Detection: Does Context Really Matter?
  38. SemEval-2021 Task 5: Toxic Spans Detection. In Proceedings of the SemEval 2021, 59–69. ACL.
  39. Detecting and Monitoring Hate Speech in Twitter. Sensors, 19(21).
  40. Plutchik, R. 1980. A general psychoevolutionary theory of emotion. Theories of emotion, 1: 3–31.
  41. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation, 55(2): 477–523.
  42. A Benchmark Dataset for Learning to Intervene in Online Hate Speech. In Proceedings of the EMNLP-IJCNLP 2019, 4755–4764. ACL.
  43. HateCheck: Functional Tests for Hate Speech Detection Models. In Proceedings of the ACL-IJCNLP, 41–58. ACL.
  44. The Measuring Hate Speech Corpus: Leveraging Rasch Measurement Theory for Data Perspectivism. In Proceedings of the LREC 2022, 83–94. ELRA.
  45. Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for Identifying and Classifying Hate in Online News Media. Proceedings of the ICWSM 2018, 12(1).
  46. “Call me sexist, but…” : Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples. Proceedings of the ICWSM 2021, 15: 573–584.
  47. HaSpeeDe 2 @ EVALITA2020: Overview of the EVALITA 2020 Hate Speech Detection Task, 93–101. Accademia University Press.
  48. An Italian Twitter Corpus of Hate Speech against Immigrants. In Proceedings of the LREC 2018. ELRA.
  49. Analyzing the Targets of Hate in Online Social Media. Proceedings of the ICWSM 2021, 10(1): 687–690.
  50. Spertus, E. 1997. Smokey: Automatic Recognition of Hostile Messages. In AAAI/IAAI.
  51. Large-Scale Hate Speech Detection with Cross-Domain Transfer. In Proceedings of the LREC 2022, 2215–2225. ELRA.
  52. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9(86): 2579–2605.
  53. Directions in abusive language training data, a systematic review: Garbage in, garbage out. PLOS ONE, 15(12): e0243300.
  54. Introducing CAD: the Contextual Abuse Dataset. In Proceedings of the NAACL 2021, 2289–2303. ACL.
  55. Vogels, E. A. 2021. The state of online harassment.
  56. Waseem, Z. 2016. Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter. In Proceedings of the NLP + CSS 2016, 138–142. ACL.
  57. Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In Proceedings of the NAACL 2016. ACL.
  58. Ex Machina: Personal Attacks Seen at Scale.
  59. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of the NAACL 2019, 1415–1420. ACL.
Citations (2)
View on Semantic Scholar

Summary

  • The paper introduces MetaHate, a meta-dataset unifying 36 hate speech collections with over 1.2M human comments to enhance detection model benchmarking.
  • The methodology leverages lexical and psycholinguistic analyses, identifying approximately 20% hate speech and demonstrating BERT’s superior performance over SVM and CNN.
  • The work sets a foundation for advanced, multilingual hate speech detection while addressing ethical challenges in deploying such datasets.

Introduction to MetaHate

The paper under discussion presents an impressive contribution to the ongoing efforts in combating hate speech online, a problem recognized for causing harm on a global scale. The endeavor, titled "MetaHate," provides a meta-collection of databases that could significantly impact the detection and moderation of hate speech on social media platforms. The collective represents an aggregate of 36 datasets, meticulously curated to create a unified resource aimed at improving the accuracy and adaptability of detection models.

Challenges and Solutions

Although data scientists and researchers have already produced several databases for analyzing online hate speech, their efforts have been fragmented, making it difficult to develop models with broader, more generalizable applications. Furthermore, the absence of a unified definition of hate speech has further compounded the difficulty in creating consistent detection mechanisms. The paper delineates a clear definition that aligns with that of the United Nations, differentiating between hate speech and merely offensive content.

The central obstacle addressed by the paper is the lack of standardized datasets, models, and metrics which hampers the development of effective hate speech detection systems. In response, MetaHate compiles a significant volume of data—more than 1.2 million non-duplicated comments sourced from various social media platforms, standing as a substantial, human-authored dataset. A vital aspect of this collection is its exclusivity to human-generated text, negating synthetic data and alternative sources, thereby enhancing relevance and coherence.

Data Analysis Insights

A comprehensive analysis of the MetaHate collection reveals that around 20% of included posts are classified as hate speech. The paper describes both lexical and psycholinguistic analyses, uncovering commonly used words and terms, as well as emotional connotations present in hate speech versus non-hate speech posts. Additionally, the dataset underwent a series of modelling experiments to provide benchmarks for future research, utilizing standard methods such as Support Vector Machine (SVM), Convolutional Neural Networks (CNN), and BERT. BERT, in particular, demonstrated superior performance outdoing both SVM and CNN in accurately classifying hate speech content.

Future Directions and Ethical Considerations

The paper suggests that the MetaHate dataset enables the advancement of research in the field of hate speech detection by providing access to a more extensive and varied set of data points. It encourages the adoption of sophisticated approaches that can process emotional context and multilingual inputs, offering potential for a more inclusive and omni-faceted perspective. The paper concludes with an acknowledgment of the ethical dimensions associated with such a dataset, particularly noting the potential misuse of offensive language contained within.

The dataset and supporting code are made freely available for research purposes, emphasizing the objective to foster collaborative efforts toward a safer online environment. This positions MetaHate not only as a research tool but as a societal asset in the struggle to diminish the negative impact of hate speech faced by individuals and communities across digital platforms.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown