Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Modeling Uncertainty in Personalized Emotion Prediction with Normalizing Flows (2312.06034v1)

Published 10 Dec 2023 in cs.AI

Abstract: Designing predictive models for subjective problems in NLP remains challenging. This is mainly due to its non-deterministic nature and different perceptions of the content by different humans. It may be solved by Personalized Natural Language Processing (PNLP), where the model exploits additional information about the reader to make more accurate predictions. However, current approaches require complete information about the recipients to be straight embedded. Besides, the recent methods focus on deterministic inference or simple frequency-based estimations of the probabilities. In this work, we overcome this limitation by proposing a novel approach to capture the uncertainty of the forecast using conditional Normalizing Flows. This allows us to model complex multimodal distributions and to compare various models using negative log-likelihood (NLL). In addition, the new solution allows for various interpretations of possible reader perception thanks to the available sampling function. We validated our method on three challenging, subjective NLP tasks, including emotion recognition and hate speech. The comparative analysis of generalized and personalized approaches revealed that our personalized solutions significantly outperform the baseline and provide more precise uncertainty estimates. The impact on the text interpretability and uncertainty studies are presented as well. The information brought by the developed methods makes it possible to build hybrid models whose effectiveness surpasses classic solutions. In addition, an analysis and visualization of the probabilities of the given decisions for texts with high entropy of annotations and annotators with mixed views were carried out.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. L. F. Barrett, How emotions are made: The secret life of the brain. Pan Macmillan, 2017.
  2. E. Pavlick and T. Kwiatkowski, “Inherent Disagreements in Human Textual Inferences,” Transactions of the Association for Computational Linguistics, vol. 7, pp. 677–694, 11 2019.
  3. C. Beck, H. Booth, M. El-Assady, and M. Butt, “Representation problems in linguistic annotations: Ambiguity, variation, uncertainty, error and bias,” in Proceedings of the 14th Linguistic Annotation Workshop, (Barcelona, Spain), pp. 60–73, Association for Computational Linguistics, Dec. 2020.
  4. A. M. Davani, M. Díaz, and V. Prabhakaran, “Dealing with disagreements: Looking beyond the majority vote in subjective annotations,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 92–110, 2022.
  5. E. Troiano, S. Padó, and R. Klinger, “Emotion ratings: How intensity, annotation confidence and agreements are entangled,” in Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, (Online), pp. 40–49, Association for Computational Linguistics, Apr. 2021.
  6. K. Krippendorff, “Computing krippendorff’s alpha-reliability,” Annenberg School for Communication Departmental Papers: Philadelphia, 2011.
  7. J. L. Fleiss, “Measuring nominal scale agreement among many raters.,” Psychological bulletin, vol. 76, no. 5, p. 378, 1971.
  8. P. Miłkowski, S. Saganowski, M. Gruza, P. Kazienko, M. Piasecki, and J. Kocoń, “Multitask personalized recognition of emotions evoked by textual content,” in EmotionAware 2022: Sixth International Workshop on Emotion Awareness for Pervasive Computing Beyond Traditional Approaches at PerCom 2022, (online), pp. 347–352, mar 2022.
  9. C. Strapparava and R. Mihalcea, “Learning to identify emotions in text,” in Proceedings of the 2008 ACM symposium on Applied computing, pp. 1556–1560, 2008.
  10. F. S. Tabak and V. Evrim, “Comparison of emotion lexicons,” in 2016 HONET-ICT, pp. 154–158, IEEE, 2016.
  11. R. Plutchik, “A general psychoevolutionary theory of emotion,” in Theories of emotion, pp. 3–33, Elsevier, 1980.
  12. P. Ekman, “An argument for basic emotions,” Cognition & emotion, vol. 6, no. 3-4, pp. 169–200, 1992.
  13. L. A. M. Oberländer and R. Klinger, “An analysis of annotated corpora for emotion classification in text,” in Proceedings of the 27th International Conference on Computational Linguistics, pp. 2104–2119, 2018.
  14. J. Kocoń, A. Janz, and M. Piasecki, “Context-sensitive sentiment propagation in wordnet,” in Proceedings of the 9th global wordnet conference, pp. 329–334, 2018.
  15. J. Kocoń and A. Janz, “Propagation of emotions, arousal and polarity in wordnet using heterogeneous structured synset embeddings,” in Proceedings of the 10th Global Wordnet Conference, pp. 336–341, 2019.
  16. J. Kocoń, A. Janz, P. Miłkowski, M. Riegel, M. Wierzba, A. Marchewka, A. Czoska, D. Grimling, B. Konat, K. Juszczyk, et al., “Recognition of emotions, valence and arousal in large-scale multi-domain text reviews,” in 9th Language & Technology Conference, 2019.
  17. J. Kocoń, P. Miłkowski, M. Wierzba, B. Konat, K. Klessa, A. Janz, M. Riegel, K. Juszczyk, D. Grimling, A. Marchewka, et al., “Multilingual and language-agnostic recognition of emotions, valence and arousal in large-scale multi-domain text reviews,” in Language and Technology Conference, pp. 214–231, Springer, 2019.
  18. D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “Goemotions: A dataset of fine-grained emotions,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4040–4054, 2020.
  19. J. Kocoń, J. Radom, E. Kaczmarz-Wawryk, K. Wabnic, A. Zajączkowska, and M. Zaśko-Zielińska, “Aspectemo: multi-domain corpus of consumer reviews for aspect-based sentiment analysis,” in 2021 International Conference on Data Mining Workshops (ICDMW), pp. 166–173, IEEE, 2021.
  20. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” Transactions on Machine Learning Research, 2023.
  21. S. Mohammad, “# emotional tweets,” in * SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 246–255, 2012.
  22. M. Abdul-Mageed and L. Ungar, “Emonet: Fine-grained emotion detection with gated recurrent neural networks,” in Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 718–728, 2017.
  23. C.-C. Hsu and L.-W. Ku, “Socialnlp 2018 emotionx challenge overview: Recognizing emotions in dialogues,” in Proceedings of the sixth international workshop on natural language processing for social media, pp. 27–31, 2018.
  24. E. Wulczyn, N. Thain, and L. Dixon, “Ex machina: Personal attacks seen at scale,” in Proceedings of the 26th international conference on world wide web, pp. 1391–1399, 2017.
  25. C. J. Kennedy, G. Bacon, A. Sahn, and C. von Vacano, “Constructing interval variables via faceted rasch measurement and multitask deep learning: a hate speech application,” arXiv preprint arXiv:2009.10277, 2020.
  26. S. Akhtar, V. Basile, and V. Patti, “Modeling annotator perspective and polarized opinions to improve hate speech detection,” in Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 8, pp. 151–154, 2020.
  27. J. Kocoń, A. Figas, M. Gruza, D. Puchalska, T. Kajdanowicz, and P. Kazienko, “Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approach,” Information Processing & Management, vol. 58, no. 5, p. 102643, 2021.
  28. F. Mireshghallah, V. Shrivastava, M. Shokouhi, T. Berg-Kirkpatrick, R. Sim, and D. Dimitriadis, “Useridentifier: implicit user representations for simple and effective personalized sentiment analysis,” arXiv preprint arXiv:2110.00135, 2021.
  29. K. Kanclerz, A. Figas, M. Gruza, T. Kajdanowicz, J. Kocoń, D. Puchalska, and P. Kazienko, “Controversy and conformity: from generalized to personalized aggressiveness detection,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5915–5926, 2021.
  30. P. Milkowski, M. Gruza, K. Kanclerz, P. Kazienko, D. Grimling, and J. Kocon, “Personal bias in prediction of emotions elicited by textual opinions,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021): Student Research Workshop, (Online), pp. 248–259, Association for Computational Linguistics, Aug. 2021.
  31. A. Ngo, A. Candri, T. Ferdinan, J. Kocoń, and W. Korczynski, “Studemo: A non-aggregated review dataset for personalized emotion recognition,” in Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022, pp. 46–55, 2022.
  32. K. Kanclerz, M. Gruza, K. Karanowski, J. Bielaniewicz, P. Miłkowski, J. Kocoń, and P. Kazienko, “What if ground truth is subjective? personalized deep neural hate speech detection,” in Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022, pp. 37–45, 2022.
  33. J. Bielaniewicz, K. Kanclerz, P. Miłkowski, M. Gruza, K. Karanowski, P. Kazienko, and J. Kocoń, “Deep-sheep: Sense of humor extraction from embeddings in the personalized context,” in 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 967–974, IEEE, 2022.
  34. T. Ferdinan and J. Kocoń, “Personalized models resistant to malicious attacks for human-centered trusted ai,” in The AAAI-23 Workshop on Artificial Intelligence Safety (SafeAI 2023), CEUR Workshop Proceedings, 2023.
  35. W. Mieleszczenko-Kowszewicz, K. Kanclerz, J. Bielaniewicz, M. Oleksy, M. Gruza, S. Woźniak, E. Dzięcioł, P. Kazienko, and J. Kocoń, “Capturing human perspectives in nlp: Questionnaires, annotations, and biases,” in The ECAI 2023 2nd Workshop on Perspectivist Approaches to NLP, CEUR Workshop Proceedings, 2023.
  36. J. Kocoń, J. Baran, K. Kanclerz, M. Kajstura, and P. Kazienko, “Differential dataset cartography: Explainable artificial intelligence in comparative personalized sentiment analysis,” in International Conference on Computational Science, pp. 148–162, Springer, 2023.
  37. E. Cambria, Q. Liu, S. Decherchi, F. Xing, and K. Kwok, “Senticnet 7: A commonsense-based neurosymbolic ai framework for explainable sentiment analysis,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 3829–3839, 2022.
  38. R. Mao, Q. Liu, K. He, W. Li, and E. Cambria, “The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,” IEEE Transactions on Affective Computing, 2022.
  39. M. M. Amin, R. Mao, E. Cambria, and B. W. Schuller, “A wide evaluation of chatgpt on affective computing tasks,” arXiv preprint arXiv:2308.13911, 2023.
  40. L. Zhu, W. Li, R. Mao, V. Pandelea, and E. Cambria, “Paed: zero-shot persona attribute extraction in dialogues,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9771–9787, 2023.
  41. L. Dinh, D. Krueger, and Y. Bengio, “Nice: Non-linear independent components estimation,” arXiv, 2014.
  42. L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using Real NVP,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net, 2017.
  43. G. Papamakarios, T. Pavlakou, and I. Murray, “Masked autoregressive flow for density estimation,” 2018.
  44. W. Grathwohl, R. T. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud, “Ffjord: Free-form continuous dynamics for scalable reversible generative models,” arXiv preprint arXiv:1810.01367, 2018.
  45. M. Zieba, M. Przewieźlikowski, M. Śmieja, J. Tabor, T. Trzcinski, and P. Spurek, “RegFlow: Probabilistic Flow-based Regression for Future Prediction,” CoRR, vol. abs/2011.14620, 2020.
  46. P. Wielopolski, M. Koperski, and M. Zieba, “Flow plugin network for conditional generation,” arXiv preprint arXiv:2110.04081, 2021.
  47. M. Wolczyk, M. Proszewska, L. Maziarka, M. Zieba, P. Wielopolski, R. Kurczab, and M. Smieja, “Plugen: Multi-label conditional generation from pre-trained models,” in Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pp. 8647–8656, AAAI Press, 2022.
  48. R. Abdal, P. Zhu, N. J. Mitra, and P. Wonka, “Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows,” ACM Trans. Graph., vol. 40, no. 3, pp. 21:1–21:21, 2021.
  49. P. Wielopolski and M. Zieba, “Treeflow: Going beyond tree-based gaussian probabilistic regression,” CoRR, vol. abs/2206.04140, 2022.
  50. D. Tran, K. Vafa, K. K. Agrawal, L. Dinh, and B. Poole, “Discrete flows: Invertible generative models of discrete data,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada (H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, eds.), pp. 14692–14701, 2019.
  51. J. Kocoń, M. Gruza, J. Bielaniewicz, D. Grimling, K. Kanclerz, P. Miłkowski, and P. Kazienko, “Learning personal human biases and representations for subjective tasks in natural language processing,” in 2021 IEEE International Conference on Data Mining (ICDM), pp. 1168–1173, IEEE, 2021.
  52. P. Kazienko, J. Bielaniewicz, M. Gruza, K. Kanclerz, K. Karanowski, P. Miłkowski, and J. Kocoń, “Human-centred neural reasoning for subjective content processing: Hate speech, emotions, and humor,” Information Fusion, 2023.
  53. D. J. Rezende and S. Mohamed, “Variational Inference with Normalizing Flows,” in Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, vol. 37 of JMLR Workshop and Conference Proceedings, pp. 1530–1538, JMLR.org, 2015.
  54. J. Vig and Y. Belinkov, “Analyzing the structure of attention in a transformer language model,” arXiv preprint arXiv:1906.04284, 2019.
  55. P. Miłkowski, M. Gruza, K. Kanclerz, P. Kazienko, D. Grimling, and J. Kocoń, “Personal bias in prediction of emotions elicited by textual opinions,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pp. 248–259, 2021.
  56. M. Wierzba, M. Riegel, J. Kocoń, P. Miłkowski, A. Janz, K. Klessa, K. Juszczyk, B. Konat, D. Grimling, M. Piasecki, et al., “Emotion norms for 6000 polish word meanings with a direct mapping to the polish wordnet,” Behavior Research Methods, pp. 1–16, 2021.
  57. R. Plutchik, The emotions. University Press of America, 1991.
  58. F. Feng, Y. Yang, D. Cer, N. Arivazhagan, and W. Wang, “Language-agnostic BERT sentence embedding,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (Dublin, Ireland), pp. 878–891, Association for Computational Linguistics, May 2022.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now