Risk or Chance? Large Language Models and Reproducibility in HCI Research (2404.15782v3)
Abstract: Reproducibility is a major concern across scientific fields. Human-Computer Interaction (HCI), in particular, is subject to diverse reproducibility challenges due to the wide range of research methodologies employed. In this article, we explore how the increasing adoption of LLMs across all user experience (UX) design and research activities impacts reproducibility in HCI. In particular, we review upcoming reproducibility challenges through the lenses of analogies from past to future (mis)practices like p-hacking and prompt-hacking, general bias, support in data analysis, documentation and education requirements, and possible pressure on the community. We discuss the risks and chances for each of these lenses with the expectation that a more comprehensive discussion will help shape best practices and contribute to valid and reproducible practices around using LLMs in HCI research.
- The illusion of artificial inclusion. arXiv preprint arXiv:2401.08572 (2024). https://doi.org/10.48550/arXiv.2401.08572
- Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 337–371. https://proceedings.mlr.press/v202/aher23a.html
- Philip Ball. 2023. Is AI leading to a reproducibility crisis in science? Nature 624, 7990 (2023), 22–25.
- The Role of HCI in Reproducible Science: Understanding, Supporting and Motivating Core Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312905
- Elizabeth Gibney. 2022. Is AI fuelling a reproducibility crisis in science. Nature 608, 7922 (2022), 250–251.
- Simulating the Human in HCD with ChatGPT: Redesigning Interaction Design with AI. Interactions 31, 1 (jan 2024), 24–31. https://doi.org/10.1145/3637436
- AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation. arXiv:2402.14978 [cs.HC]
- Wilbert Tabone and Joost de Winter. 2023. Using ChatGPT for human–computer interaction research: a primer. Royal Society Open Science 10, 9 (2023), 231053. https://doi.org/10.1098/rsos.231053
- Transparency of CHI Research Artifacts: Results of a Self-Reported Survey. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376448
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.