Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

How Good is ChatGPT in Giving Advice on Your Visualization Design? (2310.09617v3)

Published 14 Oct 2023 in cs.HC

Abstract: Data visualization practitioners often lack formal training, resulting in a knowledge gap in visualization design best practices. Large-LLMs like ChatGPT, with their vast internet-scale training data, offer transformative potential in addressing this gap. To explore this potential, we adopted a mixed-method approach. Initially, we analyzed the VisGuide forum, a repository of data visualization questions, by comparing ChatGPT-generated responses to human replies. Subsequently, our user study delved into practitioners' reactions and attitudes toward ChatGPT as a visualization assistant. Participants, who brought their visualizations and questions, received feedback from both human experts and ChatGPT in a randomized order. They filled out experience surveys and shared deeper insights through post-interviews. The results highlight the unique advantages and disadvantages of ChatGPT, such as its ability to quickly provide a wide range of design options based on a broad knowledge base, while also revealing its limitations in terms of depth and critical thinking capabilities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. S. Esteves and M. Neves, ““i learned it on the job” becoming a data visualization professional in news media,” Information Design Journal, vol. 27, no. 3, pp. 309–319, 2022. [Online]. Available: https://www.jbe-platform.com/content/journals/10.1075/idj.22004.est
  2. Data visualization society surveys. Accessed on Sep 9, 2023. [Online]. Available: https://www.datavisualizationsociety.org/survey-history
  3. J. Choi, C. Oh, Y.-S. Kim, and N. W. Kim, “Vislab: Enabling visualization designers to gather empirically informed design feedback,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI ’23.   New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3544548.3581132
  4. H. K. Bako, X. Liu, L. Battle, and Z. Liu, “Understanding how designers find and use data visualization examples,” IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 1, pp. 1048–1058, 2023.
  5. K. Luther, J.-L. Tolentino, W. Wu, A. Pavel, B. P. Bailey, M. Agrawala, B. Hartmann, and S. P. Dow, “Structuring, aggregating, and evaluating crowdsourced design critique,” in Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, ser. CSCW ’15.   New York, NY, USA: Association for Computing Machinery, 2015, p. 473–485. [Online]. Available: https://doi.org/10.1145/2675133.2675283
  6. J. Okerlund, E. Klasky, A. Middha, S. Kim, H. Rosenfeld, M. Kleinman, and S. Parthasarathy, “What’s in the chatterbox? large language models, why they matter, and what we should do about them,” 2022.
  7. OpenAI, “ChatGPT,” https://openai.com/chatgpt, accessed on Sep 7, 2023.
  8. A. Diehl, A. Abdul-Rahman, M. El-Assady, B. Bach, D. Keim, and M. Chen, “VisGuides: A Forum for Discussing Visualization Guidelines,” in EuroVis 2018 - Short Papers, J. Johansson, F. Sadlo, and T. Schreck, Eds.   The Eurographics Association, 2018.
  9. J. Choi, C. Oh, B. Suh, and N. W. Kim, “Toward a unified framework for visualization design guidelines,” in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’21.   New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3411763.3451702
  10. P. Parsons, “Understanding data visualization design practice,” IEEE Transactions on Visualization; Computer Graphics, vol. 28, no. 01, pp. 665–675, jan 2022.
  11. P. Parsons, C. M. Gray, A. Baigelenov, and I. Carr, “Design judgment in data visualization practice,” in 2020 IEEE Visualization Conference (VIS), Oct 2020, pp. 176–180.
  12. P. Parsons and P. Shukla, “Data visualization practitioners’ perspectives on chartjunk,” in 2020 IEEE Visualization Conference (VIS), Oct 2020, pp. 211–215.
  13. I. Chalkidis, I. Androutsopoulos, and N. Aletras, “Neural legal judgment prediction in English,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.   Florence, Italy: Association for Computational Linguistics, 2019, pp. 4317–4323. [Online]. Available: https://aclanthology.org/P19-1424
  14. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  15. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33.   Curran Associates, Inc., 2020, pp. 1877–1901. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  16. R. OpenAI, “Gpt-4 technical report,” arXiv, pp. 2303–08 774, 2023.
  17. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023.
  18. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. S. Chatterji, A. S. Chen, K. A. Creel, J. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. E. Gillespie, K. Goel, N. D. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. F. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. S. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. P. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. F. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. H. Roohani, C. Ruiz, J. Ryan, C. R’e, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. P. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. A. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang, “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021. [Online]. Available: https://crfm.stanford.edu/assets/report.pdf
  19. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, and J.-R. Wen, “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023.
  20. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  21. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018.
  22. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent abilities of large language models,” Transactions on Machine Learning Research, 2022.
  23. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, ser. FAccT ’21.   New York, NY, USA: Association for Computing Machinery, 2021, p. 610–623. [Online]. Available: https://doi.org/10.1145/3442188.3445922
  24. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar et al., “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110, 2022.
  25. Y. Chang, X. Wang, J. Wang, Y. Wu, K. Zhu, H. Chen, L. Yang, X. Yi, C. Wang, Y. Wang et al., “A survey on evaluation of large language models,” arXiv preprint arXiv:2307.03109, 2023.
  26. C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, “Is chatgpt a general-purpose natural language processing task solver?” arXiv preprint arXiv:2302.06476, vol. abs/2302.06476, 2023.
  27. Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola, “Multimodal chain-of-thought reasoning in language models,” arXiv preprint arXiv:2302.00923, 2023.
  28. M. Nadeem, A. Bethke, and S. Reddy, “Stereoset: Measuring stereotypical bias in pretrained language models,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).   Association for Computational Linguistics, 2021, pp. 5356–5371.
  29. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “RealToxicityPrompts: Evaluating neural toxic degeneration in language models,” in Findings of the Association for Computational Linguistics: EMNLP 2020.   Association for Computational Linguistics, 2020, pp. 3356–3369.
  30. E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and G. Kasneci, “Chatgpt for good? on opportunities and challenges of large language models for education,” Learning and Individual Differences, vol. 103, p. 102274, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1041608023000195
  31. D. Baidoo-Anu and L. Owusu Ansah, “Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning,” Available at SSRN 4337484, 2023.
  32. A. Tlili, B. Shehata, M. A. Adarkwah, A. Bozkurt, D. T. Hickey, R. Huang, and B. Agyemang, “What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education,” Smart Learning Environments, vol. 10, no. 1, p. 15, 2023. [Online]. Available: https://doi.org/10.1186/s40561-023-00237-x
  33. A. Gilson, C. W. Safranek, T. Huang, V. Socrates, L. Chi, R. A. Taylor, D. Chartash et al., “How does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment,” JMIR Medical Education, vol. 9, no. 1, p. e45312, 2023.
  34. Z. Fei, X. Shen, D. Zhu, F. Zhou, Z. Han, S. Zhang, K. Chen, Z. Shen, and J. Ge, “Lawbench: Benchmarking legal knowledge of large language models,” arXiv preprint arXiv:2309.16289, 2023.
  35. J. H. Choi, K. E. Hickman, A. Monahan, and D. Schwarcz, “Chatgpt goes to law school,” Available at SSRN, 2023.
  36. S. Mollman, “Chatgpt passed a wharton mba exam and it’s still in its infancy. one professor is sounding the alarm,” https://fortune.com/2023/01/21/chatgpt-passed-wharton-mba-exam-one-professor-is-sounding-alarm-artificial-intelligence/, 2023, accessed on Sep 7, 2023.
  37. P. Lee, S. Bubeck, and J. Petro, “Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine,” New England Journal of Medicine, vol. 388, no. 13, pp. 1233–1239, 2023.
  38. A. Lahat, E. Shachar, B. Avidan, Z. Shatz, B. S. Glicksberg, and E. Klang, “Evaluating the use of large language model in identifying top research questions in gastroenterology,” Scientific Reports, vol. 13, no. 1, p. 4164, Mar. 2023. [Online]. Available: https://doi.org/10.1038/s41598-023-31412-2
  39. J. Pan, “Large language model for molecular chemistry,” Nature Computational Science, vol. 3, no. 1, pp. 5–5, 2023. [Online]. Available: https://doi.org/10.1038/s43588-023-00399-1
  40. S. Paik, “How i created a data visualization with zero coding skills, thanks to chatgpt,” Nightingale magazine by the Data Visualization Society, April 2023. [Online]. Available: https://nightingaledvs.com/data-visualization-using-chatgpt-to-code/
  41. M. Hutchinson, A. Slingsby, R. Jianu, and P. Madhyastha, “Towards Visualisation Specifications from Multilingual Natural Language Queries using Large Language Models,” 2023.
  42. V. Dibia, “Lida: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).   Toronto, Canada: Association for Computational Linguistics, 2023, pp. 113–126. [Online]. Available: https://aclanthology.org/2023.acl-demo.11
  43. S. D. Bartolomeo, G. Severi, V. Schetinger, and C. Dunne, “Ask and You Shall Receive (a Graph Drawing): Testing ChatGPT’s Potential to Apply Graph Layout Algorithms,” in EuroVis 2023 - Short Papers, T. Hoellt, W. Aigner, and B. Wang, Eds.   The Eurographics Association, 2023.
  44. P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’22.   New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491101.3519665
  45. V. Schetinger, S. Di Bartolomeo, M. El-Assady, A. McNutt, M. Miller, J. P. A. Passos, and J. L. Adams, “Doom or deliciousness: Challenges and opportunities for visualization in the age of generative models,” Computer Graphics Forum, vol. 42, no. 3, pp. 423–435, 2023. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14841
  46. A. Diehl, E. E. Firat, T. Torsney-Weir, A. Abdul-Rahman, B. Bach, R. Laramee, R. Pajarola, and M. Chen, “VisGuided: A Community-driven Approach for Education in Visualization,” in Eurographics 2021 - Education Papers, B. Sousa Santos and G. Domik, Eds.   The Eurographics Association, 2021.
  47. Data visualization society. https://www.datavisualizationsociety.com. Accessed: March 28, 2023.
  48. A. Ihwan, “Role prompting,” https://www.linkedin.com/pulse/role-prompting-aris-ihwan/, accessed on Sep 8, 2023.
  49. M. Yin, J. Wortman Vaughan, and H. Wallach, “Understanding the effect of accuracy on trust in machine learning models,” in Proceedings of the 2019 chi conference on human factors in computing systems, 2019, pp. 1–12.
  50. E. Toreini, M. Aitken, K. Coopamootoo, K. Elliott, C. G. Zelaya, and A. Van Moorsel, “The relationship between trust in ai and trustworthy machine learning technologies,” in Proceedings of the 2020 conference on fairness, accountability, and transparency, 2020, pp. 272–283.
  51. J. Drozdal, J. Weisz, D. Wang, G. Dass, B. Yao, C. Zhao, M. Muller, L. Ju, and H. Su, “Trust in automl: exploring information needs for establishing trust in automated machine learning systems,” in Proceedings of the 25th international conference on intelligent user interfaces, 2020, pp. 297–307.
  52. F. Yang, Z. Huang, J. Scholtz, and D. L. Arendt, “How do visual explanations foster end users’ appropriate trust in machine learning?” in Proceedings of the 25th international conference on intelligent user interfaces, 2020, pp. 189–201.
  53. A. Chatzimparmpas, R. M. Martins, I. Jusufi, K. Kucher, F. Rossi, and A. Kerren, “The state of the art in enhancing trust in machine learning models with the use of visualizations,” in Computer Graphics Forum, vol. 39, no. 3.   Wiley Online Library, 2020, pp. 713–756.
  54. B. Liu, M. Ding, S. Shaham, W. Rahayu, F. Farokhi, and Z. Lin, “When machine learning meets privacy: A survey and outlook,” ACM Computing Surveys (CSUR), vol. 54, no. 2, pp. 1–36, 2021.
  55. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” arXiv preprint arXiv:2306.05685, 2023.
  56. Z. Bylinskii, N. W. Kim, P. O’Donovan, S. Alsheikh, S. Madan, H. Pfister, F. Durand, B. Russell, and A. Hertzmann, “Learning visual importance for graphic designs and data visualizations,” in Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’17.   New York, NY, USA: Association for Computing Machinery, 2017, p. 57–69. [Online]. Available: https://doi.org/10.1145/3126594.3126653
  57. E. Hoque, P. Kavehzadeh, and A. Masry, “Chart question answering: State of the art and future directions,” Computer Graphics Forum, vol. 41, no. 3, pp. 555–572, 2022. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14573
  58. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022.
  59. “Github copilot,” https://github.com/features/copilot, accessed on Sep 9, 2023.
  60. B. Bach, M. Keck, F. Rajabiyazdi, T. Losev, I. Meirelles, J. Dykes, R. S. Laramee, M. AlKadi, C. Stoiber, S. Huron et al., “Challenges and opportunities in data visualization education: A call to action,” arXiv preprint arXiv:2308.07703, 2023.
  61. D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” Applied Sciences, vol. 11, no. 14, p. 6421, 2021.
  62. T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. McKeown, and T. B. Hashimoto, “Benchmarking large language models for news summarization,” arXiv preprint arXiv:2301.13848, 2023.
  63. Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “Gpteval: Nlg evaluation using gpt-4 with better human alignment,” arXiv preprint arXiv:2303.16634, 2023.
  64. J. Wang, Y. Liang, F. Meng, H. Shi, Z. Li, J. Xu, J. Qu, and J. Zhou, “Is chatgpt a good nlg evaluator? a preliminary study,” arXiv preprint arXiv:2303.04048, 2023.
  65. J. Fu, S.-K. Ng, Z. Jiang, and P. Liu, “Gptscore: Evaluate as you desire,” arXiv preprint arXiv:2302.04166, 2023.
  66. S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne et al., “Prometheus: Inducing fine-grained evaluation capability in language models,” arXiv preprint arXiv:2310.08491, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Nam Wook Kim (14 papers)
  2. Grace Myers (3 papers)
  3. Benjamin Bach (31 papers)
Citations (16)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets