Delving into ChatGPT usage in academic writing through excess vocabulary
(2406.07016)Abstract
Recent LLMs can generate and revise text with human-level performance, and have been widely commercialized in systems like ChatGPT. These models come with clear limitations: they can produce inaccurate information, reinforce existing biases, and be easily misused. Yet, many scientists have been using them to assist their scholarly writing. How wide-spread is LLM usage in the academic literature currently? To answer this question, we use an unbiased, large-scale approach, free from any assumptions on academic LLM usage. We study vocabulary changes in 14 million PubMed abstracts from 2010-2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. Our analysis based on excess words usage suggests that at least 10% of 2024 abstracts were processed with LLMs. This lower bound differed across disciplines, countries, and journals, and was as high as 30% for some PubMed sub-corpora. We show that the appearance of LLM-based writing assistants has had an unprecedented impact in the scientific literature, surpassing the effect of major world events such as the Covid pandemic.
Overview
-
The study by Dmitry Kobak, Rita Gonzalez-Marquez, and Emőke-Agnes Horvát investigates the impact of LLMs, particularly ChatGPT, on scientific writing by analyzing shifts in vocabulary within PubMed abstracts from 2010 to 2024.
-
The researchers found an abrupt increase in certain style words in PubMed abstracts following the release of ChatGPT in late 2022, estimating that at least 10% of the 2024 abstracts were processed with LLMs. This percentage varied significantly across disciplines, countries, and journals.
-
The paper discusses both practical and theoretical implications of LLM usage in academic writing. Practically, it suggests LLMs can improve grammatical correctness but risk propagating biases. Theoretically, it highlights the importance of developing methods to detect and monitor AI-generated content in scholarly work.
Delving into ChatGPT Usage in Academic Writing Through Excess Vocabulary
The paper authored by Dmitry Kobak, Rita Gonzalez-Marquez, and Emőke-Agnes Horvát investigates the unprecedented impact of LLMs, specifically ChatGPT, on scientific writing by analyzing shifts in vocabulary within PubMed abstracts. This work employs a novel, data-driven approach, free of ground-truth assumptions, to uncover the extent of LLM usage.
Summary of Findings
Kobak et al. analyzed over 14 million PubMed abstracts from 2010 to 2024 to quantify changes in vocabulary and inferred the influence of LLMs. The analysis revealed an abrupt increase in certain style words following the release of ChatGPT in late 2022. Specifically, the 2024 corpus exhibited an unprecedented quantity of excess vocabulary, suggesting at least 10% of the abstracts were processed with LLMs, a conservative lower-bound estimate. This percentage varied significantly across disciplines, countries, and journals, reaching as high as 30% in some cases.
Methodology
The researchers utilized a large-scale approach based on tracking "excess words" - vocabulary showing significant increases in usage frequency post-LLM availability. This method is inspired by the concept of excess mortality used during the COVID-19 pandemic. By comparing observed 2024 word frequencies with counterfactual projections based on pre-LLM years (2021-2022), they identified words with large frequency gaps.
Key Results
Key findings of this study include:
- Excess Style Words: The analysis identified hundreds of style words whose frequency abruptly increased in 2024. These included verbs and adjectives such as "delves," "showcasing," "crucial," and "pivotal."
- Quantitative Impact: The overall lower bound for LLM-processed abstracts in 2024 was estimated at 10%, reaching up to 30% in computational fields and certain geographical regions.
- Heterogeneity: There were significant variations in LLM adoption rates among different countries, fields, and journals. For instance, journals with simplified review processes, like those by MDPI, showed much higher LLM usage.
- Comparative Analysis: The vocabulary shifts related to LLMs surpassed even those seen during the COVID-19 pandemic in terms of style rather than content-related words.
Implications
Practical Implications
The study underscores the substantial and growing influence of LLMs in scientific writing. These models, while improving grammatical correctness and readability, can propagate biases, fabricate information, and produce less diverse and innovative content. Moreover, the detection of such widespread LLM usage, even in high-prestige journals, calls into question the integrity of current publication practices.
Theoretical Implications
From a theoretical perspective, the unprecedented shifts in writing style imposed by LLMs highlight the emergent capabilities and integration of artificial intelligence in academic practices. This trend stresses the importance of developing robust methods to detect and monitor AI-generated content in scholarly work. It also suggests a potential future where AI not only assists but co-authors research, raising ethical and practical challenges regarding authorship and accountability.
Future Prospects
Given the trends observed, future research could delve into more sophisticated detection mechanisms for LLM usage, considering the rapid advancements in AI technologies. Additionally, there is a need for ongoing monitoring to reassess the extent of LLM impact as their adoption continues to grow. Policy changes advocating for transparency and responsible AI usage in academic contexts are essential to mitigate potential risks while harnessing the benefits of these powerful tools.
Conclusion
Kobak et al. have highlighted an important transition in academic writing precipitated by the adoption of LLMs. Their pioneering approach offers a robust framework for tracking and understanding this shift, providing critical insights for both the academic community and policymakers. As LLMs continue to evolve, their influence on the scientific literature will likely intensify, necessitating vigilant and adaptive strategies to balance innovation with integrity in scholarly communication.
This paper serves as a foundational analysis for understanding the nuanced impacts of LLMs on academic writing, setting the stage for future work to explore and address the challenges and opportunities posed by AI in academia.
Create an account to read this summary for free: