ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up? (2311.16989v4)

Abstract: Upon its release in late 2022, ChatGPT has brought a seismic shift in the entire landscape of AI, both in research and commerce. Through instruction-tuning a LLM with supervised fine-tuning and reinforcement learning from human feedback, it showed that a model could answer human questions and follow instructions on a broad panel of tasks. Following this success, interests in LLMs have intensified, with new LLMs flourishing at frequent interval across academia and industry, including many start-ups focused on LLMs. While closed-source LLMs (e.g., OpenAI's GPT, Anthropic's Claude) generally outperform their open-source counterparts, the progress on the latter has been rapid with claims of achieving parity or even better on certain tasks. This has crucial implications not only on research but also on business. In this work, on the first anniversary of ChatGPT, we provide an exhaustive overview of this success, surveying all tasks where an open-source LLM has claimed to be on par or better than ChatGPT.

• The paper evaluates ChatGPT and open-source LLMs, demonstrating that open models are rapidly improving in conversation, reasoning, and specialized tasks.
• The study employs diverse evaluation methods—from human feedback to alternative sequence generation—to rigorously benchmark model performance.
• The findings highlight that open-source LLMs are closing the performance gap, offering enhanced transparency and reproducibility compared to closed-source alternatives.

Upon the one-year commemoration of ChatGPT's introduction, a comprehensive analysis was carried out to compare the effectiveness of open-source LLMs against the closed-source ChatGPT. The paper evaluates a range of tasks where open-source models have been claimed to perform on par with, or even exceed, the capabilities of ChatGPT, which, as a closed-source model, does not provide full access to its internal workings.

ChatGPT has had a substantial impact on both research and commercial AI applications, illustrated by its rapid user growth and substantial business investments. However, because of its non-public nature, there are limitations around understanding associated societal risks, difficulties ensuring reproducible research, and reliance on a single company's infrastructure and policies, which can lead to issues concerning access, data privacy, and costs.

The paper observes that although open-source models like Llama-2 and Falcon initially lagged behind their closed-source counterparts, they are rapidly closing the gap in performance across an array of tasks. Instances where the open-source LLMs excel include handling multi-turn conversations, agent capabilities, logical reasoning such as mathematics and coding, and domain-specific tasks like medical analysis.

The research consolidates various evaluation methods ranging from human feedback to alternative sequence generation, emphasizing the importance of data quality and training strategies in LLM development. The diversity of evaluations showcases the complexity of accurately assessing LLM capabilities and the challenges faced in establishing standardized benchmarks.

Ultimately, the paper aims to serve as a critical resource for both the research community and the business sector. It highlights the recent strides made by open-source LLMs, the evolving strategies for improving these models, and the potential issues encountered in open-source LLM development. This overview allows stakeholders to make informed decisions about the development and adoption of open-source LLMs.