SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis (2403.01976v5)
Abstract: Recent breakthroughs in LLMs have revolutionized scientific literature analysis. However, existing benchmarks fail to adequately evaluate the proficiency of LLMs in this domain, particularly in scenarios requiring higher-level abilities beyond mere memorization and the handling of multimodal data. In response to this gap, we introduce SciAssess, a benchmark specifically designed for the comprehensive evaluation of LLMs in scientific literature analysis. It aims to thoroughly assess the efficacy of LLMs by evaluating their capabilities in Memorization (L1), Comprehension (L2), and Analysis & Reasoning (L3). It encompasses a variety of tasks drawn from diverse scientific fields, including biology, chemistry, material, and medicine. To ensure the reliability of SciAssess, rigorous quality control measures have been implemented, ensuring accuracy, anonymization, and compliance with copyright standards. SciAssess evaluates 11 LLMs, highlighting their strengths and areas for improvement. We hope this evaluation supports the ongoing development of LLM applications in scientific literature analysis. SciAssess and its resources are available at \url{https://github.com/sci-assess/SciAssess}.
- Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.
- Gemini Team Google. Gemini: A family of highly capable multimodal models. CoRR, abs/2312.11805, 2023.
- OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
- Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303.12712, 2023.
- Measuring massive multitask language understanding. In ICLR. OpenReview.net, 2021.
- Agieval: A human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364, 2023.
- C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. CoRR, abs/2305.08322, 2023.
- Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206.04615, 2022.
- Challenging big-bench tasks and whether chain-of-thought can solve them. In ACL (Findings), pages 13003–13051. Association for Computational Linguistics, 2023.
- The impact of large language models on scientific discovery: a preliminary study using GPT-4. CoRR, abs/2311.07361, 2023.
- Large language models for scientific synthesis, inference and explanation. CoRR, abs/2310.07984, 2023.
- C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024.
- Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.
- Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
- Scieval: A multi-level large language model evaluation benchmark for scientific research. arXiv preprint arXiv:2308.13149, 2023.
- Krathwohl, d. r. (2002). a revision of bloom’s taxonomy: An overview. theory into practice, 41 (4), 212- 218. 2009.
- Pierre Caron and T Khan. Improvement of creep strength in a nickel-base single-crystal superalloy by heat treatment. Materials Science and Engineering, 61(2):173–184, 1983.
- Prediction of reversible α𝛼\alphaitalic_α/γ𝛾\gammaitalic_γ phase transformation in multi-pass weld of fe-cr-ni ternary alloy by phase-field method. Journal of Advanced Joining Processes, 4:100067, 2021.
- Physical characterization of sintered nimnga ferromagnetic shape memory alloy. Materials, 13(21):4806, 2020.
- Evaluation of hardening and softening behaviors in zn–21al–2cu alloy processed by equal channel angular pressing. Journal of Materials Research and Technology, 6(4):329–333, 2017.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.