Performance Evaluation of Lightweight Open-source Large Language Models in Pediatric Consultations: A Comparative Analysis

Evaluation of Lightweight Open-source Language Models in Pediatric Consultations

The paper, titled "Performance Evaluation of Lightweight Open-source LLMs in Pediatric Consultations: A Comparative Analysis," offers a critical examination of lightweight LLMs in the domain of pediatric healthcare. The study highlights the performance capabilities and limitations of these models in addressing pediatric consultation queries, thus offering valuable insights for the integration of LLMs into healthcare settings.

Study Design and Methods

To assess the performance of lightweight LLMs in pediatric consultations, the study employed a cross-sectional design involving 250 consultation questions sourced from a public online medical forum, Haodf.com. These queries spanned 25 pediatric departments, capturing a broad spectrum of medical conditions. Four LLMs were selected for evaluation: ChatGLM3-6B, Vicuna-7B, Vicuna-13B, and the proprietary ChatGPT-3.5. Each model independently answered the questions in Chinese. Their performance was subsequently evaluated by three qualified pediatricians across five dimensions: accuracy, completeness, readability, empathy, and safety.

Findings

The study revealed several critical findings regarding the comparative performance of the LLMs:

• Accuracy: ChatGLM3-6B surpassed Vicuna-13B and Vicuna-7B (P < .001) but was outperformed by ChatGPT-3.5, which received the highest accuracy ratings at 65.2% “good” or “very good” evaluations.
• Completeness: ChatGPT-3.5 led with 78.4% of responses rated as “complete” or “very complete,” while ChatGLM3-6B also performed well at 76.0%. Vicuna-13B and Vicuna-7B lagged significantly behind.
• Readability: ChatGLM3-6B matched ChatGPT-3.5 in readability, outperforming the Vicuna models significantly.
• Empathy: ChatGPT-3.5 exhibited superior empathy (P < .001), indicating a higher capacity for humanistic care in its responses.
• Safety: All models demonstrated comparable safety, with over 98.4% of responses deemed safe.

These results were reproducible across repeated inquiries, affirming the robustness of the findings.

Implications

The paper underscores the potential of lightweight LLMs in pediatric healthcare environments, particularly when these models are tailored to specific linguistic contexts, as evidenced by ChatGLM3-6B's strong performance in Chinese-language medical consultations. Despite these promising results, the performance gap between lightweight models and the proprietary ChatGPT-3.5 suggests ongoing refinement is necessary. The authors advocate for further development to enhance the capabilities of lightweight LLMs, especially in terms of accuracy, completeness, and empathy.

Future Directions

The research indicates several avenues for future exploration and improvement:

1. Language and Cultural Contextualization: Tailoring LLMs to specific linguistic and cultural contexts can significantly enhance their performance, as seen with ChatGLM3-6B's success in the Chinese medical context.
2. Advanced Training Techniques: Employing techniques such as knowledge distillation, domain-specific pre-training, and continuous learning could improve model performance while maintaining computational efficiency.
3. Integration of Human Feedback: Continuous adaptation based on real-world interactions can refine the models' responses, ensuring greater relevance and accuracy in clinical settings.

Limitations

The study's limitations include the sample's potential lack of representativeness across the global pediatric landscape and the exclusive focus on single-round structured dialogues rather than multi-round conversations typical of real-world clinical interactions. Additionally, direct comparisons with human pediatricians' performance were not undertaken, which limits insights into LLM efficacy in practical healthcare scenarios.

Conclusion

This research contributes a substantial evaluation of lightweight LLMs in pediatric consultations, highlighting both their promise and the areas requiring development. The findings advocate for ongoing refinement and adaptation of these models, emphasizing the necessity of context-specific training and enhanced capabilities. As LLM technology continues to evolve, its integration into pediatric healthcare presents an opportunity to address critical shortfalls in medical consultation accessibility and efficiency, particularly in resource-limited settings.