2000 character limit reached
LG AI Research & KAIST at EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs (2405.11162v1)
Published 18 May 2024 in cs.CL
Abstract: Text-to-SQL models are pivotal for making Electronic Health Records (EHRs) accessible to healthcare professionals without SQL knowledge. With the advancements in LLMs, these systems have become more adept at translating complex questions into SQL queries. Nonetheless, the critical need for reliability in healthcare necessitates these models to accurately identify unanswerable questions or uncertain predictions, preventing misinformation. To address this problem, we present a self-training strategy using pseudo-labeled unanswerable questions to enhance the reliability of text-to-SQL models for EHRs. This approach includes a two-stage training process followed by a filtering method based on the token entropy and query execution. Our methodology's effectiveness is validated by our top performance in the EHRSQL 2024 shared task, showcasing the potential to improve healthcare decision-making through more reliable text-to-SQL systems.
- Self-training: A survey. CoRR, abs/2202.12040.
- Electronic health record challenges, workarounds, and solutions observed in practices integrating behavioral health and primary care. The Journal of the American Board of Family Medicine, 28(Supplement 1):S63–S72.
- C3: zero-shot text-to-sql with chatgpt. CoRR, abs/2307.07306.
- Text-to-sql empowered by large language models: A benchmark evaluation. Proc. VLDB Endow., 17(5):1132–1145.
- Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196.
- A comprehensive exploration on wikisql with table-aware word contextualization. arXiv preprint arXiv:1902.01069.
- A comprehensive exploration on wikisql with table-aware word contextualization. CoRR, abs/1902.01069.
- Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1–9.
- EHRSQL: A practical text-to-sql benchmark for electronic health records. CoRR, abs/2301.07695.
- Overview of the ehrsql 2024 shared task on reliable text-to-sql modeling on electronic health records. In Proceedings of the 6th Clinical Natural Language Processing Workshop, Mexico City, Mexico. Association for Computational Linguistics.
- Learning to reduce: Optimal representations of structured data in prompting large language models. arXiv preprint arXiv:2402.14195.
- Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.
- Bridging textual and tabular data for cross-domain text-to-sql semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 4870–4888. Association for Computational Linguistics.
- Hybrid ranking network for text-to-sql. CoRR, abs/2008.04759.
- Enhancing few-shot text-to-sql capabilities of large language models: A study on prompt design strategies. CoRR, abs/2305.12586.
- emrqa: A large corpus for question answering on electronic medical records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2357–2368. Association for Computational Linguistics.
- Knowledge graph-based question answering with electronic health records. In Proceedings of the Machine Learning for Healthcare Conference, MLHC 2021, 6-7 August 2021, Virtual Event, volume 149 of Proceedings of Machine Learning Research, pages 36–53. PMLR.
- Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: decomposed in-context learning of text-to-sql with self-correction. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.
- Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
- emrkbqa: A clinical knowledge-base question answering dataset. In Proceedings of the 20th Workshop on Biomedical Language Processing, BioNLP@NAACL-HLT 2021, Online, June 11, 2021, pages 64–73. Association for Computational Linguistics.
- Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
- Retrieval-augmented data augmentation for low-resource domain tasks. CoRR, abs/2402.13482.
- Ehragent: Code empowers large language models for complex tabular reasoning on electronic health records. CoRR, abs/2401.07128.
- Exploring chain of thought style prompting for text-to-sql. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 5376–5393. Association for Computational Linguistics.
- RAT-SQL: relation-aware schema encoding and linking for text-to-sql parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7567–7578. Association for Computational Linguistics.
- Text-to-sql generation for question answering on electronic medical records. In WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 350–361. ACM / IW3C2.
- Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13484–13508. Association for Computational Linguistics.
- Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
- Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3911–3921. Association for Computational Linguistics.
- Self-rewarding language models. CoRR, abs/2401.10020.
- Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
- Yongrae Jo (9 papers)
- Seongyun Lee (13 papers)
- Minju Seo (7 papers)
- Sung Ju Hwang (178 papers)
- Moontae Lee (54 papers)