2000 character limit reached
ProbGate at EHRSQL 2024: Enhancing SQL Query Generation Accuracy through Probabilistic Threshold Filtering and Error Handling (2404.16659v1)
Published 25 Apr 2024 in cs.CL and cs.AI
Abstract: Recently, deep learning-based LLMs have significantly enhanced text-to-SQL tasks, with promising applications in retrieving patient records within the medical domain. One notable challenge in such applications is discerning unanswerable queries. Through fine-tuning model, we demonstrate the feasibility of converting medical record inquiries into SQL queries. Additionally, we introduce an entropy-based method to identify and filter out unanswerable results. We further enhance result quality by filtering low-confidence SQL through log probability-based distribution, while grammatical and schema errors are mitigated by executing queries on the actual database. We experimentally verified that our method can filter unanswerable questions, which can be widely utilized even when the parameters of the model are not accessible, and that it can be effectively utilized in practice.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- A heterogeneous graph to abstract syntax tree framework for text-to-sql. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):13796–13813.
- Recent advances in text-to-SQL: A survey of what we have and what we expect. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2166–2187, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association for Computational Linguistics.
- Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
- George Katsogiannis-Meimarakis and Georgia Koutrika. 2023. A survey on deep learning approaches for text-to-sql. The VLDB Journal, 32(4):905–936.
- Trustsql: A reliability benchmark for text-to-sql models with diverse unanswerable questions. arXiv preprint arXiv:2403.15879.
- Ehrsql: A practical text-to-sql benchmark for electronic health records. Advances in Neural Information Processing Systems, 35:15589–15601.
- Overview of the ehrsql 2024 shared task on reliable text-to-sql modeling on electronic health records. In Proceedings of the 6th Clinical Natural Language Processing Workshop, Mexico City, Mexico. Association for Computational Linguistics.
- Starcoder: may the source be with you! Transactions on Machine Learning Research. Reproducibility Certification.
- How pre-trained language models capture factual knowledge? a causal-inspired analysis. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1720–1732, Dublin, Ireland. Association for Computational Linguistics.
- Artificial neural networks for text-to-sql task: State of the art. In Advances in Smart Technologies Applications and Case Studies, pages 557–565, Cham. Springer International Publishing.
- Recent advances in natural language processing via large pre-trained language models: A survey. ACM Comput. Surv., 56(2).
- Knowledge graph-based question answering with electronic health records. In Machine Learning for Healthcare Conference, pages 36–53. PMLR.
- Mohammadreza Pourreza and Davood Rafiei. 2024. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in Neural Information Processing Systems, 36.
- Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
- Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
- Ehragent: Code empowers large language models for complex tabular reasoning on electronic health records. arXiv preprint arXiv:2401.07128.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578, Online. Association for Computational Linguistics.
- Know what I don’t know: Handling ambiguous and unknown questions for text-to-SQL. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5701–5714, Toronto, Canada. Association for Computational Linguistics.
- Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544–3552, Online. Association for Computational Linguistics.
- Yijun Xiao and William Yang Wang. 2021. On hallucination and predictive uncertainty in conditional language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2734–2744, Online. Association for Computational Linguistics.
- Sangryul Kim (8 papers)
- Donghee Han (3 papers)
- Sehyun Kim (2 papers)