2000 character limit reached
Large Language Model Augmented Exercise Retrieval for Personalized Language Learning (2402.16877v1)
Published 8 Feb 2024 in cs.IR, cs.AI, cs.CL, and cs.LG
Abstract: We study the problem of zero-shot exercise retrieval in the context of online language learning, to give learners the ability to explicitly request personalized exercises via natural language. Using real-world data collected from language learners, we observe that vector similarity approaches poorly capture the relationship between exercise content and the language that learners use to express what they want to learn. This semantic gap between queries and content dramatically reduces the effectiveness of general-purpose retrieval models pretrained on large scale information retrieval datasets like MS MARCO. We leverage the generative capabilities of LLMs to bridge the gap by synthesizing hypothetical exercises based on the learner's input, which are then used to search for relevant exercises. Our approach, which we call mHyER, overcomes three challenges: (1) lack of relevance labels for training, (2) unrestricted learner input content, and (3) low semantic similarity between input and retrieval candidates. mHyER outperforms several strong baselines on two novel benchmarks created from crowdsourced data and publicly available data.
- Ghodai Abdelrahman and Qing Wang. 2019. Knowledge Tracing with Sequential Key-Value Memory Networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM.
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv preprint arXiv:1611.09268 (2016).
- Inpars: Unsupervised dataset generation for information retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2387–2392.
- A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML). PMLR, 1597–1607.
- Improving Contrastive Learning of Sentence Embeddings from AI Feedback. arXiv preprint arXiv:2305.01918 (2023).
- DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4207–4218.
- Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4 (1994), 253–278.
- Peng Cui and Mrinmaya Sachan. 2023. Adaptive and Personalized Exercise Generation for Online Language Learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 10184–10198.
- Promptagator: Few-shot Dense Retrieval From 8 Examples. In The Eleventh International Conference on Learning Representations.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT. 4171–4186.
- Precise Zero-Shot Dense Retrieval without Relevance Labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 1762–1777.
- SimCSE: Simple Contrastive Learning of Sentence Embeddings. In 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021. 6894–6910.
- Investigating self-directed learning and technology readiness in blending learning environment. International Journal of Educational Technology in Higher Education 16, 1 (2019), 1–22.
- Dimensionality reduction by learning an invariant mapping. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2. IEEE, 1735–1742.
- A Design of a Simple Yet Effective Exercise Recommendation System in K-12 Online Learning. In International Conference on Artificial Intelligence in Education. Springer, 208–212.
- Unsupervised Dense Information Retrieval with Contrastive Learning. Transactions on Machine Learning Research (2022).
- Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 6769–6781.
- Kyong-Jee Kim and Hye W Jang. 2015. Changes in medical students’ motivation and self-regulated learning: a preliminary study. International journal of medical education 6 (2015), 213.
- Stephen Krashen. 2005. Free voluntary reading: New research, applications, and controversies. Anthology series-Seameo regional language centre 46, 1 (2005).
- Latent Retrieval for Weakly Supervised Open Domain Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 6086–6096.
- RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 (2019).
- Self-regulated learning and academic performance in medical education. Medical teacher 38, 6 (2016), 585–593.
- Berlinda Mandasari and Achmad Yudi Wahyudin. 2021. Flipped classroom learning model: implementation and its impact on EFL learners’ satisfaction on grammar class. Ethical Lingua: Journal of Language Teaching and Literature 8, 1 (2021), 150–158.
- OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
- Shalini Pandey and George Karypis. 2019. A Self-Attentive Model for Knowledge Tracing. International Educational Data Mining Society (2019).
- The effects of choice on intrinsic motivation and related outcomes: a meta-analysis of research findings. Psychological bulletin 134, 2 (2008), 270.
- Deep knowledge tracing. Advances in Neural Information Processing Systems 28 (2015).
- The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389.
- Improving Passage Retrieval with Zero-Shot Question Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 3781–3797.
- Saint+: Integrating temporal features for ednet correctness prediction. In LAK21: 11th International Learning Analytics and Knowledge Conference. 490–496.
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Structure-based knowledge tracing: An influence propagation view. In 2020 IEEE International Conference on Data Mining (ICDM). IEEE, 541–550.
- GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2345–2360.
- English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 9122–9133.
- CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In Proceedings of the Twelfth Language Resources and Evaluation Conference. 4003–4012.
- ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding. In Proceedings of the 29th International Conference on Computational Linguistics. 3898–3907.
- Exercise recommendation based on knowledge concept prediction. Knowledge-Based Systems 210 (2020), 106481.
- Liangbei Xu and Mark A Davenport. 2020. Dynamic Knowledge Embedding and Tracing. International Educational Data Mining Society (2020).
- Generate rather than Retrieve: Large Language Models are Strong Context Generators. In The Eleventh International Conference on Learning Representations.
- COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 1462–1479.