Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answering (2309.06358v2)
Abstract: Robustness in Natural Language Processing continues to be a pertinent issue, where state of the art models under-perform under naturally shifted distributions. In the context of Question Answering, work on domain adaptation methods continues to be a growing body of research. However, very little attention has been given to the notion of domain generalization under natural distribution shifts, where the target domain is unknown. With drastic improvements in the quality and access to generative models, we answer the question: How do generated datasets influence the performance of QA models under natural distribution shifts? We perform experiments on 4 different datasets under varying amounts of distribution shift, and analyze how "in-the-wild" generation can help achieve domain generalization. We take a two-step generation approach, generating both contexts and QA pairs to augment existing datasets. Through our experiments, we demonstrate how augmenting reading comprehension datasets with generated data leads to better robustness towards natural distribution shifts.
- Synthetic qa corpora generation with roundtrip consistency.
- Types of out-of-distribution texts and how to detect them. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10687–10701, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Hritik Bansal and Aditya Grover. 2023. Leaving reality to imagination: Robust classification via generated datasets.
- Improving question answering model robustness with synthetic adversarial data generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8830–8848, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Language models are few-shot learners.
- Sparks of artificial general intelligence: Early experiments with gpt-4.
- Improving robustness using generated data.
- An exploration of data augmentation and sampling techniques for domain-agnostic question answering. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 220–227, Hong Kong, China. Association for Computational Linguistics.
- Transformer-based end-to-end question generation. ArXiv, abs/2005.01107.
- Leveraging QA datasets to improve generative data augmentation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9737–9750, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- The effect of natural distribution shift on question answering models.
- SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
- Alan Ramponi and Barbara Plank. 2020. Neural unsupervised domain adaptation in NLP—A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6838–6855, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- End-to-end synthetic data generation for domain adaptation of question answering systems. arXiv preprint arXiv:2010.06028.
- Llama: Open and efficient foundation language models.
- Measure and improve robustness in NLP models: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4569–4586, Seattle, United States. Association for Computational Linguistics.
- Finetuned language models are zero-shot learners.
- Contrastive domain adaptation for question answering using limited text corpora. arXiv preprint arXiv:2108.13854.
- Domain adaptation for question answering via question classification. arXiv preprint arXiv:2209.04998.
- Arijit Ghosh Chowdhury (6 papers)
- Aman Chadha (110 papers)