A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models (2202.08772v1)

Abstract: With the increasing of model capacity brought by pre-trained LLMs, there emerges boosting needs for more knowledgeable NLP models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge. The mere pre-trained LLMs, however, lack the capacity of handling such knowledge-intensive NLP tasks alone. To address this challenge, large numbers of pre-trained LLMs augmented with external knowledge sources are proposed and in rapid development. In this paper, we aim to summarize the current progress of pre-trained LLM-based knowledge-enhanced models (PLMKEs) by dissecting their three vital elements: knowledge sources, knowledge-intensive NLP tasks, and knowledge fusion methods. Finally, we present the challenges of PLMKEs based on the discussion regarding the three elements and attempt to provide NLP practitioners with potential directions for further research.