Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model (2212.09146v3)

Abstract: Augmenting pretrained LLMs with retrievers has shown promise in effectively solving common NLP problems, such as language modeling and question answering. In this paper, we evaluate the strengths and weaknesses of popular retriever-augmented LLMs, namely kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriever + Flan-T5, in reasoning over retrieved statements across different tasks. Our findings indicate that the simple similarity metric employed by retrievers is insufficient for retrieving all the necessary statements for reasoning. Additionally, the LLMs do not exhibit strong reasoning even when provided with only the required statements. Furthermore, when combined with imperfect retrievers, the performance of the LLMs becomes even worse, e.g., Flan-T5's performance drops by 28.6% when retrieving 5 statements using Contriever. While larger LLMs improve performance, there is still a substantial room for enhancement. Our further analysis indicates that multihop retrieve-and-read is promising for LLMs like GPT-3.5, but does not generalize to other LLMs like Flan-T5-xxl.