How Does Generative Retrieval Scale to Millions of Passages?

Empirical Evaluation of Generative Retrieval Techniques at Scale

Introduction

In the ongoing evolution of information retrieval systems, generative retrieval models have emerged as a promising alternative to traditional dense retrievers. These models bypass conventional indexing by directly generating document identifiers (docids) for a given query. This study, conducted by researchers affiliated with Google Research and the University of Waterloo, represents the first systematic empirical evaluation of generative retrieval across various corpus scales, culminating in an assessment involving the entire MS MARCO passages corpus with model sizes up to 11 billion parameters.

Findings on Synthetic Queries and Model Scaling

The research underscores the pivotal role of synthetic queries as document representations, particularly as corpus size increases. Unlike other proposed architecture modifications, synthetic queries consistently enhance retrieval performance. Furthermore, this research highlights the limited benefits of escalating parameter counts beyond certain thresholds, challenging the notion that generative retrieval's effectiveness is intrinsically tied to model size.

Synthetic Queries as Central to Success

One of the study's critical revelations is the singular importance of synthetic queries in enhancing retrieval effectiveness, especially against the backdrop of growing corpus sizes. Amidst the array of strategies explored, only synthetic query generation -- used as a means of simulating document content for indexing -- remained effective and crucial for performance as the corpus expanded. Moreover, the study indicates that the enhanced performance afforded by synthetic queries surpasses that yielded by intricate model modifications or adjustments.

Compute Cost and the Naive Scaling Advantage

An intriguing outcome of this investigation is the efficiency of naively scaling model parameters, particularly when held against more sophisticated strategies like atomic identifiers or PAWA decoder enhancements. In scenarios where computational efficiency is paramount, naive parameter scaling emerges as superior in augmenting retrieval performance, provided the computational trade-offs are acceptable. This finding is particularly pronounced in experiments involving the full MS MARCO dataset, where a relatively straightforward approach of scaling model size to T5-XL dimensions and employing synthetic queries with Naive IDs outperformed more complex configurations.

Practical Implications and Future Research Directions

The insights gleaned from this investigation bear significant practical implications for the ongoing enhancement and application of generative retrieval models. Firstly, the critical role of synthetic queries in bolstering retrieval performance underscores the necessity of sophisticated query generation mechanisms, especially as the technology is applied to larger and more complex corpuses.

Secondly, the nuanced understanding of computational trade-offs in model scaling provides valuable guidance for future research endeavors. It suggests that while increasing model parameters can yield performance gains, there exists a point of diminishing returns. Consequently, future research might focus on optimizing parameter efficiency and exploring alternative scaling strategies that maximally leverage computational resources.

Concluding Remarks

This empirical study marks a sizable advancement in our comprehension of generative retrieval's dynamics across varying corpus scales. By methodically evaluating the impacts of synthetic queries and model scaling strategies, the research delineates a path forward for enhancing the effectiveness of generative retrieval systems. As the field continues to evolve, these findings are set to inform the development of more efficient, scalable, and accurate information retrieval systems.