Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
60 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring the Best Practices of Query Expansion with Large Language Models (2401.06311v3)

Published 12 Jan 2024 in cs.IR

Abstract: LLMs are foundational in language technologies, particularly in information retrieval (IR). Previous studies have utilized LLMs for query expansion, achieving notable improvements in IR. In this paper, we thoroughly explore the best practice of leveraging LLMs for query expansion. To this end, we introduce a training-free, straightforward yet effective framework called Multi-Text Generation Integration (\textsc{MuGI}). It leverages LLMs to generate multiple pseudo-references, integrating them with queries to enhance both sparse and dense retrievers. Our empirical findings reveal that: (1) Increasing the number of samples from LLMs benefits IR systems; (2) A balance between the query and pseudo-documents, and an effective integration strategy, is critical for high performance; (3) Contextual information from LLMs is essential, even boost a 23M model to outperform a 7B baseline model; (4) Pseudo relevance feedback can further calibrate queries for improved performance; and (5) Query expansion is widely applicable and versatile, consistently enhancing models ranging from 23M to 7B parameters. Our code and all generated references are made available at \url{https://github.com/lezhang7/Retrieval_MuGI}

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. Inpars: Data augmentation for information retrieval using large language models. arXiv preprint arXiv:2202.05144.
  2. Language models are few-shot learners. ArXiv, abs/2005.14165.
  3. Overview of the trec 2020 deep learning track.
  4. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820.
  5. Recommender systems in the era of large language models (llms). ArXiv, abs/2307.02046.
  6. Precise zero-shot dense retrieval without relevance labels.
  7. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.
  8. Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
  9. Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprint arXiv:2301.01820.
  10. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547.
  11. Dense passage retrieval for open-domain question answering.
  12. Retrieval-augmented generation for knowledge-intensive nlp tasks.
  13. Making large language models a better foundation for dense retrieval. arXiv preprint arXiv:2312.15503.
  14. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356–2362.
  15. Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156.
  16. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.
  17. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713.
  18. Learning implicit user profile for personalized retrieval-based chatbot. Proceedings of the 30th ACM International Conference on Information & Knowledge Management.
  19. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In North American Chapter of the Association for Computational Linguistics.
  20. Okapi at trec-3. In Text Retrieval Conference.
  21. Improving passage retrieval with zero-shot question generation. arXiv preprint arXiv:2204.07496.
  22. Large language models are strong zero-shot retriever. arXiv preprint arXiv:2304.14233.
  23. Is chatgpt good at search? investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542.
  24. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.
  25. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.
  26. Text embeddings by weakly-supervised contrastive pre-training.
  27. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.
  28. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678.
  29. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808.
  30. Multi-hop selector network for multi-turn response selection in retrieval-based chatbots. In Conference on Empirical Methods in Natural Language Processing.
  31. Recommendation as instruction following: A large language model empowered recommendation approach. ArXiv, abs/2305.07001.
  32. Retgen: A joint framework for retrieval and grounded text generation modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11739–11747.
  33. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107.

Summary

  • The paper introduces MuGI, a framework that integrates multiple LLM-generated pseudo references to augment query expansion for both sparse and dense information retrieval systems.
  • MuGI improves retrieval performance with significant gains, achieving up to a 19.8% boost on TREC benchmarks and superior dense re-ranking over standard methods.
  • MuGI operates without additional training, offering a scalable, efficient solution that bridges semantic gaps by enriching query contexts dynamically.

Analyzing MuGI: Multi-Text Generation Integration in Information Retrieval Systems

The paper "MuGI: Enhancing Information Retrieval through Multi-Text Generation Integration with LLMs" proposes a novel framework for augmenting the capabilities of Information Retrieval (IR) systems utilizing LLMs. The authors focus on overcoming the limitations of traditional IR methods by introducing a method termed Multi-Text Generation Integration (MuGI), which leverages LLMs for generating multiple pseudo references that enrich the original queries.

Core Methodology

MuGI is designed to improve both sparse and dense retrievers without any additional training requirements. The framework enhances IR by dynamically integrating multiple generative text samples with the original input query. This approach provides two primary functions: boosting the retrieval phase with an enriched query that carries more context and relevant keywords, and enabling a re-ranking phase that better captures document relevance.

  1. MuGI for Sparse Retrieval: The approach utilizes lexical-based methods like BM25, augmented with LLM-generated pseudo references. Instead of merely expanding the query with static terms, MuGI employs an adaptive query repetition strategy, dynamically balancing the weight of the original query against the generated content based on pseudo-reference length.
  2. MuGI for Dense Retrieval: For dense retrieval, MuGI augments query embeddings by concatenating multiple generated passages. This enhances the semantic richness of queries, thereby improving the alignment with relevant document embeddings in high-dimensional space.
  3. MuGI Pipeline: This comprehensive application combines sparse and dense retrieval enhancements. Initially, MuGI-influenced queries retrieve a broad set of potential matches, which are subsequently refined in a dense re-ranking phase.

Experimental Results

The authors conducted extensive evaluations on both in-domain and out-of-distribution datasets using benchmarks like TREC DL19/DL20 and BEIR. Here are some critical findings:

  • Improved Performance: MuGI dramatically enhances the performance of the BM25 model, achieving improvements of 19.8% on TREC DL19 and up to 7.6% on the BEIR benchmarks. This demonstrates the efficacy of MuGI in improving sparse retrieval methods, even outperforming advanced dense retrievers such as ANCE in certain contexts.
  • Robust Reranking: The integration of MuGI notably enhances the reranking capabilities of dense retrieval models. When compared to traditional re-ranking methods, MuGI significantly boosts retrieval effectiveness, showing superior results against baselines including MonoT5 and Cohere Re-rankv2, especially under in-domain conditions.

Theoretical and Practical Implications

The research underscores the potential of generative models in bridging semantic gaps in IR tasks. By integrating multiple pseudo references, MuGI supplies additional context and vocabulary, thus enhancing both lexical and semantic retrieval dimensions. The implication is a more refined and effective IR pipeline capable of handling diverse queries with greater accuracy.

The approach not only offers a straightforward method to augment existing IR systems without necessitating dataset-dependent retraining but also paves the way for more sophisticated retrieval systems that capitalize on the capabilities of LLMs to provide enriched query contexts.

Future Directions

The paper suggests multiple avenues for further research. One pertinent direction is investigating the scalability of MuGI across varying domains and datasets. Additionally, the adaptation of this framework to incorporate continuous advancements in LLM architectures could further amplify retrieval precision and scalability. Exploring the integration of MuGI with emerging IR paradigms, like Retrieval-Augmented Generation models, may present opportunities for more nuanced and comprehensive information discovery systems.

In conclusion, this paper presents a significant contribution to the field of Information Retrieval, offering new insights into the application of LLMs for query expansion and retrieval enhancement. It highlights the potential of MuGI as a highly adaptable and efficient enhancement for both sparse and dense retrieval frameworks.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube