Improving Retrieval in Sponsored Search by Leveraging Query Context Signals
(2407.14346)Abstract
Accurately retrieving relevant bid keywords for user queries is critical in Sponsored Search but remains challenging, particularly for short, ambiguous queries. Existing dense and generative retrieval models often fail to capture nuanced user intent in these cases. To address this, we propose an approach to enhance query understanding by augmenting queries with rich contextual signals derived from web search results and LLMs, stored in an online cache. Specifically, we use web search titles and snippets to ground queries in real-world information and utilize GPT-4 to generate query rewrites and explanations that clarify user intent. These signals are efficiently integrated through a Fusion-in-Decoder based Unity architecture, enabling both dense and generative retrieval with serving costs on par with traditional context-free models. To address scenarios where context is unavailable in the cache, we introduce context glancing, a curriculum learning strategy that improves model robustness and performance even without contextual signals during inference. Extensive offline experiments demonstrate that our context-aware approach substantially outperforms context-free models. Furthermore, online A/B testing on a prominent search engine across 160+ countries shows significant improvements in user engagement and revenue.
Overview
-
The paper presents a context-aware retrieval model, called Augmented Unity, improving keyword matching in sponsored search by leveraging query context signals such as web search titles, snippets, and LLM-generated query profiles.
-
The methodology incorporates Fusion-in-Decoder (FiD) architecture for efficient integration of diverse contexts and employs a curriculum learning strategy termed 'Context Glancing' to ensure model robustness.
-
Experimental results demonstrate significant improvements in retrieval performance, engagement metrics, and ad revenue, showcasing the potential for practical deployment and future research extensions.
Improving Retrieval in Sponsored Search by Leveraging Query Context Signals
The paper, Improving Retrieval in Sponsored Search by Leveraging Query Context Signals, addresses the critical need for improved keyword retrieval in sponsored search, particularly focusing on contextually enhancing query understanding to increase the relevance of retrieved keywords. This research incorporates web search results and LLMs to provide richer contextual signals for queries, thereby significantly improving the accuracy of keyword retrieval.
Summary of Contributions
Key Problem Addressed: Sponsored search, a primary revenue model for many search engines, relies heavily on accurately matching user queries to advertiser bid keywords. The challenge intensifies with short or ambiguous queries where existing retrieval models often fail to capture nuanced user intent. The proposed work seeks to bridge this gap by leveraging advanced query context signals.
Innovative Approach: The paper introduces Augmented Unity, which enhances context-aware retrieval by integrating:
- Web Search Titles and Snippets: These provide real-world, contextually relevant information for grounding queries.
- LLM-generated Query Profiles: Utilizing GPT-4, the system generates query rewrites and explanations that clarify user intent.
- Fusion-in-Decoder (FiD) Architecture: This architecture allows efficient integration of diverse contexts, facilitating both dense and generative retrieval approaches.
- Context Glancing: A curriculum learning strategy used to train the model, ensuring robustness even when contextual signals are absent during inference.
Methodology
Query Context Signals: The research leverages two primary sources:
- Web Search Results: Titles and snippets from top-ranking web documents provide valuable information.
- LLM-generated Query Profiles: GPT-4 outputs multiple rewrites and a description of potential user intent, thus enhancing disambiguation.
Model Architecture and Training: The model employs a shared encoder-decoder framework. Queries and contexts are first encoded independently and then processed using the Fusion-in-Decoder approach. This setup enables both dense vector retrieval and non-autoregressive keyword generation. Training utilizes a combination of negative log-likelihood loss for NLG and contrastive loss for DR.
Context Glancing: This strategy employs a gradual removal of contextual information during training to simulate scenarios where context might be unavailable. This makes the model robust and adaptable to varying degrees of context availability at inference time.
Results
Offline Experiments:
- Performance: Augmented Unity outperformed context-free baselines substantially. Specifically, a 19.9% improvement in exact match P@100 was recorded when compared to the context-free Unity model.
- Efficiency: Despite processing significantly more tokens, the GPU serving costs remained similar to traditional models due to the efficient use of FiD architecture.
GPT-4 Evaluation: Augmented Unity showed consistent gains across languages, with a notable relative improvement of 12.6% and 17.7% for NLG and DR respectively, when GPT-4 was used as an evaluation judge.
Ablation Studies: These confirmed the contribution of different context types (e.g., web snippets, query rewrites) to the overall performance. The context glancing approach proved particularly beneficial, with significant performance improvements both in the absence and presence of full context signals.
Online A/B Testing:
- Revenue and Engagement: Augmented Unity achieved a 1% increase in ad revenue for English queries and a 1.4% increase for non-English queries. User engagement metrics, such as click-through rates, also showed significant improvements without negatively impacting ad quality.
Implications and Future Directions
Theoretical Implications: This work highlights the potential of integrating extensive contextual information via web searches and LLMs to enhance retrieval tasks in AI-powered systems. By demonstrating substantial gains in real-world applications, it opens new avenues for leveraging similar techniques in other AI-driven information retrieval ecosystems.
Practical Implications: For commercial search engines, employing Augmented Unity could lead to significantly increased revenue and user satisfaction. The practical deployment of such models further underscores the relevance of combining DR and NLG methodologies for operational efficiency and improved performance.
Future Developments: The promising results indicate a few potential research extensions:
- Contextual Optimization: Fine-tuning the balance between different context types and quantities to optimize performance further.
- Cross-lingual Applications: Expanding the multilingual capabilities to enhance retrieval in lesser-resourced languages.
- Integration with Other ML Systems: Exploring integration with other machine learning components within search engines could yield holistic enhancements across the system.
Conclusion
The paper presents a robust, context-aware retrieval model that significantly improves keyword matching in sponsored search scenarios. By leveraging rich contextual signals and optimizing model training strategies, Augmented Unity sets a new benchmark in retrieval performance, demonstrating both theoretical and practical advancements in the field of AI-powered information retrieval.
Create an account to read this summary for free: