DUQGen: Effective Unsupervised Domain Adaptation of Neural Rankers by Diversifying Synthetic Query Generation (2404.02489v1)

Abstract: State-of-the-art neural rankers pre-trained on large task-specific training data such as MS-MARCO, have been shown to exhibit strong performance on various ranking tasks without domain adaptation, also called zero-shot. However, zero-shot neural ranking may be sub-optimal, as it does not take advantage of the target domain information. Unfortunately, acquiring sufficiently large and high quality target training data to improve a modern neural ranker can be costly and time-consuming. To address this problem, we propose a new approach to unsupervised domain adaptation for ranking, DUQGen, which addresses a critical gap in prior literature, namely how to automatically generate both effective and diverse synthetic training data to fine tune a modern neural ranker for a new domain. Specifically, DUQGen produces a more effective representation of the target domain by identifying clusters of similar documents; and generates a more diverse training dataset by probabilistic sampling over the resulting document clusters. Our extensive experiments, over the standard BEIR collection, demonstrate that DUQGen consistently outperforms all zero-shot baselines and substantially outperforms the SOTA baselines on 16 out of 18 datasets, for an average of 4% relative improvement across all datasets. We complement our results with a thorough analysis for more in-depth understanding of the proposed method's performance and to identify promising areas for further improvements.

• The paper introduces a novel approach that automatically generates diverse synthetic query-document pairs to adapt neural rankers to specialized domains.
• It employs domain document clustering, few-shot LLM prompting, and negative pair mining to create high-quality training data without manual curation.
• Experimental evaluations on 18 BEIR datasets reveal a consistent 4% improvement over state-of-the-art methods.

DUQGen: Unsupervised Domain Adaptation for Neural Rankers through Diversified Query Generation

Introduction to DUQGen

The traditional approach to neural ranking involves training LLMs on extensive, general datasets like MS-MARCO, allowing them to learn domain-general features that can be applied across various tasks without domain adaptation. However, the performance of these models can often suffer when applied to specialized domains due to the lack of domain-specific training data. DUQGen addresses the challenges associated with acquiring high-quality, domain-specific training data for neural ranker fine-tuning by automating the generation of synthetic query-document pairs that are both representative of and diverse within the target domain. This approach overcomes the limitations of prior unsupervised domain adaptation methods, consistently improving upon state-of-the-art (SOTA) baselines across 16 out of 18 datasets within the BEIR benchmark.

Methodology behind DUQGen

DUQGen's methodology comprises four main components:

1. Domain Document Selection: This involves clustering documents within the target domain to capture representative samples. Probabilistic sampling over these clusters then ensures the selection of a diverse set of documents.
2. Synthetic Query Generation: Utilizing a pre-trained LLM, DUQGen generates synthetic queries from the sampled documents. This process is steered by a few-shot prompting approach, which leverages in-domain examples to produce more relevant queries.
3. Negative Pairs Mining: To build a comprehensive training dataset, DUQGen generates negative query-document pairs by leveraging first-stage retrievers like BM25, ensuring the training set contains a balanced mix of positive and negative examples.
4. Fine-tuning with Synthetic Data: The generated synthetic dataset is used to fine-tune any pre-trained neural ranker, adapting it to the specific target domain without necessitating manual data curation.

Comprehensive Evaluation

DUQGen was evaluated against various baselines using the BEIR benchmark, a collection of 18 diverse datasets covering different domains and tasks. The experiments demonstrated that DUQGen consistently outperforms zero-shot baselines and SOTA unsupervised domain adaptation methods, achieving an average of 4% relative improvement across all datasets. Notably, DUQGen's efficiency was highlighted by its ability to generate a smaller but more effective training dataset, significantly outperforming methods that rely on larger synthetic datasets for fine-tuning.

Theoretical and Practical Implications

• Improving Domain Adaptation: DUQGen offers a novel and effective approach to unsupervised domain adaptation for neural ranking, addressing the challenges associated with acquiring domain-specific training data. This has significant implications for the development of more adaptable and efficient neural ranking models.
• Reducing Dependency on Large Synthetic Datasets: By focusing on the quality rather than the quantity of synthetic training data, DUQGen reduces the computational resources required for fine-tuning neural rankers, making the domain adaptation process more accessible.

Future Directions

The success of DUQGen in leveraging LLMs for generating representative and diverse synthetic training data opens up new avenues for research, particularly in exploring other domains where acquiring labeled data is challenging. Future work could also entail refining the clustering and sampling techniques to further enhance the representativeness and diversity of the synthetic training data, potentially leading to even greater improvements in domain-adapted neural ranking performance.

Conclusion

DUQGen sets a new benchmark for unsupervised domain adaptation in neural ranking, offering a scalable and effective solution to the challenges of fine-tuning neural rankers for specialized domains. By generating high-quality, domain-specific training data, DUQGen enables significant improvements in ranking performance with minimal manual effort, signifying a substantial advancement in the field of information retrieval.