Enhancing Domain Adaptation through Prompt Gradient Alignment (2406.09353v3)

Abstract: Prior Unsupervised Domain Adaptation (UDA) methods often aim to train a domain-invariant feature extractor, which may hinder the model from learning sufficiently discriminative features. To tackle this, a line of works based on prompt learning leverages the power of large-scale pre-trained vision-LLMs to learn both domain-invariant and specific features through a set of domain-agnostic and domain-specific learnable prompts. Those studies typically enforce invariant constraints on representation, output, or prompt space to learn such prompts. In contrast, we cast UDA as a multiple-objective optimization problem in which each objective is represented by a domain loss. Under this new framework, we propose to align per-objective gradients to foster consensus between them. Additionally, to prevent potential overfitting when fine-tuning this deep learning architecture, we penalize the norm of these gradients. To achieve these goals, we devise a practical gradient update procedure that can work under both single-source and multi-source UDA. Empirically, our method consistently outperforms other vision-LLM adaptation methods. The implementation is available at https://github.com/VietHoang1512/PGA.

• The paper presents a novel approach to unsupervised domain adaptation by reframing the problem as multiple-objective optimization via prompt gradient alignment.
• It leverages domain-agnostic and domain-specific prompts with gradient norm penalization to balance feature discriminability and generalization.
• Empirical results on benchmarks like ImageCLEF, Office-Home, and DomainNet demonstrate superior performance over traditional UDA methods.

Enhancing Domain Adaptation through Prompt Gradient Alignment

Introduction

The paper "Enhancing Domain Adaptation through Prompt Gradient Alignment" presents a novel approach to Unsupervised Domain Adaptation (UDA) by leveraging large-scale pre-trained vision-LLMs through prompt learning. Traditional UDA methods focus on creating domain-invariant features, which may compromise feature discriminability. Instead, this method reframes UDA as a multiple-objective optimization problem, with each task represented by a domain loss. By aligning per-objective gradients, the authors aim to foster consensus between source and target domains. The procedure includes fine-tuning through prompt learning and penalizing gradient norms to ensure the model's generalization capability.

Framework and Implementation

The proposed method employs prompt learning on CLIP-based models, which utilize vision-language components to achieve high generalization. In the approach, domain-specific and domain-agnostic prompts are constructed. This allows for capturing shared knowledge across domains while maintaining specific features.

1. Prompt Design:
   • Domain-Agnostic Prompt $\boldsymbol{P}_{sh}$: Shared among all domains, includes class-specific tokens.
   • Domain-Specific Prompts $\boldsymbol{P}_{S,i}$ and $\boldsymbol{P}_T$: Tailored for individual source and target domains.
2. Gradient Alignment Strategy:
   • Aligning gradients involves maximizing the cosine similarity between the gradients of source and target losses. This encourages learning domain-common features that reduce loss for both domains.

     Figure 1: Performance of ERM and PGA on the in-domain data (validation set) and out-of-distribution data (test set). Average results and shaded standard errors are obtained from 10 random seeds.

3. Gradient Norm Penalization:
   • To counteract overfitting, the method penalizes the norm of gradients. This steers optimization toward flatter minima, thus enhancing the model's generalization potential.

Theoretical Analysis

The paper proposes an information-theoretic generalization bound to explain the benefits of prompt gradient alignment. Under the assumption of R-subgaussian loss functions, the analysis provides bounds on the generalization error and suggests reductions through gradient norm penalization and inter-domain gradient matching. This theoretical insight supports the intuition that gradient alignment and norm penalties can significantly improve prediction accuracy on target domains.

Experimental Validation

Empirical tests demonstrate that the proposed Prompt Gradient Alignment (PGA) approach surpasses existing UDA methods across various benchmarks like ImageCLEF, Office-Home, and DomainNet. Results indicate marked improvements in scenarios where CLIP's zero-shot performance is robust, and even in challenging domains like QuickDraw. Multi-source extensions of PGA, named MPGA, showcase state-of-the-art results through careful inter-domain gradient alignment across several domains.

Conclusion

The method "Enhancing Domain Adaptation through Prompt Gradient Alignment" defines a practical and theoretically grounded process for UDA by utilizing pre-trained vision-LLMs and adaptable prompt learning. Through gradient alignment and norm penalization, this approach achieves superior performance on benchmark datasets. Future explorations could explore sophisticated prompt designs to further narrow domain shifts, expanding the applicability and efficiency of the gradient alignment strategy.