Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
124 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning (2211.07864v4)

Published 15 Nov 2022 in cs.LG and cs.CV

Abstract: Federated learning (FL) enables multiple clients to collaboratively train a global model without disclosing their data. Previous researches often require training the complete model parameters. However, the emergence of powerful pre-trained models makes it possible to achieve higher performance with fewer learnable parameters in FL. In this paper, we propose a federated adaptive prompt tuning algorithm, FedAPT, for multi-domain collaborative image classification with powerful foundation models, like CLIP. Compared with direct federated prompt tuning, our core idea is to adaptively unlock specific domain knowledge for each test sample in order to provide them with personalized prompts. To implement this idea, we design an adaptive prompt tuning module, which consists of a meta prompt, an adaptive network, and some keys. The server randomly generates a set of keys and assigns a unique key to each client. Then all clients cooperatively train the global adaptive network and meta prompt with the local datasets and the frozen keys. Ultimately, the global aggregation model can assign a personalized prompt to CLIP based on the domain features of each test sample. We perform extensive experiments on two multi-domain image classification datasets across two different settings -- supervised and unsupervised. The results show that FedAPT can achieve better performance with less than 10\% of the number of parameters of the fully trained model, and the global model can perform well in diverse client domains simultaneously. The source code is available at \url{https://github.com/leondada/FedAPT}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. On pre-training for federated learning. arXiv preprint arXiv:2206.11488.
  2. A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597–1607. PMLR.
  3. Personalized Federated Learning with Moreau Envelopes. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  4. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
  5. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33: 3557–3568.
  6. Towards Instance-adaptive Inference for Federated Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 23287–23296.
  7. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, 2066–2073. IEEE.
  8. Caltech-256 object category dataset.
  9. PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models–Federated Learning in Age of Foundation Model. arXiv preprint arXiv:2208.11625.
  10. Lower bounds and optimal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 33: 2304–2315.
  11. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  12. Federated visual classification with real-world data distribution. In European Conference on Computer Vision, 76–92. Springer.
  13. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, 4904–4916. PMLR.
  14. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, 5132–5143. PMLR.
  15. Support cluster machine. In Proceedings of the 24th International Conference on Machine Learning, 505–512.
  16. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10713–10722.
  17. Federated Optimization in Heterogeneous Networks. In Dhillon, I. S.; Papailiopoulos, D. S.; and Sze, V., eds., MLSys. mlsys.org.
  18. FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  19. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
  20. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
  21. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602.
  22. Fedvision: An online visual object detection platform powered by federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 13172–13179.
  23. FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning. arXiv preprint arXiv:2302.13485.
  24. Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferring. In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvári, C.; Niu, G.; and Sabato, S., eds., International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, 14527–14541. PMLR.
  25. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, 1273–1282. PMLR.
  26. Where to Begin? Exploring the Impact of Pre-Training and Initialization in Federated. In International Conference on Learning Representations.
  27. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, 1406–1415.
  28. Federated adversarial domain adaptation. arXiv preprint arXiv:1911.02054.
  29. Text-driven Prompt Generation for Vision-Language Models in Federated Learning. arXiv preprint arXiv:2310.06123.
  30. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 8748–8763. PMLR.
  31. Adaptive federated optimization. arXiv preprint arXiv:2003.00295.
  32. Adapting visual category models to new domains. In European conference on computer vision, 213–226. Springer.
  33. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33.
  34. One-shot Federated Learning without server-side training. Neural Networks, 164: 203–215.
  35. Cross-domain Federated Object Detection. In 2023 IEEE International Conference on Multimedia and Expo (ICME), 1469–1474. IEEE.
  36. Federated learning from pre-trained models: A contrastive learning approach. Advances in neural information processing systems.
  37. Attention is all you need. Advances in neural information processing systems, 30.
  38. Federated Learning with Matched Averaging. In International Conference on Learning Representations.
  39. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Information Processing Systems 33.
  40. MedCLIP: Contrastive Learning from Unpaired Medical Images and Text. arXiv preprint arXiv:2210.10163.
  41. Exploring One-shot Semi-supervised Federated Learning with A Pre-trained Diffusion Model. arXiv preprint arXiv:2305.04063.
  42. Attracting and dispersing: A simple approach for source-free domain adaptation. In Advances in Neural Information Processing Systems.
  43. DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection. arXiv preprint arXiv:2209.09407.
  44. Federated learning with only positive labels. In International Conference on Machine Learning, 10946–10956. PMLR.
  45. Chinese Text Recognition with A Pre-Trained CLIP-Like Model Through Image-IDS Aligning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 11943–11952.
  46. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16793–16803.
  47. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16816–16825.
  48. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9): 2337–2348.
  49. Joint optimization in edge-cloud continuum for federated unsupervised person re-identification. In Proceedings of the 29th ACM International Conference on Multimedia, 433–441.
  50. Performance optimization of federated person re-identification via benchmark analysis. In Proceedings of the 28th ACM International Conference on Multimedia, 955–963.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now