Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Learning with Extremely Noisy Clients via Negative Distillation (2312.12703v2)

Published 20 Dec 2023 in cs.LG

Abstract: Federated learning (FL) has shown remarkable success in cooperatively training deep models, while typically struggling with noisy labels. Advanced works propose to tackle label noise by a re-weighting strategy with a strong assumption, i.e., mild label noise. However, it may be violated in many real-world FL scenarios because of highly contaminated clients, resulting in extreme noise ratios, e.g., $>$90%. To tackle extremely noisy clients, we study the robustness of the re-weighting strategy, showing a pessimistic conclusion: minimizing the weight of clients trained over noisy data outperforms re-weighting strategies. To leverage models trained on noisy clients, we propose a novel approach, called negative distillation (FedNed). FedNed first identifies noisy clients and employs rather than discards the noisy clients in a knowledge distillation manner. In particular, clients identified as noisy ones are required to train models using noisy labels and pseudo-labels obtained by global models. The model trained on noisy labels serves as a `bad teacher' in knowledge distillation, aiming to decrease the risk of providing incorrect information. Meanwhile, the model trained on pseudo-labels is involved in model aggregation if not identified as a noisy client. Consequently, through pseudo-labeling, FedNed gradually increases the trustworthiness of models trained on noisy clients, while leveraging all clients for model aggregation through negative distillation. To verify the efficacy of FedNed, we conduct extensive experiments under various settings, demonstrating that FedNed can consistently outperform baselines and achieve state-of-the-art performance. Our code is available at https://github.com/linChen99/FedNed.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. FOCUS: Dealing with label quality disparity in federated learning. arXiv preprint arXiv:2001.11359.
  2. Tackling data heterogeneity in federated learning with class prototypes. In AAAI.
  3. Federated Learning Based on Dynamic Regularization. In ICLR.
  4. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML.
  5. Robust loss functions under label noise for deep neural networks. In AAAI.
  6. Masking: A new perspective of noisy supervision. In NeurIPS.
  7. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS.
  8. To trust or not to trust a classifier. In NeurIPS.
  9. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1-2): 1–210.
  10. Unicon: Combating label noise through uniform selection and contrastive learning. In CVPR.
  11. Scaffold: Stochastic controlled averaging for federated learning. In ICML.
  12. FedRN: Exploiting k-Reliable neighbors towards robust federated learning. In CIKM.
  13. Nlnl: Negative learning for noisy labels. In ICCV.
  14. Learning multiple layers of features from tiny images. In Technical Report, 32–33. University of Toronto.
  15. DivideMix: Learning with noisy labels as semi-supervised learning. In ICLR.
  16. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2: 429–450.
  17. FedNoisy: Federated noisy label learning benchmark. arXiv preprint arXiv:2306.11650.
  18. Ensemble distillation for robust model fusion in federated learning. In NeurIPS.
  19. Does label smoothing mitigate label noise? In ICML.
  20. A state-of-the-art survey on solving non-IID data in federated learning. Future Generation Computer Systems, 135: 244–258.
  21. Communication-efficient learning of deep networks from decentralized data. In AISTATS.
  22. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3): 211–252.
  23. Meta-weight-net: Learning an explicit mapping for sample weighting. In NeurIPS.
  24. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems.
  25. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In ICML.
  26. Fednoil: A simple two-level sampling method for federated learning with noisy labels. arXiv preprint arXiv:2205.10110.
  27. FedNoRo: Towards noise-robust federated learning by addressing class imbalance and label noise heterogeneity. In IJCAI.
  28. FedNP: Towards non-IID federated learning via federated neural propagation. In AAAI.
  29. Robust early-learning: Hindering the memorization of noisy labels. In ICLR.
  30. Fedcorr: Multi-stage federated learning for label noise correction. In CVPR.
  31. Client selection for federated learning with label noise. IEEE Transactions on Vehicular Technology, 71(2): 2193–2197.
  32. Robust federated learning with noisy labels. IEEE Intelligent Systems, 37(2): 35–43.
  33. FedFed: Feature distillation against data heterogeneity in federated learning. In NeurIPS.
  34. Jo-SRC: A contrastive approach for combating noisy labels. In CVPR.
  35. Bayesian nonparametric federated learning of neural networks. In ICML.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - linChen99/FedNed (5 stars)