Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Privacy Preserving Bayesian Federated Learning in Heterogeneous Settings (2306.07959v1)

Published 13 Jun 2023 in cs.LG

Abstract: In several practical applications of federated learning (FL), the clients are highly heterogeneous in terms of both their data and compute resources, and therefore enforcing the same model architecture for each client is very limiting. Moreover, the need for uncertainty quantification and data privacy constraints are often particularly amplified for clients that have limited local data. This paper presents a unified FL framework to simultaneously address all these constraints and concerns, based on training customized local Bayesian models that learn well even in the absence of large local datasets. A Bayesian framework provides a natural way of incorporating supervision in the form of prior distributions. We use priors in the functional (output) space of the networks to facilitate collaboration across heterogeneous clients. Moreover, formal differential privacy guarantees are provided for this framework. Experiments on standard FL datasets demonstrate that our approach outperforms strong baselines in both homogeneous and heterogeneous settings and under strict privacy constraints, while also providing characterizations of model uncertainties.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2021.
  2. GP-Tree: A Gaussian process classifier for few-shot incremental learning. In Proceedings of the 38th International Conference on Machine Learning, pages 54–65. PMLR, 2021a.
  3. Personalized federated learning with gaussian processes. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021b.
  4. A general survey of privacy-preserving data mining models and algorithms. In Privacy-preserving data mining, pages 11–52. Springer, 2008.
  5. Federated learning via posterior averaging: A new perspective and practical algorithms, 2021.
  6. The privacy blanket of the shuffle model. In Alexandra Boldyreva and Daniele Micciancio, editors, Advances in Cryptology – CRYPTO 2019, pages 638–667, Cham, 2019. Springer International Publishing.
  7. WAFFLE: weighted averaging for personalized federated learning. CoRR, abs/2110.06978, 2021.
  8. Bayesian federated learning via predictive distribution distillation, 2022.
  9. Weight uncertainty in neural network. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1613–1622, Lille, France, 07–09 Jul 2015. PMLR.
  10. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Martin Hirt and Adam Smith, editors, Theory of Cryptography, pages 635–658, Berlin, Heidelberg, 2016. Springer Berlin Heidelberg. ISBN 978-3-662-53641-4.
  11. Leaf: A benchmark for federated settings, 2019.
  12. Bayesian federated learning: A survey, 2023.
  13. Fed{be}: Making bayesian model ensemble applicable to federated learning. In International Conference on Learning Representations, 2021.
  14. Exploiting shared representations for personalized federated learning. arXiv preprint arXiv:2102.07078, 2021.
  15. Federated bayesian optimization via thompson sampling. CoRR, abs/2010.10154, 2020. URL https://arxiv.org/abs/2010.10154.
  16. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3–4):211–407, 2014. ISSN 1551-305X.
  17. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  18. Daniel Flam-Shepherd. Mapping gaussian process priors to bayesian neural networks. 2017.
  19. Robbing the fed: Directly obtaining private data in federated learning with modified models. ArXiv, abs/2110.13057, 2021.
  20. Decepticons: Corrupted transformers breach privacy in federated learning for language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=r0BrY4BiEXO.
  21. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS ’15, page 1322–1333. Association for Computing Machinery, 2015. ISBN 9781450338325.
  22. Data mining in distributed environment: a survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 7:e1216, 07 2017. doi: 10.1002/widm.1216.
  23. Differentially Private Federated Learning: A Client Level Perspective. ArXiv e-prints, December 2017.
  24. Scalable and differentially private distributed aggregation in the shuffled model. ArXiv, abs/1906.08320, 2019.
  25. An efficient framework for clustered federated learning. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  26. Shuffled model of differential privacy in federated learning. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2521–2529. PMLR, 13–15 Apr 2021.
  27. Personalized federated learning with differential privacy. IEEE Internet of Things Journal, 7(10):9530–9539, 2020. doi: 10.1109/JIOT.2020.2991416.
  28. Improving federated learning personalization via model agnostic meta learning. CoRR, abs/1909.12488, 2019.
  29. The composition theorem for differential privacy. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1376–1385, Lille, France, 07–09 Jul 2015. PMLR.
  30. H. Kargupta and B.-H. Park. Collective data mining: A new perspective toward distributed data mining. In H. Kargupta and P. Chan, editors, Advances in Distributed and Parallel Knowledge Discovery. AAAI/MIT Press, 2000.
  31. Distributed clustering using collective principal component analysis. Knowledge and Information Systems Journal Special Issue on Distributed and Parallel Knowledge Discovery, 3:422–448, 2001.
  32. Adaptive gradient-based meta-learning methods. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5915–5926, 2019.
  33. Federated optimization in heterogeneous networks. In Inderjit S. Dhillon, Dimitris S. Papailiopoulos, and Vivienne Sze, editors, Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org, 2020a.
  34. Ditto: Fair and robust federated learning through personalization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 6357–6368. PMLR, 2021.
  35. Secure federated averaging algorithm with differential privacy. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6, 2020b. doi: 10.1109/MLSP49062.2020.9231531.
  36. Ensemble distillation for robust model fusion in federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2351–2363. Curran Associates, Inc., 2020.
  37. A bayesian federated learning framework with online laplace approximation. 2021.
  38. Architecture agnostic federated learning for neural networks. In International Conference on Machine Learning, 2022.
  39. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273–1282. PMLR, 20–22 Apr 2017a.
  40. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273–1282. PMLR, 20–22 Apr 2017b.
  41. S. Merugu and J. Ghosh. A distributed learning framework for heterogeneous data sources. In Proc. KDD, pages 208–217, 2005.
  42. S. Merugu and J. Ghosh. Privacy perserving distributed clustering using generative models. In Proc. ICDM, pages 211–218, Nov, 2003.
  43. Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 263–275, 2017. doi: 10.1109/CSF.2017.11.
  44. Differentially private federated learning on heterogeneous data. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 10110–10145. PMLR, 28–30 Mar 2022.
  45. How to combine variational bayesian networks in federated learning. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022), 2022. URL https://openreview.net/forum?id=AkPwb9dvAlP.
  46. Semi-supervised knowledge transfer for deep learning from private training data. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=HkwoSDPgg.
  47. Fedsplit: an algorithmic framework for fast federated optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7057–7066. Curran Associates, Inc., 2020.
  48. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Trans. Neural Networks Learn. Syst., 32(8):3710–3722, 2021.
  49. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, pages 9489–9502. PMLR, 2021.
  50. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33, 2020.
  51. Federated multi-task learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017a.
  52. Federated multi-task learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4424–4434, 2017b.
  53. FUNCTIONAL VARIATIONAL BAYESIAN NEURAL NETWORKS. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rkxacs0qY7.
  54. All You Need is a Good Functional Prior for Bayesian Deep Learning. Journal of Machine Learning Research, 23:1–56, 2022.
  55. A. Triastcyn and B. Faltings. Federated learning with bayesian differential privacy. In 2019 IEEE International Conference on Big Data (Big Data), pages 2587–2596. IEEE Computer Society, 2019.
  56. Federated learning for iout: Concepts, applications, challenges and future directions. IEEE Internet of Things Magazine, 5(4):36–41, 2022. doi: 10.1109/IOTM.001.2200067.
  57. Federated learning with matched averaging. In International Conference on Learning Representations, 2020.
  58. Flexifed: Personalized federated learning for edge clients with heterogeneous model architectures. In Proceedings of the ACM Web Conference 2023, WWW ’23, page 2979–2990, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394161. doi: 10.1145/3543507.3583347. URL https://doi.org/10.1145/3543507.3583347.
  59. Fishing for user data in large-batch federated learning via gradient magnification. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23668–23684. PMLR, 17–23 Jul 2022.
  60. Stochastic variational deep kernel learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_files/paper/2016/file/bcc0d400288793e8bdcd7c19a8ac0c2b-Paper.pdf.
  61. Fedloc: Federated learning framework for data-driven cooperative localization and location data processing. IEEE Open Journal of Signal Processing, 1:187–215, 2020.
  62. Privacy-preserving svm using nonlinear kernels on horizontally partitioned data. In Proceedings of the 2006 ACM Symposium on Applied Computing, SAC ’06, page 603–610, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595931082. doi: 10.1145/1141277.1141415.
  63. Differentially private model publishing for deep learning. In 2019 IEEE Symposium on Security and Privacy (SP), pages 332–349, 2019. doi: 10.1109/SP.2019.00019.
  64. Salvaging federated learning by local adaptation. CoRR, abs/2002.04758, 2020.
  65. Bayesian nonparametric federated learning of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7252–7261, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
  66. Fedpd: A federated learning framework with adaptivity to non-iid data. IEEE Transactions on Signal Processing, 69:6055–6070, 2021. doi: 10.1109/TSP.2021.3115952.
  67. Personalized federated learning via variational Bayesian inference. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26293–26310. PMLR, 17–23 Jul 2022.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now