Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Learning with Convex Global and Local Constraints (2310.10117v3)

Published 16 Oct 2023 in cs.LG and math.OC

Abstract: In practice, many ML problems come with constraints, and their applied domains involve distributed sensitive data that cannot be shared with others, e.g., in healthcare. Collaborative learning in such practical scenarios entails federated learning (FL) for ML problems with constraints, or FL with constraints for short. Despite the extensive developments of FL techniques in recent years, these techniques only deal with unconstrained FL problems or FL problems with simple constraints that are amenable to easy projections. There is little work dealing with FL problems with general constraints. To fill this gap, we take the first step toward building an algorithmic framework for solving FL problems with general constraints. In particular, we propose a new FL algorithm for constrained ML problems based on the proximal augmented Lagrangian (AL) method. Assuming convex objective and convex constraints plus other mild conditions, we establish the worst-case complexity of the proposed algorithm. Our numerical experiments show the effectiveness of our algorithm in performing Neyman-Pearson classification and fairness-aware learning with nonconvex constraints, in an FL setting.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. A reductions approach to fair classification. In International Conference on Machine Learning, pages 60–69. PMLR, 2018.
  2. A primal-dual method for conic constrained distributed optimization problems. Advances in Neural Information Processing Systems, 29, 2016.
  3. A distributed ADMM-like method for resource sharing over time-varying networks. SIAM Journal on Optimization, 29(4):3036–3068, 2019.
  4. An augmented Lagrangian method for conic convex programming. arXiv preprint arXiv:1302.6322, 2013.
  5. Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, 2nd edition, 2017.
  6. Complexity and performance of an augmented Lagrangian algorithm. Optimization Methods and Software, 35(5):885–920, 2020.
  7. Sequential quadratic programming. Acta Numerica, 4:1–51, 1995.
  8. A trust region algorithm for nonlinearly constrained optimization. SIAM Journal on Numerical Analysis, 24(5):1152–1170, 1987.
  9. Classification with fairness constraints: A meta-algorithm with provable guarantees. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 319–328, 2019.
  10. An Introduction to Structural Optimization, volume 153. Springer Science & Business Media, 2008.
  11. FedFair: Training fair models in cross-silo federated learning. arXiv preprint arXiv:2109.05662, 2021.
  12. A sequential quadratic programming algorithm for nonconvex, nonsmooth constrained optimization. SIAM Journal on Optimization, 22(2):474–500, 2012.
  13. Fairness-aware agnostic federated learning. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 181–189. SIAM, 2021.
  14. Enforcing fairness in private federated learning via the modified method of differential multipliers. In NeurIPS 2021 Workshop Privacy in Machine Learning, 2021.
  15. Combining ADMM and the augmented Lagrangian method for efficiently handling many constraints. In International Joint Conference on Artificial Intelligence, pages 4525–4531, 2019.
  16. FedADMM: A robust federated deep learning framework with adaptivity to system heterogeneity. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 2575–2587. IEEE, 2022.
  17. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  18. On the complexity of an augmented Lagrangian method for nonconvex optimization. IMA Journal of Numerical Analysis, 41(2):1546–1568, 2021.
  19. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, 2nd edition, 2009.
  20. A Newton-CG based barrier-augmented Lagrangian method for general nonconvex conic optimization. arXiv preprint arXiv:2301.04204, 2023.
  21. A Newton-CG based augmented Lagrangian method for finding a second-order stationary point of nonconvex equality constrained optimization with complexity guarantees. SIAM Journal on Optimization, 33(3):1734–1766, 2023.
  22. Prox-PDA: The proximal primal-dual algorithm for fast distributed nonconvex optimization and learning over networks. In International Conference on Machine Learning, pages 1529–1538. PMLR, 2017.
  23. An accelerated variance reduced extra-point approach to finite-sum VI and optimization. arXiv preprint arXiv:2211.03269, 2022.
  24. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  25. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132–5143. PMLR, 2020.
  26. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.
  27. Iteration complexity of an inner accelerated inexact proximal augmented Lagrangian method based on the classical Lagrangian function. SIAM Journal on Optimization, 33(1):181–210, 2023.
  28. Iteration-complexity of first-order augmented Lagrangian methods for convex programming. Mathematical Programming, 155(1-2):511–547, 2016.
  29. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering, 2021.
  30. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357–6368. PMLR, 2021.
  31. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429–450, 2020.
  32. FedBN: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021.
  33. Rate-improved inexact augmented Lagrangian method for constrained nonconvex optimization. In International Conference on Artificial Intelligence and Statistics, pages 2170–2178. PMLR, 2021.
  34. Distributed multi-agent optimization subject to nonidentical constraints and communication delays. Automatica, 65:120–131, 2016.
  35. On the global linear convergence of the ADMM with multiblock variables. SIAM Journal on Optimization, 25(3):1478–1497, 2015.
  36. Privacy-preserving traffic flow prediction: A federated learning approach. IEEE Internet of Things Journal, 7(8):7751–7763, 2020.
  37. Federated learning for open banking. In Federated Learning: Privacy and Incentive, pages 240–254. Springer, 2020.
  38. Songtao Lu. A single-loop gradient descent and perturbed ascent algorithm for nonconvex functional constrained optimization. In International Conference on Machine Learning, pages 14315–14357. PMLR, 2022.
  39. Accelerated first-order methods for convex optimization with locally Lipschitz continuous gradient. SIAM Journal on Optimization, 33(3):2275–2310, 2023.
  40. Iteration-complexity of first-order augmented Lagrangian methods for convex conic programming. SIAM Journal on Optimization, 33(2):1159–1190, 2023.
  41. Self-adaptive physics-informed neural networks using a soft attention mechanism. arXiv preprint arXiv:2009.04544, 2020.
  42. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017.
  43. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1–35, 2021.
  44. Communication-efficient federated learning for wireless edge intelligence in IoT. IEEE Internet of Things Journal, 7(7):5986–5994, 2019.
  45. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning, pages 15750–15769. PMLR, 2022.
  46. Complexity of first-order inexact Lagrangian and penalty methods for conic convex programming. Optimization Methods and Software, 34(2):305–335, 2019.
  47. Constrained consensus and optimization in multi-agent networks. IEEE Transactions on Automatic Control, 55(4):922–938, 2010.
  48. Yurii Nesterov et al. Lectures on convex optimization, volume 137. Springer, 2018.
  49. Office for Civil Rights (OCR). Hipaa home, Aug 2023.
  50. FedSplit: An algorithmic framework for fast federated optimization. In Advances in neural information processing systems, volume 33, pages 7057–7066, 2020.
  51. Adaptive inexact fast augmented Lagrangian methods for constrained convex optimization. Optimization Letters, 11:609–626, 2017.
  52. Evaluation of federated learning variations for COVID-19 diagnosis using chest radiographs from 42 US and European hospitals. Journal of the American Medical Informatics Association, 30(1):54–63, 2023.
  53. A systematic evaluation of federated learning on biomedical natural language processing. In International Workshop on Federated Learning for Distributed Data Mining, 2023.
  54. A trust region algorithm for equality constrained optimization. Mathematical Programming, 49(1):189–211, 1991.
  55. The future of digital health with federated learning. NPJ Digital Medicine, 3(1):119, 2020.
  56. Neyman-Pearson classification, convexity and stochastic constraints. Journal of Machine Learning Research, 2011.
  57. R Tyrrell Rockafellar. Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Mathematics of Operations Research, 1(2):97–116, 1976.
  58. R Tyrrell Rockafellar and Roger J-B Wets. Variational Analysis, volume 317. Springer Science & Business Media, 2009.
  59. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):e0118432, 2015.
  60. Robust and communication-efficient federated learning from non-IID data. IEEE Transactions on Neural Networks and Learning Systems, 31(9):3400–3413, 2019.
  61. Clayton Scott. Performance measures for Neyman-Pearson classification. IEEE Transactions on Information Theory, 53(8):2852–2863, 2007.
  62. An agnostic approach to federated learning with class imbalance. In International Conference on Learning Representations, 2021.
  63. Neyman-Pearson classification algorithms and NP receiver operating characteristics. Science Advances, 4(2):eaao1659, 2018.
  64. A survey on Neyman-Pearson classification and suggestions for future research. Wiley Interdisciplinary Reviews: Computational Statistics, 8(2):64–81, 2016.
  65. FedDR–randomized Douglas-Rachford splitting algorithms for nonconvex federated composite optimization. In Advances in Neural Information Processing Systems, volume 34, pages 30326–30338, 2021.
  66. Welfare and fairness dynamics in federated learning: A client selection perspective. Statistics and Its Interface, 2023.
  67. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical Programming, 106:25–57, 2006.
  68. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Information Processing Systems, 33:7611–7623, 2020.
  69. Distributed subgradient-based multiagent optimization with more general step sizes. IEEE Transactions on Automatic Control, 63(7):2295–2302, 2017.
  70. Yangyang Xu. Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming. Mathematical Programming, 185:199–244, 2021.
  71. A survey of distributed optimization. Annual Reviews in Control, 47:278–305, 2019.
  72. Constrained bi-level optimization: Proximal Lagrangian value function approach and Hessian-free algorithm. In International Conference on Machine Learning, 2024.
  73. Distributed primal-dual subgradient method for multiagent optimization via consensus algorithms. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 41(6):1715–1724, 2011.
  74. Federated composite optimization. In International Conference on Machine Learning, pages 12253–12266. PMLR, 2021.
  75. First-order algorithms without Lipschitz gradient: A sequential local optimization approach. INFORMS Journal on Optimization, 2024.
  76. FedPD: A federated learning framework with adaptivity to non-iid data. IEEE Transactions on Signal Processing, 69:6055–6070, 2021.
  77. Federated learning via inexact ADMM. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  78. On distributed convex optimization under inequality and equality constraints. IEEE Transactions on Automatic Control, 57(1):151–164, 2011.
  79. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8697–8710, 2018.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets