Measuring Data Similarity for Efficient Federated Learning: A Feasibility Study (2403.07450v1)
Abstract: In multiple federated learning schemes, a random subset of clients sends in each round their model updates to the server for aggregation. Although this client selection strategy aims to reduce communication overhead, it remains energy and computationally inefficient, especially when considering resource-constrained devices as clients. This is because conventional random client selection overlooks the content of exchanged information and falls short of providing a mechanism to reduce the transmission of semantically redundant data. To overcome this challenge, we propose clustering the clients with the aid of similarity metrics, where a single client from each of the formed clusters is selected in each round to participate in the federated training. To evaluate our approach, we perform an extensive feasibility study considering the use of nine statistical metrics in the clustering process. Simulation results reveal that, when considering a scenario with high data heterogeneity of clients, similarity-based clustering can reduce the number of required rounds compared to the baseline random client selection. In addition, energy consumption can be notably reduced from 23.93% to 41.61%, for those similarity metrics with an equivalent number of clients per round as the baseline random scheme.
- B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics. PMLR, 2017, pp. 1273–1282.
- O. Shahid, S. Pouriyeh, R. M. Parizi, Q. Z. Sheng, G. Srivastava, and L. Zhao, “Communication efficiency in federated learning: Achievements and challenges,” arXiv preprint arXiv:2107.10996, 2021.
- R. Balakrishnan, T. Li, T. Zhou, N. Himayat, V. Smith, and J. Bilmes, “Diverse client selection for federated learning via submodular maximization,” in Int. Conf. on Learning Representations, 2022.
- X. Ma, J. Zhu, Z. Lin, S. Chen, and Y. Qin, “A state-of-the-art survey on solving non-IID data in Federated Learning,” Future Generation Computer Systems, vol. 135, pp. 244–258, 2022.
- A. Visús, J. García, and F. Fernández, “A taxonomy of similarity metrics for markov decision processes,” arXiv preprint arXiv:2103.04706, 2021.
- F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, “Robust and communication-efficient federated learning from non-iid data,” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 9, pp. 3400–3413, 2019.
- Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” in 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 2022, pp. 965–978.
- J. Luo and S. Wu, “Fedsld: Federated Learning with Shared Label Distribution for Medical Image Classification,” Proceedings - International Symposium on Biomedical Imaging, 2022.
- Z. Wang, Y. Zhu, D. Wang, and Z. Han, “FedACS: Federated Skewness Analytics in Heterogeneous Decentralized Data Environments,” 2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS), 2021.
- H. Lu, A. Thelen, O. Fink, C. Hu, and S. Laflamme, “Federated learning with uncertainty-based client clustering for fleet-wide fault diagnosis,” arXiv preprint arXiv:2304.13275, 2023.
- R. Song, L. Lyu, W. Jiang, A. Festag, and A. Knoll, “V2X-Boosted Federated Learning for Cooperative Intelligent Transportation Systems with Contextual Client Selection,” arXiv preprint arXiv:2305.11654, 2023.
- Z. Jiang, Y. Xu, H. Xu, Z. Wang, and C. Qian, “Adaptive control of client selection and gradient compression for efficient federated learning,” arXiv preprint arXiv:2212.09483, 2022.
- Y. Fraboni, R. Vidal, L. Kameni, and M. Lorenzi, “Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learning,” vol. 139, 2021.
- C. Briggs, Z. Fan, and P. Andras, “Federated learning with hierarchical clustering of local updates to improve training on non-IID data,” in 2020 Int. Joint Conf. on Neural Networks (IJCNN). IEEE, 2020, pp. 1–9.
- M. Chen, Y. Wang, and H. V. Poor, “Performance optimization for wireless semantic communications over energy harvesting networks,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 8647–8651.
- U. Sara, M. Akter, and M. S. Uddin, “Image quality assessment through FSIM, SSIM, MSE and PSNR—a comparative study,” Journal of Computer and Communications, vol. 7, no. 3, pp. 8–18, 2019.
- Ş. Öztürk, “Comparison of Pairwise Similarity Distance Methods for Effective Hashing,” IOP Conference Series: Materials Science and Engineering, vol. 1099, 2021.
- F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,” Proceedings of the IEEE, vol. 109, no. 1, pp. 43–76, 2020.
- M. Ghifary, W. B. Kleijn, and M. Zhang, “Domain adaptive neural networks for object recognition,” in PRICAI 2014: Trends in Artificial Intelligence: 13th Pacific Rim Int. Conf. on Artificial Intelligence, Gold Coast, QLD, Australia, December 1-5, 2014. Proceedings 13. Springer, 2014, pp. 898–904.
- F. Zhuang, X. Cheng, P. Luo, S. J. Pan, and Q. He, “Supervised representation learning: Transfer learning with deep autoencoders,” in Twenty-fourth international joint conference on artificial intelligence, 2015.
- W.-H. Chen, P.-C. Cho, and Y.-L. Jiang, “Activity recognition using transfer learning,” Sens. Mater, vol. 29, no. 7, pp. 897–904, 2017.
- C.-Y. Lee, T. Batra, M. H. Baig, and D. Ulbricht, “Sliced wasserstein discrepancy for unsupervised domain adaptation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 10 285–10 295.
- P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” Journal of Computational and Applied Mathematics, vol. 20, p. 53–65, 1987.
- E. Guerra, F. Wilhelmi, M. Miozzo, and P. Dini, “The cost of training machine learning models over distributed data sources,” IEEE Open Journal of the Communications Society, vol. 4, pp. 1111–1126, 2023.
- Y. LeCun and C. Cortes, “Mnist handwritten digit database,” AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, vol. 7, 2010.
- V. Schmidt, K. Goyal, A. Joshi, B. Feld, L. Conell, N. Laskaris, D. Blank, J. Wilson, S. Friedler, and S. Luccioni, “Codecarbon: estimate and track carbon emissions from machine learning computing (2021),” Available: https://github.com/mlco2/codecarbon, 2021.
- V. M. Panaretos and Y. Zemel, “Statistical Aspects of Wasserstein Distances,” Annual Review of Statistics and Its Application, vol. 6, no. 1, pp. 405–431, 2019. [Online]. Available: https://doi.org/10.1146%2Fannurev-statistics-030718-104938