Federated Learning Aggregation Algorithm Based on AP Clustering Algorithm

doi:10.3969/j.issn.1006-2475.2024.04.002

Abstract

Abstract:
Abstract: In traditional federation learning， multiple clients’ local models are trained independently from their private data， and the central server generates a shared global model by aggregating the local models. However， due to statistical heterogeneity such as non-independent identically distributed （Non-IID） data， a global model often cannot be adapted to each client. To address this problem， this paper proposes an AP clustering algorithm-based federation learning aggregation algorithm （APFL） for Non-IID data. In APFL， the server calculates the similarity matrix between each client based on the data characteristics of the clients， and then uses the AP clustering algorithm to divide the clients into different clusters and construct a polycentric framework to calculate the suitable personalized model weights for each client. This algorithm is experimented on FMINST dataset and CIFAR10 dataset， and APFL improves 1.88 percentage points on FMNIST dataset and 6.08 percentage points on CIFAR10 dataset compared with traditional Federated Learning FedAvg. The results show that the proposed APFL improves the accuracy performance of Federated Learning on Non-IID data in this paper.

Key words: Key words: federal learning： non-independent identical distribution： AP clustering algorithm

CLC Number:

TP309

AO Bochao, FAN Bingbing. Federated Learning Aggregation Algorithm Based on AP Clustering Algorithm[J]. Computer and Modernization, 2024, 0(04): 5-11.

References

［1］周志华. 机器学习［M］. 北京:清华大学出版社， 2016.
［2］ MATER A C， COOTE M L. Deep learning in chemistry［J］. Journal of Chemical Information and Modeling， 2019，59（6）:2545-2559.
［3］刘铁岩，陈薇，王太峰，等. 分布式机器学习算法、理论与实践［M］. 北京:机械工业出版社， 2018.
［4］ SO J， GULER B， AVESTIMEHR A S. CodedPrivateML: A fast and privacy-preserving framework for distributed machine learning［J］. IEEE Journal on Selected Areas in Information Theory， 2021，2（1）:441-451.
［5］ SHOKRI R， SHMATIKOV V. Privacy-preserving deep learning［C］// Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. ACM， 2015:1310-1321.
［6］ YANG Q， LIU Y， CHEN T J， et al. Federated machine learning: Concept and applications［J］. ACM Transactions on Intelligent Systems and Technology （TIST）， 2019，10（2）. DOI: 10.1145/3298981.
［7］ HSIEH K， PHANISHAYEE A， MUTLU O， et al. The non-IID data quagmire of decentralized machine learning［C］// Proceedings of the 37th International Conference on Machine Learning. ACM， 2020:4387-4398.
［8］ ZHENG S X， MENG Q， WANG T F， et al. Asynchronous stochastic gradient descent with delay compensation［C］// Proceedings of the 34th International Conference on Machine Learning. ACM， 2017:4120-4129.
［9］ WANG S Q， TUOR T， SALONIDIS T， et al. Adaptive federated learning in resource constrained edge computing systems［J］. IEEE Journal on Selected Areas in Communications， 2019，37（6）:1205-1221.
［10］郭桂娟，田晖，皮慧娟，等. 面向非独立同分布数据的联邦学习研究进展［J］. 小型微型计算机系统， 2023，44（11）:2442-2449.
［11］ NGUYEN H T， SEHWAG V， HOSSEINALIPOUR S， et al. Fast-convergent federated learning［J］. IEEE Journal on Selected Areas in Communications， 2021，39（1）:201-218.
［12］ MCMAHAN H B， MOORE E， RAMAGE D， et al. Communication-efficient learning of deep networks from decentralized data［C］// Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. JMLR， 2017:1273-1282.
［13］ ZHAO Y， LI M， LAI L Z， et al. Federated learning with non-IID data［J］. arXiv preprint arXiv:1806.00582， 2018.
［14］ LI T， SAHU A K， ZAHEER M， et al. Federated optimization in heterogeneous networks［J］. arXiv preprint arXiv:1812.06127， 2018.
［15］ KARIMIREDDY S P， KALE S， MOHRI M， et al. Scaffold: Stochastic controlled averaging for federated learning［C］// Proceedings of the 37th International Conference on Machine Learning. JMLR， 2020:5132-5143.
［16］ WANG J Y， LIU Q H， LIANG H， et al. Tackling the objective inconsistency problem in heterogeneous federated optimization［J］. arXiv preprint arXiv:2007.07481， 2020.
［17］ LIN T， KONG L J， STICH S U， et al. Ensemble distillation for robust model fusion in federated learning［J］. arXiv preprint arXiv:2006.07242， 2020.
［18］ MANSOUR Y， MOHRI M， RO J， et al. Three approaches for personalization with applications to federated learning［J］. arXiv preprint arXiv:2002.10619， 2020.
［19］ KOPPARAPU K， LIN E. FedFMC: Sequential efficient federated learning on non-IID data［J］. arXiv preprint arXiv:2006.10937， 2020.
［20］ GHOSH A， HONG J， YIN D， et al. Robust federated learning in a heterogeneous environment［J］. arXiv preprint arXiv:1906.06629， 2019.
［21］ GHOSH A， CHUNG J， YIN D， et al. An efficient framework for clustered federated learning［J］. arXiv preprint arXiv:2006.04088， 2020.
［22］ BRIGGS C， FAN Z， ANDRAS P. Federated learning with hierarchical clustering of local updates to improve training on non-IID data［C］// Proceedings of the 2020 International Joint Conference on Neural Networks （IJCNN）. IEEE， 2020. DOI: 10.1109/IJCNN48605.2020.9207469.
［23］ SATTLER F， MULLER K R， SAMEK W. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints［J］. IEEE Transactions on Neural Networks and Learning Systems， 2021，32（8）:3710-3722.
［24］ LONG G D， XIE M， SHEN T， et al. Multi-center federated learning: Clients clustering for better personalization［J］. arXiv preprint arXiv:2108.08647， 2021.
［25］常黎明，刘颜红，徐恕贞. 基于数据分布的聚类联邦学习［J］. 计算机应用研究， 2023，40（6）:1697-1701.
［26］ FREY B J， DUECK D. Clustering by passing messages between data points［J］. Science， 2007，315（5814）:972-976.
［27］ ALAMI N， MEKNASSI M， EN-NAHNAHI N， et al. Unsupervised neural networks for automatic Arabic text summarization using document clustering and topic modeling［J］. Expert Systems with Applications， 2021，172. DOI: 10.1016/j.eswa.2021.114652.

[1]	ZENG Zhong-jing-xin, GAN Gang. Side Channel Analysis Based on Convolutional Auto-encoder [J]. Computer and Modernization, 2024, 0(03): 110-114.
[2]	HAN Dong-song, SHA Le-tian, ZHAO Chuang-ye. Worm and Agent-based Attack Modeling for Industrial Control Systems [J]. Computer and Modernization, 2023, 0(10): 107-114.