Ren Yaoyao, Cao Yu, Ye Chengyin, Cheng Xu
School of Information and Control Engineering, Liaoning Petrochemical University, Fushun, Liaoning, People's Republic of China.
School of Economics and Management, Shenyang Agricultural University, Shengyang, Liaoning, People's Republic of China.
Sci Rep. 2023 Jul 19;13(1):11658. doi: 10.1038/s41598-023-38916-x.
Federated learning enables multiple nodes to perform local computations and collaborate to complete machine learning tasks without centralizing private data of nodes. However, the frequent model gradients upload/download operations required by the framework result in high communication costs, which have become the main bottleneck for federated learning as deep models scale up, hindering its performance. In this paper, we propose a two-layer accumulated quantized compression algorithm (TLAQC) that effectively reduces the communication cost of federated learning. TLAQC achieves this by reducing both the cost of individual communication and the number of global communication rounds. TLAQC introduces a revised quantization method called RQSGD, which employs zero-value correction to mitigate ineffective quantization phenomena and minimize average quantization errors. Additionally, TLAQC reduces the frequency of gradient information uploads through an adaptive threshold and parameter self-inspection mechanism, further reducing communication costs. It also accumulates quantization errors and retained weight deltas to compensate for gradient knowledge loss. Through quantization correction and two-layer accumulation, TLAQC significantly reduces precision loss caused by communication compression. Experimental results demonstrate that RQSGD achieves an incidence of ineffective quantization as low as 0.003% and reduces the average quantization error to 1.6 × [Formula: see text]. Compared to full-precision FedAVG, TLAQC compresses uploaded traffic to only 6.73% while increasing accuracy by 1.25%.
联邦学习使多个节点能够在不集中节点私有数据的情况下执行本地计算并协作完成机器学习任务。然而，该框架所需的频繁模型梯度上传/下载操作导致了高昂的通信成本，随着深度模型规模的扩大，这已成为联邦学习的主要瓶颈，阻碍了其性能。在本文中，我们提出了一种两层累积量化压缩算法（TLAQC），该算法有效地降低了联邦学习的通信成本。TLAQC通过降低单次通信成本和全局通信轮数来实现这一目标。TLAQC引入了一种名为RQSGD的改进量化方法，该方法采用零值校正来减轻无效量化现象并最小化平均量化误差。此外，TLAQC通过自适应阈值和参数自检机制降低梯度信息上传的频率，进一步降低通信成本。它还累积量化误差和保留的权重增量以补偿梯度知识损失。通过量化校正和两层累积，TLAQC显著降低了通信压缩导致的精度损失。实验结果表明，RQSGD实现了低至0.003%的无效量化发生率，并将平均量化误差降低至1.6×[公式：见原文]。与全精度FedAVG相比，TLAQC将上传流量压缩至仅6.73%，同时提高了1.25%的准确率。
