Bo Yanan, Wang Yongqiang
Department of Electrical and Computer Engineering, Clemson University, Clemson, SC 29634.
Proc Natl Acad Sci U S A. 2024 Apr 23;121(17):e2319625121. doi: 10.1073/pnas.2319625121. Epub 2024 Apr 19.
Distributed nonconvex optimization underpins key functionalities of numerous distributed systems, ranging from power systems, smart buildings, cooperative robots, vehicle networks to sensor networks. Recently, it has also merged as a promising solution to handle the enormous growth in data and model sizes in deep learning. A fundamental problem in distributed nonconvex optimization is avoiding convergence to saddle points, which significantly degrade optimization accuracy. We find that the process of quantization, which is necessary for all digital communications, can be exploited to enable saddle-point avoidance. More specifically, we propose a stochastic quantization scheme and prove that it can effectively escape saddle points and ensure convergence to a second-order stationary point in distributed nonconvex optimization. With an easily adjustable quantization granularity, the approach allows a user to control the number of bits sent per iteration and, hence, to aggressively reduce the communication overhead. Numerical experimental results using distributed optimization and learning problems on benchmark datasets confirm the effectiveness of the approach.
分布式非凸优化是众多分布式系统关键功能的基础，这些系统涵盖电力系统、智能建筑、协作机器人、车辆网络以及传感器网络等。近来，它还作为一种有前景的解决方案出现，以应对深度学习中数据量和模型规模的巨大增长。分布式非凸优化中的一个基本问题是避免收敛到鞍点，因为这会显著降低优化精度。我们发现，所有数字通信所必需的量化过程可被利用来实现鞍点避免。具体而言，我们提出一种随机量化方案，并证明它能在分布式非凸优化中有效逃离鞍点并确保收敛到二阶驻点。该方法具有易于调整的量化粒度，允许用户控制每次迭代发送的比特数，从而大幅减少通信开销。使用基准数据集上的分布式优化和学习问题所进行的数值实验结果证实了该方法的有效性。
