College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China; Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA 15261, USA.
Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA 15261, USA.
Neural Netw. 2022 Sep;153:224-234. doi: 10.1016/j.neunet.2022.06.004. Epub 2022 Jun 11.
In the paper, we study a class of novel stochastic composition optimization problems over Riemannian manifold, which have been raised by multiple emerging machine learning applications such as distributionally robust learning in Riemannian manifold setting. To solve these composition problems, we propose an effective Riemannian compositional gradient (RCG) algorithm, which has a sample complexity of O(ϵ) for finding an ϵ-stationary point. To further reduce sample complexity, we propose an accelerated momentum-based Riemannian compositional gradient (M-RCG) algorithm. Moreover, we prove that the M-RCG obtains a lower sample complexity of Õ(ϵ) without large batches, which achieves the best known sample complexity for its Euclidean counterparts. Extensive numerical experiments on training deep neural networks (DNNs) over Stiefel manifold and learning principal component analysis (PCA) over Grassmann manifold demonstrate effectiveness of our proposed algorithms. To the best of our knowledge, this is the first study of the composition optimization problems over Riemannian manifold.
在本文中，我们研究了一类新颖的随机复合优化问题，这些问题是由多个新兴的机器学习应用领域提出的，例如黎曼流形中的分布鲁棒学习。为了解决这些组合问题，我们提出了一种有效的黎曼复合梯度（RCG）算法，该算法的样本复杂度为 O(ϵ)，可以找到 ϵ-稳定点。为了进一步降低样本复杂度，我们提出了一种基于加速动量的黎曼复合梯度（M-RCG）算法。此外，我们证明 M-RCG 在不需要大数据集的情况下可以获得更低的样本复杂度 Õ(ϵ)，这达到了其欧几里得对应物的最佳已知样本复杂度。在 Stiefel 流形上训练深度神经网络（DNN）和在 Grassmann 流形上学习主成分分析（PCA）的大量数值实验表明了我们提出的算法的有效性。据我们所知，这是首次对黎曼流形上的组合优化问题进行研究。
