Yang Yahong, Chen Qipin, Hao Wenrui
Department of Mathematics, The Pennsylvania State University, University Park, State College, PA 16802, USA.
Amazon Prime Video, Seattle, MA 98109, USA.
J Sci Comput. 2025 Feb;102(2). doi: 10.1007/s10915-024-02761-5. Epub 2025 Jan 3.
In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates two key mechanisms: one involves building a homotopy activation function that seamlessly connects the linear activation function with the activation function; the other technique entails relaxing the homotopy parameter to enhance the training refinement process. We have conducted an in-depth analysis of this novel method within the context of the neural tangent kernel (NTK), revealing significantly improved convergence rates. Our experimental results, especially when considering networks with larger widths, validate the theoretical conclusions. This proposed HRTA exhibits the potential for other activation functions and deep neural networks.
在本文中，我们提出了一种名为同伦松弛训练算法（HRTA）的新颖训练方法，旨在与传统方法相比加速训练过程。我们的算法包含两个关键机制：一个是构建一个同伦激活函数，该函数将线性激活函数与激活函数无缝连接；另一种技术是放松同伦参数以增强训练优化过程。我们在神经切线核（NTK）的背景下对这种新方法进行了深入分析，结果显示收敛速度有显著提高。我们的实验结果，特别是考虑宽度较大的网络时，验证了理论结论。所提出的HRTA在其他激活函数和深度神经网络方面展现出潜力。
