Zhang Yongliang, Huang He, Shen Gangxiang
IEEE Trans Neural Netw Learn Syst. 2023 Sep;34(9):6313-6327. doi: 10.1109/TNNLS.2021.3135553. Epub 2023 Sep 1.
Complex-valued limited-memory BFGS (CL-BFGS) algorithm is efficient for the training of complex-valued neural networks (CVNNs). As an important parameter, the memory size represents the number of saved vector pairs and would essentially affect the performance of the algorithm. However, the determination of a suitable memory size for the CL-BFGS algorithm remains challenging. To deal with this issue, an adaptive method is proposed in which the memory size is allowed to vary during the iteration process. Basically, at each iteration, with the help of multistep quasi-Newton method, an appropriate memory size is chosen from a variable set {1,2, ... , M} by approximating complex Hessian matrix as close as possible. To reduce the computational complexity and ensure desired performance, the upper bound M is adjustable according to the moving average of memory sizes found in previous iterations. The proposed adaptive CL-BFGS (ACL-BFGS) algorithm can be efficiently applied for the training of CVNNs. Moreover, it is suggested to take multiple memory sizes to construct the search direction, which further improves the performance of the ACL-BFGS algorithm. Experimental results on some benchmark problems including the pattern classification, complex function approximation, and nonlinear channel equalization problems are given to illustrate the advantages of the developed algorithms over some previous ones.
复值有限内存BFGS（CL - BFGS）算法在训练复值神经网络（CVNNs）方面效率很高。作为一个重要参数，内存大小表示保存的向量对数量，并且会对算法性能产生实质性影响。然而，为CL - BFGS算法确定合适的内存大小仍然具有挑战性。为了解决这个问题，提出了一种自适应方法，其中内存大小在迭代过程中允许变化。基本上，在每次迭代时，借助多步拟牛顿法，通过尽可能逼近复海森矩阵，从变量集{1, 2, ... , M}中选择合适的内存大小。为了降低计算复杂度并确保期望的性能，上限M可根据先前迭代中找到的内存大小的移动平均值进行调整。所提出的自适应CL - BFGS（ACL - BFGS）算法可有效地应用于CVNNs的训练。此外，建议采用多个内存大小来构建搜索方向，这进一步提高了ACL - BFGS算法的性能。给出了一些基准问题（包括模式分类、复函数逼近和非线性信道均衡问题）的实验结果，以说明所开发算法相对于一些先前算法的优势。
