Kaveh Mehrdad, Mesgari Mohammad Saadi
Department of Geodesy and Geomatics, K. N. Toosi University of Technology, Tehran, 19967-15433 Iran.
Neural Process Lett. 2022 Oct 31:1-104. doi: 10.1007/s11063-022-11055-6.
The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.
人工神经网络（ANNs）和深度学习（DL）架构的学习过程及超参数优化被认为是最具挑战性的机器学习问题之一。过去的一些研究使用基于梯度的反向传播方法来训练DL架构。然而，基于梯度的方法存在主要缺点，例如在多目标成本函数中陷入局部最小值、由于数千次迭代计算梯度信息而导致执行时间昂贵，以及需要成本函数连续。由于训练ANNs和DLs是一个NP难优化问题，使用元启发式（MH）算法对其结构和参数进行优化受到了广泛关注。MH算法可以准确地制定DL组件（如超参数、权重、层数、神经元数量、学习率等）的最优估计。本文对使用MH算法优化ANNs和DLs进行了全面综述。在本文中，我们回顾了在DL和ANN方法中使用MH算法的最新进展，介绍了它们的缺点和优点，并指出了一些研究方向以填补MHs和DL方法之间的差距。此外，还解释了进化混合架构在文献中的适用性仍然有限。同时，本文对文献中的最新MH算法进行了分类，以展示它们在DL和ANN训练中对各种应用的有效性。大多数研究人员倾向于通过组合MHs来扩展新颖的混合算法，以优化DLs和ANNs的超参数。混合MHs的发展有助于提高算法性能，并能够解决复杂的优化问题。一般来说，MHs的最优性能应该能够在探索和利用特征之间实现适当的权衡。因此，本文试图从收敛趋势、探索、利用以及避免局部最小值的能力等方面总结各种MH算法。预计在未来几年，MH与DLs的集成将加速训练过程。然而，以这种方式的相关出版物仍然很少。
