Hu Haigen, Liu Aizhu, Guan Qiu, Qian Hanwang, Li Xiaoxin, Chen Shengyong, Zhou Qianwei
IEEE Trans Neural Netw Learn Syst. 2023 Sep;34(9):6096-6107. doi: 10.1109/TNNLS.2021.3133263. Epub 2023 Sep 1.
To enhance the nonlinearity of neural networks and increase their mapping abilities between the inputs and response variables, activation functions play a crucial role to model more complex relationships and patterns in the data. In this work, a novel methodology is proposed to adaptively customize activation functions only by adding very few parameters to the traditional activation functions such as Sigmoid, Tanh, and rectified linear unit (ReLU). To verify the effectiveness of the proposed methodology, some theoretical and experimental analysis on accelerating the convergence and improving the performance is presented, and a series of experiments are conducted based on various network models (such as AlexNet, VggNet, GoogLeNet, ResNet and DenseNet), and various datasets (such as CIFAR10, CIFAR100, miniImageNet, PASCAL VOC, and COCO). To further verify the validity and suitability in various optimization strategies and usage scenarios, some comparison experiments are also implemented among different optimization strategies (such as SGD, Momentum, AdaGrad, AdaDelta, and ADAM) and different recognition tasks such as classification and detection. The results show that the proposed methodology is very simple but with significant performance in convergence speed, precision, and generalization, and it can surpass other popular methods such as ReLU and adaptive functions such as Swish in almost all experiments in terms of overall performance.
为了增强神经网络的非线性并提高其在输入和响应变量之间的映射能力，激活函数对于在数据中建模更复杂的关系和模式起着至关重要的作用。在这项工作中，提出了一种新颖的方法，仅通过在诸如Sigmoid、Tanh和整流线性单元（ReLU）等传统激活函数上添加极少的参数来自适应地定制激活函数。为了验证所提方法的有效性，给出了一些关于加速收敛和提高性能的理论和实验分析，并基于各种网络模型（如AlexNet、VggNet、GoogLeNet、ResNet和DenseNet）以及各种数据集（如CIFAR10、CIFAR100、miniImageNet、PASCAL VOC和COCO）进行了一系列实验。为了进一步验证在各种优化策略和使用场景中的有效性和适用性，还在不同的优化策略（如SGD、Momentum、AdaGrad、AdaDelta和ADAM）以及不同的识别任务（如分类和检测）之间进行了一些比较实验。结果表明，所提方法非常简单，但在收敛速度、精度和泛化方面具有显著性能，并且在几乎所有实验的整体性能方面都能超越其他流行方法（如ReLU）和自适应函数（如Swish）。
