Information and Communication Engineering, Chosun University, Gwangju, 61452, South Korea.
Gwangju Alzheimer's Disease and Related Dementia Cohort Research Center, Gwangju, South Korea.
Sci Rep. 2022 Sep 2;12(1):14978. doi: 10.1038/s41598-022-19020-y.
Activation functions in the neural network are responsible for 'firing' the nodes in it. In a deep neural network they 'activate' the features to reduce feature redundancy and learn the complex pattern by adding non-linearity in the network to learn task-specific goals. In this paper, we propose a simple and interesting activation function based on the combination of scaled gamma correction and hyperbolic tangent function, which we call Scaled Gamma Tanh (SGT) activation. The proposed activation function is applied in two steps, first is the calculation of gamma version as y = f(x) = ax for x < 0 and y = f(x) = bx for x ≥ 0, second is obtaining the squashed value as z = tanh(y). The variables a and b are user-defined constant values whereas [Formula: see text] and [Formula: see text] are channel-based learnable parameters. We analyzed the behavior of the proposed SGT activation function against other popular activation functions like ReLU, Leaky-ReLU, and tanh along with their role to confront vanishing/exploding gradient problems. For this, we implemented the SGT activation functions in a 3D Convolutional neural network (CNN) for the classification of magnetic resonance imaging (MRIs). More importantly to support our proposed idea we have presented a thorough analysis via histogram of inputs and outputs in activation layers along with weights/bias plot and t-SNE (t-Distributed Stochastic Neighbor Embedding) projection of fully connected layer for the trained CNN models. Our results in MRI classification show SGT outperforms standard ReLU and tanh activation in all cases i.e., final validation accuracy, final validation loss, test accuracy, Cohen's kappa score, and Precision.
神经网络中的激活函数负责“触发”其中的节点。在深度神经网络中，它们通过在网络中添加非线性来“激活”特征，以减少特征冗余并学习复杂模式，从而实现特定于任务的目标。在本文中，我们提出了一种简单而有趣的激活函数，它基于缩放伽马校正和双曲正切函数的组合，我们称之为缩放伽马双曲正切（SGT）激活函数。所提出的激活函数分两步应用，第一步是计算伽马版本，即当 x < 0 时 y = f(x) = ax，当 x ≥ 0 时 y = f(x) = bx；第二步是获得挤压值 z = tanh(y)。变量 a 和 b 是用户定义的常数，而 [公式：见正文] 和 [公式：见正文] 是基于通道的可学习参数。我们分析了所提出的 SGT 激活函数与其他流行的激活函数（如 ReLU、Leaky-ReLU 和 tanh）的行为，以及它们在解决消失/爆炸梯度问题方面的作用。为此，我们在 3D 卷积神经网络（CNN）中实现了 SGT 激活函数，用于 MRI 的分类。更重要的是，为了支持我们的提议，我们通过激活层中的输入和输出直方图、权重/偏差图以及完全连接层的 t-SNE（t-Distributed Stochastic Neighbor Embedding）投影，对训练好的 CNN 模型进行了彻底的分析。我们在 MRI 分类中的结果表明，SGT 在所有情况下（即最终验证准确性、最终验证损失、测试准确性、科恩氏kappa 得分和精度）都优于标准的 ReLU 和 tanh 激活函数。
