School of Electrical and Computer Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan, Republic of Korea.
Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea.
Neural Netw. 2021 Feb;134:95-106. doi: 10.1016/j.neunet.2020.10.017. Epub 2020 Nov 10.
In this study, we present a neural network that consists of nodes with heterogeneous sensitivity. Each node in a network is assigned a variable that determines the sensitivity with which it learns to perform a given task. The network is trained via a constrained optimization that maximizes the sparsity of the sensitivity variables while ensuring optimal network performance. As a result, the network learns to perform a given task using only a few sensitive nodes. Insensitive nodes, which are nodes with zero sensitivity, can be removed from a trained network to obtain a computationally efficient network. Removing zero-sensitivity nodes has no effect on the performance of the network because the network has already been trained to perform the task without them. The regularization parameter used to solve the optimization problem was simultaneously found during the training of the networks. To validate our approach, we designed networks with computationally efficient architectures for various tasks such as autoregression, object recognition, facial expression recognition, and object detection using various datasets. In our experiments, the networks designed by our proposed method provided the same or higher performances but with far less computational complexity.
在这项研究中，我们提出了一种由具有异质敏感性的节点组成的神经网络。网络中的每个节点都被分配了一个变量，该变量决定了它学习执行给定任务的敏感性。通过最大化敏感性变量的稀疏性同时确保网络性能最优的约束优化来训练网络。结果，网络学会了仅使用少数敏感节点执行给定任务。不敏感节点（即具有零敏感性的节点）可以从训练后的网络中删除，以获得计算效率更高的网络。由于网络已经经过训练可以在没有它们的情况下执行任务，因此删除零敏感性节点对网络的性能没有影响。用于解决优化问题的正则化参数是在网络训练过程中同时找到的。为了验证我们的方法，我们针对各种任务（如自回归、目标识别、面部表情识别和对象检测）设计了具有计算效率架构的网络，并使用各种数据集进行了实验。在我们的实验中，我们提出的方法设计的网络提供了相同或更高的性能，但计算复杂度要低得多。
