Ohn Ilsang, Kim Yongdai
Department of Statistics, Seoul National University, Seoul 08826, Korea.
Entropy (Basel). 2019 Jun 26;21(7):627. doi: 10.3390/e21070627.
There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functions. This class of activation functions includes most of frequently used activation functions. We derive the required depth, width and sparsity of a deep neural network to approximate any Hölder smooth function upto a given approximation error for the large class of activation functions. Based on our approximation error analysis, we derive the minimax optimality of the deep neural network estimators with the general activation functions in both regression and classification problems.
人们对深度神经网络的表现力越来越感兴趣。然而，关于这个主题的现有工作大多只关注特定的激活函数，如ReLU或sigmoid。在本文中，我们研究了具有广泛激活函数类别的深度神经网络的逼近能力。这类激活函数包括了大多数常用的激活函数。我们推导了深度神经网络所需的深度、宽度和稀疏性，以便在给定的逼近误差范围内，对一大类激活函数逼近任何赫尔德平滑函数。基于我们的逼近误差分析，我们推导了在回归和分类问题中具有一般激活函数的深度神经网络估计器的极小极大最优性。
