Skolkovo Institute of Science and Technology, Skolkovo Innovation Center, Building 3, Moscow 143026, Russia; Institute for Information Transmission Problems, Bolshoy Karetny per. 19, Building 1, Moscow 127051, Russia.
Neural Netw. 2017 Oct;94:103-114. doi: 10.1016/j.neunet.2017.07.002. Epub 2017 Jul 13.
We study expressive power of shallow and deep neural networks with piece-wise linear activation functions. We establish new rigorous upper and lower bounds for the network complexity in the setting of approximations in Sobolev spaces. In particular, we prove that deep ReLU networks more efficiently approximate smooth functions than shallow networks. In the case of approximations of 1D Lipschitz functions we describe adaptive depth-6 network architectures more efficient than the standard shallow architecture.
我们研究了具有分段线性激活函数的浅层和深层神经网络的表达能力。我们在 Sobolev 空间逼近的背景下，为网络复杂度建立了新的严格的上下界。特别地，我们证明了深度 ReLU 网络比浅层网络更有效地逼近光滑函数。在逼近 1D Lipschitz 函数的情况下，我们描述了比标准浅层架构更有效的自适应深度-6 网络架构。
