Department of Mathematics, McMaster University, Canada; Vector Institute, Canada.
University of Oxford, UK.
Neural Netw. 2024 Oct;178:106420. doi: 10.1016/j.neunet.2024.106420. Epub 2024 Jun 4.
We study the representation capacity of deep hyperbolic neural networks (HNNs) with a ReLU activation function. We establish the first proof that HNNs can ɛ-isometrically embed any finite weighted tree into a hyperbolic space of dimension d at least equal to 2 with prescribed sectional curvature κ<0, for any ɛ>1 (where ɛ=1 being optimal). We establish rigorous upper bounds for the network complexity on an HNN implementing the embedding. We find that the network complexity of HNN implementing the graph representation is independent of the representation fidelity/distortion. We contrast this result against our lower bounds on distortion which any ReLU multi-layer perceptron (MLP) must exert when embedding a tree with L>2 leaves into a d-dimensional Euclidean space, which we show at least Ω(L); independently of the depth, width, and (possibly discontinuous) activation function defining the MLP.
我们研究了具有 ReLU 激活函数的深层双曲神经网络 (HNN) 的表示能力。我们首次证明，对于任何 ɛ>1（其中 ɛ=1 是最优的），HNN 可以在至少维度为 d（d>=2）的具有规定截面曲率 κ<0 的双曲空间中 ɛ-isometrically 嵌入任何有限加权树。我们为在实现嵌入的 HNN 上的网络复杂度建立了严格的上界。我们发现，实现图表示的 HNN 的网络复杂度与表示保真度/失真无关。我们将这一结果与我们对任何具有 L>2 个叶子的树嵌入到 d 维欧几里得空间时 ReLU 多层感知机 (MLP) 必须施加的失真的下界进行对比，我们证明了至少 Ω(L)；这与定义 MLP 的深度、宽度和（可能不连续的）激活函数无关。
