Laboratory of Computer Science and Mathematics and their Applications (LIMA), Faculty of Science, University Chouaib Doukkali, El Jadida 24000, Morocco.
Laboratory of Intelligent Systems, Georesources and Renewable Energies (SIGER), University Sidi Mohamed Ben Abdellah, Fez, Morocco.
Neural Netw. 2022 Jun;150:149-166. doi: 10.1016/j.neunet.2022.03.008. Epub 2022 Mar 10.
Graph Neural Networks (GNNs) are powerful architectures for learning on graphs. They are efficient for predicting nodes, links and graphs properties. Standard GNN variants follow a message passing schema to update nodes representations using information from higher-order neighborhoods iteratively. Consequently, deeper GNNs make it possible to define high-level nodes representations generated based on local as well as distant neighborhoods. However, deeper networks are prone to suffer from over-smoothing. To build deeper GNN architectures and avoid losing the dependency between lower (the layers closer to the input) and higher (the layers closer to the output) layers, networks can integrate residual connections to connect intermediate layers. We propose the Augmented Graph Neural Network (AGNN) model with hierarchical global-based residual connections. Using the proposed residual connections, the model generates high-level nodes representations without the need for a deeper architecture. We disclose that the nodes representations generated through our proposed AGNN model are able to define an expressive all-encompassing representation of the entire graph. As such, the graph predictions generated through the AGNN model surpass considerably state-of-the-art results. Moreover, we carry out extensive experiments to identify the best global pooling strategy and attention weights to define the adequate hierarchical and global-based residual connections for different graph property prediction tasks. Furthermore, we propose a reversible variant of the AGNN model to address the extensive memory consumption problem that typically arises from training networks on large and dense graph datasets. The proposed Reversible Augmented Graph Neural Network (R-AGNN) only stores the nodes representations acquired from the output layer as opposed to saving all representations from intermediate layers as it is conventionally done when optimizing the parameters of other GNNs. We further refine the definition of the backpropagation algorithm to fit the R-AGNN model. We evaluate the proposed models AGNN and R-AGNN on benchmark Molecular, Bioinformatics and Social Networks datasets for graph classification and achieve state-of-the-art results. For instance the AGNN model realizes improvements of +39% on IMDB-MULTI reaching 91.7% accuracy and +16% on COLLAB reaching 96.8% accuracy compared to other GNN variants.
图神经网络（GNN）是用于图上学习的强大架构。它们在预测节点、链接和图属性方面非常高效。标准的 GNN 变体遵循消息传递模式，使用高阶邻域的信息迭代更新节点表示。因此，更深的 GNN 可以定义基于局部和远程邻域的高级节点表示。然而，更深的网络更容易受到过度平滑的影响。为了构建更深的 GNN 架构并避免丢失较低（靠近输入的层）和较高（靠近输出的层）层之间的依赖性，网络可以集成残差连接来连接中间层。我们提出了具有分层全局残差连接的增强图神经网络（AGNN）模型。使用提出的残差连接，模型无需更深的架构即可生成高级节点表示。我们揭示了通过我们提出的 AGNN 模型生成的节点表示能够定义整个图的富有表现力的全面表示。因此，通过 AGNN 模型生成的图预测大大超过了最先进的结果。此外，我们进行了广泛的实验，以确定最佳的全局池化策略和注意力权重，以定义不同图属性预测任务的适当分层和全局残差连接。此外，我们提出了 AGNN 模型的可逆变体，以解决在大型密集图数据集上训练网络时通常出现的大量内存消耗问题。所提出的可逆增强图神经网络（R-AGNN）仅存储从输出层获得的节点表示，而不是像传统方法那样存储中间层的所有表示，因为这是优化其他 GNN 参数时的传统做法。我们进一步细化了反向传播算法的定义，以适应 R-AGNN 模型。我们在基准分子、生物信息学和社交网络数据集上评估了所提出的模型 AGNN 和 R-AGNN 进行图分类，并实现了最先进的结果。例如，与其他 GNN 变体相比，AGNN 模型在 IMDB-MULTI 上实现了 +39%的改进，达到了 91.7%的准确率，在 COLLAB 上实现了 +16%的改进，达到了 96.8%的准确率。
