Shang Yuying, Fu Kun, Zhang Zequn, Jin Li, Liu Zinan, Wang Shensi, Li Shuchao
Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China.
Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China.
Sensors (Basel). 2024 Nov 28;24(23):7605. doi: 10.3390/s24237605.
The multi-modal knowledge graph completion (MMKGC) task aims to automatically mine the missing factual knowledge from the existing multi-modal knowledge graphs (MMKGs), which is crucial in advancing cross-modal learning and reasoning. However, few methods consider the adverse effects caused by different missing modal information in the model learning process. To address the above challenges, we innovatively propose a odal quilibrium elational raph framwork, called . By constructing three modal-specific directed relational graph attention networks, MERGE can implicitly represent missing modal information for entities by aggregating the modal embeddings from neighboring nodes. Subsequently, a fusion approach based on low-rank tensor decomposition is adopted to align multiple modal features in both the explicit structural level and the implicit semantic level, utilizing the structural information inherent in the original knowledge graphs, which enhances the interpretability of the fused features. Furthermore, we introduce a novel interpolation re-ranking strategy to adjust the importance of modalities during inference while preserving the semantic integrity of each modality. The proposed framework has been validated on four publicly available datasets, and the experimental results have demonstrated the effectiveness and robustness of our method in the MMKGC task.
多模态知识图谱补全(MMKGC)任务旨在从现有的多模态知识图谱(MMKGs)中自动挖掘缺失的事实性知识,这对于推进跨模态学习和推理至关重要。然而,很少有方法考虑模型学习过程中不同缺失模态信息所带来的不利影响。为应对上述挑战,我们创新性地提出了一种模态均衡关系图框架,称为MERGE。通过构建三个特定模态的有向关系图注意力网络,MERGE可以通过聚合来自相邻节点的模态嵌入来隐式地表示实体的缺失模态信息。随后,采用基于低秩张量分解的融合方法,利用原始知识图谱中固有的结构信息,在显式结构层面和隐式语义层面上对齐多个模态特征,这增强了融合特征的可解释性。此外,我们引入了一种新颖的插值重排策略,在推理过程中调整模态的重要性,同时保持每个模态的语义完整性。所提出的框架已在四个公开可用的数据集上得到验证,实验结果证明了我们的方法在MMKGC任务中的有效性和鲁棒性。