Suppr超能文献

MERGE:一种用于多模态知识图谱补全的模态平衡关系图框架。

MERGE: A Modal Equilibrium Relational Graph Framework for Multi-Modal Knowledge Graph Completion.

作者信息

Shang Yuying, Fu Kun, Zhang Zequn, Jin Li, Liu Zinan, Wang Shensi, Li Shuchao

机构信息

Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China.

Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China.

出版信息

Sensors (Basel). 2024 Nov 28;24(23):7605. doi: 10.3390/s24237605.

Abstract

The multi-modal knowledge graph completion (MMKGC) task aims to automatically mine the missing factual knowledge from the existing multi-modal knowledge graphs (MMKGs), which is crucial in advancing cross-modal learning and reasoning. However, few methods consider the adverse effects caused by different missing modal information in the model learning process. To address the above challenges, we innovatively propose a odal quilibrium elational raph framwork, called . By constructing three modal-specific directed relational graph attention networks, MERGE can implicitly represent missing modal information for entities by aggregating the modal embeddings from neighboring nodes. Subsequently, a fusion approach based on low-rank tensor decomposition is adopted to align multiple modal features in both the explicit structural level and the implicit semantic level, utilizing the structural information inherent in the original knowledge graphs, which enhances the interpretability of the fused features. Furthermore, we introduce a novel interpolation re-ranking strategy to adjust the importance of modalities during inference while preserving the semantic integrity of each modality. The proposed framework has been validated on four publicly available datasets, and the experimental results have demonstrated the effectiveness and robustness of our method in the MMKGC task.

摘要

多模态知识图谱补全(MMKGC)任务旨在从现有的多模态知识图谱(MMKGs)中自动挖掘缺失的事实性知识,这对于推进跨模态学习和推理至关重要。然而,很少有方法考虑模型学习过程中不同缺失模态信息所带来的不利影响。为应对上述挑战,我们创新性地提出了一种模态均衡关系图框架,称为MERGE。通过构建三个特定模态的有向关系图注意力网络,MERGE可以通过聚合来自相邻节点的模态嵌入来隐式地表示实体的缺失模态信息。随后,采用基于低秩张量分解的融合方法,利用原始知识图谱中固有的结构信息,在显式结构层面和隐式语义层面上对齐多个模态特征,这增强了融合特征的可解释性。此外,我们引入了一种新颖的插值重排策略,在推理过程中调整模态的重要性,同时保持每个模态的语义完整性。所提出的框架已在四个公开可用的数据集上得到验证,实验结果证明了我们的方法在MMKGC任务中的有效性和鲁棒性。

https://cdn.ncbi.nlm.nih.gov/pmc/blobs/c252/11644511/909e3999c41c/sensors-24-07605-g001.jpg

文献检索

告别复杂PubMed语法,用中文像聊天一样搜索,搜遍4000万医学文献。AI智能推荐,让科研检索更轻松。

立即免费搜索

文件翻译

保留排版,准确专业,支持PDF/Word/PPT等文件格式,支持 12+语言互译。

免费翻译文档

深度研究

AI帮你快速写综述,25分钟生成高质量综述,智能提取关键信息,辅助科研写作。

立即免费体验