Mu Guangyu, Chen Chuanzhi, Li Xiurong, Chen Ying, Dai Jiaxiu, Li Jiaxue
School of Management Science and Information Engineering, Jilin University of Finance and Economics, Changchun, China.
Key Laboratory of Financial Technology of Jilin Province, Changchun, China.
PLoS One. 2025 May 7;20(5):e0322556. doi: 10.1371/journal.pone.0322556. eCollection 2025.
The widespread disinformation on social media platforms has created significant challenges in verifying the authenticity of content, especially in multimodal contexts. However, simple modality fusion can introduce much noise due to the differences in feature representations among various modalities, ultimately impacting the accuracy of detection results. Thus, this paper proposes the Contrastive Learning and Adaptive Agg-modality Fusion (CLAAF) model for multimodal fake information detection. Firstly, a contrastive learning strategy is designed to align text and image modalities, preserving essential features while minimizing redundant noise. Secondly, an adaptive agg-modality fusion module is proposed to facilitate deep interaction and integration between modalities, enhancing the model's capability to process complex multimodal information. Finally, a comprehensive multimodal dataset is constructed through web crawling from authoritative news sources and multiple fact-checking platforms, establishing a solid foundation for training and validating the model. The experimental results demonstrate that the CLAAF model achieves a 3.45% improvement in accuracy compared to the best-performing baseline models, observably advancing the precision and robustness of multimodal fake information detection.
社交媒体平台上广泛传播的虚假信息给内容真实性的验证带来了重大挑战，尤其是在多模态语境中。然而，由于各种模态之间特征表示的差异，简单的模态融合会引入大量噪声，最终影响检测结果的准确性。因此，本文提出了用于多模态虚假信息检测的对比学习与自适应聚合模态融合（CLAAF）模型。首先，设计了一种对比学习策略来对齐文本和图像模态，在保留基本特征的同时最小化冗余噪声。其次，提出了一个自适应聚合模态融合模块，以促进模态之间的深度交互和整合，增强模型处理复杂多模态信息的能力。最后，通过从权威新闻来源和多个事实核查平台进行网络爬取，构建了一个全面的多模态数据集，为模型的训练和验证奠定了坚实基础。实验结果表明，与表现最佳的基线模型相比，CLAAF模型的准确率提高了3.45%，显著提升了多模态虚假信息检测的精度和鲁棒性。
