Zhang Xi, Zhang Feifei, Xu Changsheng
IEEE Trans Image Process. 2023;32:4621-4634. doi: 10.1109/TIP.2023.3302162. Epub 2023 Aug 16.
Multiple-choice visual question answering (VQA) is a challenging task due to the requirement of thorough multimodal understanding and complicated inter-modality relationship reasoning. To solve the challenge, previous approaches usually resort to different multimodal interaction modules. Despite their effectiveness, we find that existing methods may exploit a new discovered bias (vision-answer bias) to make answer prediction, leading to suboptimal VQA performances and poor generalization. To solve the issues, we propose a Causality-based Multimodal Interaction Enhancement (CMIE) method, which is model-agnostic and can be seamlessly incorporated into a wide range of VQA approaches in a plug-and-play manner. Specifically, our CMIE contains two key components: a causal intervention module and a counterfactual interaction learning module. The former devotes to removing the spurious correlation between the visual content and the answer caused by the vision-answer bias, and the latter helps capture discriminative inter-modality relationships by directly supervising multimodal interaction training via an interactive loss. Extensive experimental results on three public benchmarks and one reorganized dataset show that the proposed method can significantly improve seven representative VQA models, demonstrating the effectiveness and generalizability of the CMIE.
多项选择视觉问答（VQA）是一项具有挑战性的任务，因为它需要全面的多模态理解和复杂的跨模态关系推理。为了解决这一挑战，以往的方法通常采用不同的多模态交互模块。尽管这些方法很有效，但我们发现现有方法可能利用一种新发现的偏差（视觉-答案偏差）来进行答案预测，从而导致次优的VQA性能和较差的泛化能力。为了解决这些问题，我们提出了一种基于因果关系的多模态交互增强（CMIE）方法，该方法与模型无关，可以以即插即用的方式无缝集成到各种VQA方法中。具体来说，我们的CMIE包含两个关键组件：一个因果干预模块和一个反事实交互学习模块。前者致力于消除由视觉-答案偏差导致的视觉内容与答案之间的虚假相关性，后者通过交互式损失直接监督多模态交互训练，帮助捕捉有区别的跨模态关系。在三个公共基准和一个重组数据集上的大量实验结果表明，所提出的方法可以显著改进七个有代表性的VQA模型，证明了CMIE的有效性和泛化能力。
