Wang Boyue, Ju Xiaoqian, Gao Junbin, Li Xiaoyan, Hu Yongli, Yin Baocai
IEEE Trans Neural Netw Learn Syst. 2025 Sep;36(9):16366-16378. doi: 10.1109/TNNLS.2025.3562085.
Visual question answering (VQA) models often face two language bias challenges. First, they tend to rely solely on the question to predict the answer, often overlooking relevant information in the accompanying images. Second, even when considering the question, they may focus only on the wh-words, neglecting other crucial keywords that could enhance interpretability and the question sensitivity. Existing debiasing methods attempt to address this by training a bias model using question-only inputs to enhance the robustness of the target VQA model. However, this approach may not fully capture the language bias present. In this article, we propose a multimodality counterfactual dual-bias model to mitigate the linguistic bias issue in target VQA models. Our approach involves designing a shared-parameterized dual-bias model that incorporates both visual and question counterfactual samples as inputs. By doing so, we aim to fully model language biases, with visual and question counterfactual samples, respectively, emphasizing important objects and keywords to relevant the answers. To ensure that our dual-bias model behaves similarly to an ordinary model, we freeze the parameters of the target VQA model, meanwhile using the cross-entropy and Kullback-Leibler (KL) divergence as the loss function to train the dual-bias model. Subsequently, to mitigate language bias in the target VQA model, we freeze the parameters of the dual-bias model to generate pseudo-labels and then incorporate a margin loss to re-train the target VQA model. Experimental results on the VQA-CP datasets demonstrate the superior effectiveness of our proposed counterfactual dual-bias model. Additionally, we conduct an analysis of the unsatisfactory performance on the VQA v2 dataset. The origin code of the proposed model is available at https://github.com/Arrow2022jv/MCD.
视觉问答（VQA）模型常常面临两种语言偏差挑战。首先，它们往往仅依赖问题来预测答案，常常忽略了附带图像中的相关信息。其次，即使考虑问题时，它们可能只关注疑问词，而忽略其他可能增强可解释性和问题敏感性的关键关键词。现有的去偏方法试图通过使用仅问题输入来训练偏差模型，以增强目标VQA模型的鲁棒性来解决这个问题。然而，这种方法可能无法完全捕捉到存在的语言偏差。在本文中，我们提出了一种多模态反事实双偏差模型，以减轻目标VQA模型中的语言偏差问题。我们的方法包括设计一个共享参数化的双偏差模型，该模型将视觉和问题反事实样本都作为输入。通过这样做，我们旨在分别利用视觉和问题反事实样本充分建模语言偏差，强调与答案相关的重要对象和关键词。为确保我们的双偏差模型表现得与普通模型类似，我们冻结目标VQA模型的参数，同时使用交叉熵和库尔贝克-莱布勒（KL）散度作为损失函数来训练双偏差模型。随后，为减轻目标VQA模型中的语言偏差，我们冻结双偏差模型的参数以生成伪标签，然后纳入边际损失来重新训练目标VQA模型。在VQA-CP数据集上的实验结果证明了我们提出的反事实双偏差模型的卓越有效性。此外，我们对VQA v2数据集上不尽人意的性能进行了分析。所提出模型的原始代码可在https://github.com/Arrow2022jv/MCD获取。
