Department of Computer Science, Kyonggi University, Suwon-si 16227, Korea.
Sensors (Basel). 2021 Apr 21;21(9):2911. doi: 10.3390/s21092911.
Visual commonsense reasoning is an intelligent task performed to decide the most appropriate answer to a question while providing the rationale or reason for the answer when an image, a natural language question, and candidate responses are given. For effective visual commonsense reasoning, both the knowledge acquisition problem and the multimodal alignment problem need to be solved. Therefore, we propose a novel Vision-Language-Knowledge Co-embedding (ViLaKC) model that extracts knowledge graphs relevant to the question from an external knowledge base, ConceptNet, and uses them together with the input image to answer the question. The proposed model uses a pretrained vision-language-knowledge embedding module, which co-embeds multimodal data including images, natural language texts, and knowledge graphs into a single feature vector. To reflect the structural information of the knowledge graph, the proposed model uses the graph convolutional neural network layer to embed the knowledge graph first and then uses multi-head self-attention layers to co-embed it with the image and natural language question. The effectiveness and performance of the proposed model are experimentally validated using the VCR v1.0 benchmark dataset.
视觉常识推理是一项智能任务，用于在提供图像、自然语言问题和候选答案的情况下，决定对问题的最合适回答，并为回答提供理由或原因。为了进行有效的视觉常识推理，需要解决知识获取问题和多模态对齐问题。因此，我们提出了一种新颖的视觉-语言-知识联合嵌入（ViLaKC）模型，该模型从外部知识库 ConceptNet 中提取与问题相关的知识图谱，并将其与输入图像一起用于回答问题。所提出的模型使用预训练的视觉-语言-知识嵌入模块，将包括图像、自然语言文本和知识图谱在内的多模态数据联合嵌入到单个特征向量中。为了反映知识图谱的结构信息，所提出的模型首先使用图卷积神经网络层嵌入知识图谱，然后使用多头自注意力层将其与图像和自然语言问题进行联合嵌入。使用 VCR v1.0 基准数据集对所提出的模型进行了实验验证，以验证其有效性和性能。
