Jin Weike, Zhao Zhou, Cao Xiaochun, Zhu Jieming, He Xiuqiang, Zhuang Yueting
IEEE Trans Image Process. 2021;30:5477-5489. doi: 10.1109/TIP.2021.3076556. Epub 2021 Jun 11.
Vision-language research has become very popular, which focuses on understanding of visual contents, language semantics and relationships between them. Video question answering (Video QA) is one of the typical tasks. Recently, several BERT style pre-training methods have been proposed and shown effectiveness on various vision-language tasks. In this work, we leverage the successful vision-language transformer structure to solve the Video QA problem. However, we do not pre-train it with any video data, because video pre-training requires massive computing resources and is hard to perform with only a few GPUs. Instead, our work aims to leverage image-language pre-training to help with video-language modeling, by sharing a common module design. We further introduce an adaptive spatio-temporal graph to enhance the vision-language representation learning. That is, we adaptively refine the spatio-temporal tubes of salient objects according to their spatio-temporal relations learned through a hierarchical graph convolution process. Finally, we can obtain a number of fine-grained tube-level video object representations, as the visual inputs of the vision-language transformer module. Experiments on three widely used Video QA datasets show that our model achieves the new state-of-the-art results.
视觉语言研究变得非常流行，其专注于对视觉内容、语言语义及其之间关系的理解。视频问答（Video QA）是典型任务之一。最近，已经提出了几种BERT风格的预训练方法，并在各种视觉语言任务上显示出有效性。在这项工作中，我们利用成功的视觉语言Transformer结构来解决视频问答问题。然而，我们没有使用任何视频数据对其进行预训练，因为视频预训练需要大量计算资源，并且仅使用几个GPU很难进行。相反，我们的工作旨在通过共享通用模块设计，利用图像语言预训练来帮助进行视频语言建模。我们进一步引入了自适应时空图，以增强视觉语言表示学习。也就是说，我们根据通过分层图卷积过程学到的时空关系，自适应地细化显著对象的时空管。最后，我们可以获得许多细粒度的管级视频对象表示，作为视觉语言Transformer模块的视觉输入。在三个广泛使用的视频问答数据集上的实验表明，我们的模型取得了新的最优结果。
