Zhao Heng, Zhou Joey Tianyi, Ong Yew-Soon
IEEE Trans Neural Netw Learn Syst. 2024 Feb;35(2):1523-1533. doi: 10.1109/TNNLS.2022.3183827. Epub 2024 Feb 5.
Current one-stage methods for visual grounding encode the language query as one holistic sentence embedding before fusion with visual features for target localization. Such a formulation provides insufficient ability to model query at the word level, and therefore is prone to neglect words that may not be the most important ones for a sentence but are critical for the referred object. In this article, we propose Word2Pix: a one-stage visual grounding network based on the encoder-decoder transformer architecture that enables learning for textual to visual feature correspondence via word to pixel attention. Each word from the query sentence is given an equal opportunity when attending to visual pixels through multiple stacks of transformer decoder layers. In this way, the decoder can learn to model the language query and fuse language with the visual features for target prediction simultaneously. We conduct the experiments on RefCOCO, RefCOCO+, and RefCOCOg datasets, and the proposed Word2Pix outperforms the existing one-stage methods by a notable margin. The results obtained also show that Word2Pix surpasses the two-stage visual grounding models, while at the same time keeping the merits of the one-stage paradigm, namely, end-to-end training and fast inference speed. Code is available at https://github.com/azurerain7/Word2Pix.
当前的单阶段视觉定位方法在将语言查询与视觉特征融合以进行目标定位之前，会将语言查询编码为一个整体的句子嵌入。这种形式在词级对查询进行建模的能力不足，因此容易忽略那些对于一个句子来说可能不是最重要，但对于所指对象至关重要的词。在本文中，我们提出了Word2Pix：一种基于编码器 - 解码器变压器架构的单阶段视觉定位网络，它能够通过词到像素的注意力学习文本到视觉特征的对应关系。在通过多堆叠的变压器解码器层关注视觉像素时，查询句子中的每个词都有平等的机会。通过这种方式，解码器可以学习对语言查询进行建模，并同时将语言与视觉特征融合以进行目标预测。我们在RefCOCO、RefCOCO + 和RefCOCOg数据集上进行了实验，所提出的Word2Pix显著优于现有的单阶段方法。获得的结果还表明，Word2Pix超越了两阶段视觉定位模型，同时保留了单阶段范式的优点，即端到端训练和快速推理速度。代码可在https://github.com/azurerain7/Word2Pix获取。
