Zhou Jing, Liu Zhiqiang, Hu Siying, Li Xiaoxue, Wang Zhiguang, Lu Qiang
China University of Petroleum (Beijing), College of Artificial Intelligence, Beijing, 102249, China.
China University of Petroleum (Beijing), Beijing Key Laboratory of Petroleum Data Mining, Beijing, 102249, China.
Sci Rep. 2024 Oct 24;14(1):25195. doi: 10.1038/s41598-024-75823-1.
Grounded Situation Recognition (GSR) aims to generate structured image descriptions. For a given image, GSR needs to identify the key verb, the nouns corresponding to roles, and their bounding-box groundings. However, current GSR research demands numerous meticulously labeled images, which are labor-intensive and time-consuming, making it costly to expand detection categories. Our study enhances model accuracy in detecting and localizing under data scarcity, reducing dependency on large datasets and paving the way for broader detection capabilities. In this paper, we propose the Grounded Situation Recognition under Data Scarcity (GSRDS) model, which uses the CoFormer model as the baseline and optimizes three subtasks: image feature extraction, verb classification, and bounding-box localization, to better adapt to data-scarce scenarios. Specifically, we replace ResNet50 with EfficientNetV2-M for advanced image feature extraction. Additionally, we introduce the Transformer Combined with CLIP for Verb Classification (TCCV) module, utilizing features extracted by CLIP's image encoder to enhance verb classification accuracy. Furthermore, we design the Multi-source Verb-Role Queries (Multi-VR Queries) and the Dual Parallel Decoders (DPD) modules to improve the accuracy of bounding-box localization. Through extensive comparative experiments and ablation studies, we demonstrate that our method achieves higher accuracy than mainstream approaches in data-scarce scenarios. Our code will be available at https://github.com/Zhou-maker-oss/GSRDS .
基于场景的目标识别（GSR）旨在生成结构化的图像描述。对于给定的图像，GSR需要识别关键动词、与角色对应的名词及其边界框定位。然而，当前的GSR研究需要大量精心标注的图像，这既耗费人力又耗时，使得扩展检测类别成本高昂。我们的研究提高了在数据稀缺情况下检测和定位的模型准确性，减少了对大型数据集的依赖，并为更广泛的检测能力铺平了道路。在本文中，我们提出了数据稀缺情况下的基于场景的目标识别（GSRDS）模型，该模型以CoFormer模型为基线，并优化了三个子任务：图像特征提取、动词分类和边界框定位，以更好地适应数据稀缺场景。具体来说，我们用EfficientNetV2-M替换ResNet50以进行先进的图像特征提取。此外，我们引入了结合CLIP的Transformer进行动词分类（TCCV）模块，利用CLIP图像编码器提取的特征来提高动词分类的准确性。此外，我们设计了多源动词-角色查询（Multi-VR Queries）和双并行解码器（DPD）模块来提高边界框定位的准确性。通过广泛的对比实验和消融研究，我们证明了我们的方法在数据稀缺场景中比主流方法具有更高的准确性。我们的代码将在https://github.com/Zhou-maker-oss/GSRDS上提供。
