Meng Shixuan, Jiang Rongxin, Tian Xiang, Zhou Fan, Chen Yaowu, Liu Junjie, Shen Chen
Zhejiang University, Hangzhou, P. R. China.
Zhejiang University, Embedded System Engineering Research Center, Ministry of Education of China, Hangzhou, P. R. China.
Int J Neural Syst. 2025 Mar;35(3):2550010. doi: 10.1142/S0129065725500108. Epub 2025 Jan 23.
Multi-label zero-shot learning (ML-ZSL) strives to recognize all objects in an image, regardless of whether they are present in the training data. Recent methods incorporate an attention mechanism to locate labels in the image and generate class-specific semantic information. However, the attention mechanism built on visual features treats label embeddings equally in the prediction score, leading to severe semantic ambiguity. This study focuses on efficiently utilizing semantic information in the attention mechanism. We propose a contrastive label-based attention method (CLA) to associate each label with the most relevant image regions. Specifically, our label-based attention, guided by the latent label embedding, captures discriminative image details. To distinguish region-wise correlations, we implement a region-level contrastive loss. In addition, we utilize a global feature alignment module to identify labels with general information. Extensive experiments on two benchmarks, NUS-WIDE and Open Images, demonstrate that our CLA outperforms the state-of-the-art methods. Especially under the ZSL setting, our method achieves 2.0% improvements in mean Average Precision (mAP) for NUS-WIDE and 4.0% for Open Images compared with recent methods.
多标签零样本学习（ML-ZSL）致力于识别图像中的所有物体，无论它们是否出现在训练数据中。最近的方法引入了注意力机制来定位图像中的标签并生成特定类别的语义信息。然而，基于视觉特征构建的注意力机制在预测分数中对标签嵌入一视同仁，导致严重的语义模糊。本研究专注于在注意力机制中有效利用语义信息。我们提出了一种基于对比标签的注意力方法（CLA），将每个标签与最相关的图像区域相关联。具体而言，我们基于标签的注意力在潜在标签嵌入的引导下，捕捉有区分力的图像细节。为了区分区域级别的相关性，我们实现了一种区域级对比损失。此外，我们利用一个全局特征对齐模块来识别具有一般信息的标签。在NUS-WIDE和Open Images这两个基准上进行的大量实验表明，我们的CLA优于当前的先进方法。特别是在零样本学习设置下，与最近的方法相比，我们的方法在NUS-WIDE上的平均精度均值（mAP）提高了2.0%，在Open Images上提高了4.0%。
