IEEE Trans Pattern Anal Mach Intell. 2020 May;42(5):1257-1271. doi: 10.1109/TPAMI.2019.2893215. Epub 2019 Jan 16.
Visual data such as images and videos contain a rich source of structured semantic labels as well as a wide range of interacting components. Visual content could be assigned with fine-grained labels describing major components, coarse-grained labels depicting high level abstractions, or a set of labels revealing attributes. Such categorization over different, interacting layers of labels evinces the potential for a graph-based encoding of label information. In this paper, we exploit this rich structure for performing graph-based inference in label space for a number of tasks: multi-label image and video classification and action detection in untrimmed videos. We consider the use of the Bidirectional Inference Neural Network (BINN) and Structured Inference Neural Network (SINN) for performing graph-based inference in label space and propose a Long Short-Term Memory (LSTM) based extension for exploiting activity progression on untrimmed videos. The methods were evaluated on (i) the Animal with Attributes (AwA), Scene Understanding (SUN) and NUS-WIDE datasets for multi-label image classification, (ii) the first two releases of the YouTube-8M large scale dataset for multi-label video classification, and (iii) the THUMOS'14 and MultiTHUMOS video datasets for action detection. Our results demonstrate the effectiveness of structured label inference in these challenging tasks, achieving significant improvements against baselines.
视觉数据（如图像和视频）包含丰富的结构化语义标签源以及广泛的交互组件。视觉内容可以分配精细标签来描述主要组件、粗粒度标签来描述高级抽象概念、或一组标签来揭示属性。这种在不同的、交互的标签层上的分类显示了标签信息的基于图的编码的潜力。在本文中，我们利用这种丰富的结构来执行标签空间中的基于图的推理，以完成多项任务：多标签图像和视频分类以及未修剪视频中的动作检测。我们考虑使用双向推理神经网络（BINN）和结构化推理神经网络（SINN）来执行标签空间中的基于图的推理，并提出了一种基于长短期记忆（LSTM）的扩展，用于利用未修剪视频中的活动进展。该方法在（i）具有属性的动物（AwA）、场景理解（SUN）和 NUS-WIDE 数据集上进行了多标签图像分类评估，（ii）YouTube-8M 大规模数据集的前两个版本进行了多标签视频分类评估，以及（iii）THUMOS'14 和 MultiTHUMOS 视频数据集进行了动作检测评估。我们的结果证明了在这些具有挑战性的任务中结构化标签推理的有效性，与基线相比取得了显著的改进。
