Ma Shuailei, Wang Yuefeng, Wang Shanze, Wei Ying
IEEE Trans Pattern Anal Mach Intell. 2024 Apr;46(4):2415-2429. doi: 10.1109/TPAMI.2023.3331738. Epub 2024 Mar 6.
Human-Object Interaction (HOI), as an important problem in computer vision, requires locating the human-object pair and identifying the interactive relationships between them. The HOI instance has a greater span in spatial, scale, and task than the individual object instance, making its detection more susceptible to noisy backgrounds. To alleviate the disturbance of noisy backgrounds on HOI detection, it is necessary to consider the input image information to generate fine-grained anchors which are then leveraged to guide the detection of HOI instances. However, it has the following challenges. i) how to extract pivotal features from the images with complex background information is still an open question. ii) how to semantically align the extracted features and query embeddings is also a difficult issue. In this paper, a novel end-to-end transformer-based framework (FGAHOI) is proposed to alleviate the above problems. FGAHOI comprises three dedicated components namely, multi-scale sampling (MSS), hierarchical spatial-aware merging (HSAM) and task-aware merging mechanism (TAM). MSS extracts features of humans, objects and interaction areas from noisy backgrounds for HOI instances of various scales. HSAM and TAM semantically align and merge the extracted features and query embeddings in the hierarchical spatial and task perspectives in turn. In the meanwhile, a novel training strategy Stage-wise Training Strategy is designed to reduce the training pressure caused by overly complex tasks done by FGAHOI. In addition, we propose two ways to measure the difficulty of HOI detection and a novel dataset, i.e., HOI-SDC for the two challenges (Uneven Distributed Area in Human-Object Pairs and Long Distance Visual Modeling of Human-Object Pairs) of HOI instances detection. Experiments are conducted on three benchmarks: HICO-DET, HOI-SDC and V-COCO. Our model outperforms the state-of-the-art HOI detection methods, and the extensive ablations reveal the merits of our proposed contribution.
人机交互（HOI）作为计算机视觉中的一个重要问题，需要定位人机对并识别它们之间的交互关系。与单个物体实例相比，HOI实例在空间、尺度和任务方面具有更大的跨度，这使得其检测更容易受到嘈杂背景的影响。为了减轻嘈杂背景对HOI检测的干扰，有必要考虑输入图像信息以生成细粒度锚点，然后利用这些锚点来指导HOI实例的检测。然而，这存在以下挑战。i） 如何从具有复杂背景信息的图像中提取关键特征仍然是一个悬而未决的问题。ii） 如何在语义上对齐提取的特征和查询嵌入也是一个难题。本文提出了一种基于端到端变压器的新型框架（FGAHOI）来缓解上述问题。FGAHOI由三个专用组件组成，即多尺度采样（MSS）、分层空间感知合并（HSAM）和任务感知合并机制（TAM）。MSS从嘈杂背景中提取各种尺度的HOI实例的人类、物体和交互区域的特征。HSAM和TAM依次从分层空间和任务角度在语义上对齐并合并提取的特征和查询嵌入。同时，设计了一种新颖的训练策略——阶段式训练策略，以减轻FGAHOI执行的过于复杂的任务所带来的训练压力。此外，我们提出了两种方法来衡量HOI检测的难度，并提出了一个新颖的数据集，即用于HOI实例检测的两个挑战（人机对中不均匀分布区域和人机对的长距离视觉建模）的HOI-SDC。在三个基准上进行了实验：HICO-DET、HOI-SDC和V-COCO。我们的模型优于现有的HOI检测方法，广泛的消融实验揭示了我们所提出贡献的优点。
