Yan Rui, Xie Lingxi, Shu Xiangbo, Zhang Liyan, Tang Jinhui
IEEE Trans Pattern Anal Mach Intell. 2023 Aug;45(8):10317-10330. doi: 10.1109/TPAMI.2023.3261659. Epub 2023 Jun 30.
In order to enable the model to generalize to unseen "action-objects" (compositional action), previous methods encode multiple pieces of information (i.e., the appearance, position, and identity of visual instances) independently and concatenate them for classification. However, these methods ignore the potential supervisory role of instance information (i.e., position and identity) in the process of visual perception. To this end, we present a novel framework, namely Progressive Instance-aware Feature Learning (PIFL), to progressively extract, reason, and predict dynamic cues of moving instances from videos for compositional action recognition. Specifically, this framework extracts features from foreground instances that are likely to be relevant to human actions (Position-aware Appearance Feature Extraction in Section III-B1), performs identity-aware reasoning among instance-centric features with semantic-specific interactions (Identity-aware Feature Interaction in Section III-B2), and finally predicts instances' position from observed states to force the model into perceiving their movement (Semantic-aware Position Prediction in Section III-B3). We evaluate our approach on two compositional action recognition benchmarks, namely, Something-Else and IKEA-Assembly. Our approach achieves consistent accuracy gain beyond off-the-shelf action recognition algorithms in terms of both ground truth and detected position of instances.
为了使模型能够推广到未见过的“动作-物体”（组合动作），先前的方法独立地对多条信息（即视觉实例的外观、位置和身份）进行编码，并将它们连接起来用于分类。然而，这些方法忽略了实例信息（即位置和身份）在视觉感知过程中的潜在监督作用。为此，我们提出了一种新颖的框架，即渐进式实例感知特征学习（PIFL），用于从视频中逐步提取、推理和预测移动实例的动态线索，以进行组合动作识别。具体而言，该框架从可能与人类动作相关的前景实例中提取特征（第三节B1中的位置感知外观特征提取），通过语义特定的交互在以实例为中心的特征之间进行身份感知推理（第三节B2中的身份感知特征交互），最后根据观察到的状态预测实例的位置，以迫使模型感知它们的运动（第三节B3中的语义感知位置预测）。我们在两个组合动作识别基准上评估了我们的方法，即Something-Else和宜家组装。在实例的真实位置和检测位置方面，我们的方法在现成的动作识别算法之上都实现了一致的准确率提升。
