Maraghi Vali Ollah, Faez Karim
Department of Electrical Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran.
Comput Intell Neurosci. 2021 Jun 9;2021:9922697. doi: 10.1155/2021/9922697. eCollection 2021.
Recognition of human activities is an essential field in computer vision. The most human activity consists of the interaction between humans and objects. Many successful works have been done on human-object interaction (HOI) recognition and achieved acceptable results in recent years. Still, they are fully supervised and need to train labeled data for all HOIs. Due to the enormous space of human-object interactions, listing and providing the training data for all possible categories is costly and impractical. We propose an approach for scaling human-object interaction recognition in video data through the zero-shot learning technique to solve this problem. Our method recognizes a verb and an object from the video and makes an HOI class. Recognition of the verbs and objects instead of HOIs allows identifying a new combination of verbs and objects. So, a new HOI class can be identified, which is not seen by the recognizer system. We introduce a neural network architecture that can understand and represent the video data. The proposed system learns verbs and objects from available training data at the training phase and can identify the verb-object pairs in a video at test time. So, the system can identify the HOI class with different combinations of objects and verbs. Also, we propose to use lateral information for combining the verbs and the objects to make valid verb-object pairs. It helps to prevent the detection of rare and probably wrong HOIs. The lateral information comes from word embedding techniques. Furthermore, we propose a new feature aggregation method for aggregating extracted high-level features from video frames before feeding them to the classifier. We illustrate that this feature aggregation method is more effective for actions that include multiple subactions. We evaluated our system by recently introduced Charades challengeable dataset, which has lots of HOI categories in videos. We show that our proposed system can detect unseen HOI classes in addition to the acceptable recognition of seen types. Therefore, the number of classes identifiable by the system is greater than the number of classes used for training.
人类活动识别是计算机视觉中的一个重要领域。大多数人类活动都包含人与物体之间的交互。近年来，在人类-物体交互（HOI）识别方面已经取得了许多成功的成果，并取得了可接受的结果。然而，它们都是完全监督式的，需要为所有HOI训练标注数据。由于人类-物体交互的空间巨大，列出并为所有可能的类别提供训练数据既昂贵又不切实际。我们提出了一种通过零样本学习技术来扩展视频数据中人类-物体交互识别的方法，以解决这个问题。我们的方法从视频中识别动词和物体，并生成一个HOI类别。识别动词和物体而不是HOI可以识别动词和物体的新组合。因此，可以识别出识别系统未见过的新HOI类别。我们引入了一种能够理解和表示视频数据的神经网络架构。所提出的系统在训练阶段从可用的训练数据中学习动词和物体，并在测试时能够识别视频中的动词-物体对。因此，该系统可以用物体和动词的不同组合来识别HOI类别。此外，我们建议使用横向信息来组合动词和物体，以形成有效的动词-物体对。这有助于防止检测到罕见且可能错误的HOI。横向信息来自词嵌入技术。此外，我们提出了一种新的特征聚合方法，用于在将从视频帧中提取的高级特征输入分类器之前进行聚合。我们表明，这种特征聚合方法对于包含多个子动作的动作更有效。我们通过最近引入的具有挑战性的Charades数据集对我们的系统进行了评估，该数据集中的视频包含许多HOI类别。我们表明，我们提出的系统除了能够对已见过的类型进行可接受的识别外，还能检测到未见过的HOI类别。因此，系统可识别的类别数量大于用于训练的类别数量。
