Brain and Behavior Discovery Institute, Medical College of Georgia, Georgia Regents University, Augusta, Georgia, United States of America.
PLoS One. 2012;7(10):e46686. doi: 10.1371/journal.pone.0046686. Epub 2012 Oct 4.
Human and many other animals can detect, recognize, and classify natural actions in a very short time. How this is achieved by the visual system and how to make machines understand natural actions have been the focus of neurobiological studies and computational modeling in the last several decades. A key issue is what spatial-temporal features should be encoded and what the characteristics of their occurrences are in natural actions. Current global encoding schemes depend heavily on segmenting while local encoding schemes lack descriptive power. Here, we propose natural action structures, i.e., multi-size, multi-scale, spatial-temporal concatenations of local features, as the basic features for representing natural actions. In this concept, any action is a spatial-temporal concatenation of a set of natural action structures, which convey a full range of information about natural actions. We took several steps to extract these structures. First, we sampled a large number of sequences of patches at multiple spatial-temporal scales. Second, we performed independent component analysis on the patch sequences and classified the independent components into clusters. Finally, we compiled a large set of natural action structures, with each corresponding to a unique combination of the clusters at the selected spatial-temporal scales. To classify human actions, we used a set of informative natural action structures as inputs to two widely used models. We found that the natural action structures obtained here achieved a significantly better recognition performance than low-level features and that the performance was better than or comparable to the best current models. We also found that the classification performance with natural action structures as features was slightly affected by changes of scale and artificially added noise. We concluded that the natural action structures proposed here can be used as the basic encoding units of actions and may hold the key to natural action understanding.
人类和许多其他动物能够在很短的时间内检测、识别和分类自然动作。视觉系统如何实现这一点，以及如何使机器理解自然动作，一直是过去几十年神经生物学研究和计算建模的焦点。一个关键问题是应该编码哪些时空特征，以及自然动作中特征的出现特征是什么。当前的全局编码方案严重依赖于分割，而局部编码方案缺乏描述能力。在这里，我们提出了自然动作结构，即局部特征的多尺寸、多尺度、时空串联，作为表示自然动作的基本特征。在这个概念中，任何动作都是一组自然动作结构的时空串联，这些结构传递了关于自然动作的全方位信息。我们采取了几个步骤来提取这些结构。首先，我们在多个时空尺度上对大量的补丁序列进行了采样。其次，我们对补丁序列进行了独立成分分析，并将独立成分分类为簇。最后，我们编译了一组大型的自然动作结构，每个结构对应于所选时空尺度上的唯一簇组合。为了对人类动作进行分类，我们使用了一组信息丰富的自然动作结构作为两个广泛使用的模型的输入。我们发现，这里获得的自然动作结构的识别性能明显优于低级特征，性能优于或与当前最好的模型相当。我们还发现，使用自然动作结构作为特征的分类性能受尺度变化和人为添加噪声的轻微影响。我们得出的结论是，这里提出的自然动作结构可以用作动作的基本编码单元，并且可能是理解自然动作的关键。
