Su Rui, Xu Dong, Sheng Lu, Ouyang Wanli
IEEE Trans Image Process. 2021;30:2103-2113. doi: 10.1109/TIP.2020.3044218. Epub 2021 Jan 25.
There are two major lines of works, i.e., anchor-based and frame-based approaches, in the field of temporal action localization. But each line of works is inherently limited to a certain detection granularity and cannot simultaneously achieve high recall rates with accurate action boundaries. In this work, we propose a progressive cross-granularity cooperation (PCG-TAL) framework to effectively take advantage of complementarity between the anchor-based and frame-based paradigms, as well as between two-view clues (i.e., appearance and motion). Specifically, our new Anchor-Frame Cooperation (AFC) module can effectively integrate both two-granularity and two-stream knowledge at the feature and proposal levels, as well as within each AFC module and across adjacent AFC modules. Specifically, the RGB-stream AFC module and the flow-stream AFC module are stacked sequentially to form a progressive localization framework. The whole framework can be learned in an end-to-end fashion, whilst the temporal action localization performance can be gradually boosted in a progressive manner. Our newly proposed framework outperforms the state-of-the-art methods on three benchmark datasets the THUMOS14, ActivityNet v1.3 and UCF-101-24, which clearly demonstrates the effectiveness of our framework.
在时间动作定位领域，主要有两大类工作，即基于锚点的方法和基于帧的方法。但每一类工作本质上都局限于特定的检测粒度，无法同时实现高召回率和精确的动作边界。在这项工作中，我们提出了一种渐进式跨粒度协作（PCG-TAL）框架，以有效利用基于锚点和基于帧的范式之间以及双视图线索（即外观和运动）之间的互补性。具体而言，我们新的锚点-帧协作（AFC）模块可以在特征和提议级别，以及在每个AFC模块内部和相邻AFC模块之间有效地整合双粒度和双流知识。具体来说，RGB流AFC模块和光流AFC模块依次堆叠，形成一个渐进式定位框架。整个框架可以以端到端的方式进行学习，同时时间动作定位性能可以以渐进的方式逐步提高。我们新提出的框架在三个基准数据集THUMOS14、ActivityNet v1.3和UCF-101-24上优于现有方法，这清楚地证明了我们框架的有效性。
