Liu Shuqi, Wang Gang, Song Yong, Huang Jinxiang, Huang Yiqian, Zhou Ya, Wang Shiqiang
School of Optics and Photonics, Beijing Institute of Technology, Beijing, China.
Center of Brain Sciences, Beijing Institute of Basic Medical Sciencesy, Beijing, China.
Front Neurosci. 2024 Aug 8;18:1453419. doi: 10.3389/fnins.2024.1453419. eCollection 2024.
Integrating RGB and Event (RGBE) multi-domain information obtained by high-dynamic-range and temporal-resolution event cameras has been considered an effective scheme for robust object tracking. However, existing RGBE tracking methods have overlooked the unique spatio-temporal features over different domains, leading to object tracking failure and inefficiency, especally for objects against complex backgrounds. To address this problem, we propose a novel tracker based on adaptive-time feature extraction hybrid networks, namely Siamese Event Frame Tracker (SiamEFT), which focuses on the effective representation and utilization of the diverse spatio-temporal features of RGBE. We first design an adaptive-time attention module to aggregate event data into frames based on adaptive-time weights to enhance information representation. Subsequently, the SiamEF module and cross-network fusion module combining artificial neural networks and spiking neural networks hybrid network are designed to effectively extract and fuse the spatio-temporal features of RGBE. Extensive experiments on two RGBE datasets (VisEvent and COESOT) show that the SiamEFT achieves a success rate of 0.456 and 0.574, outperforming the state-of-the-art competing methods and exhibiting a 2.3-fold enhancement in efficiency. These results validate the superior accuracy and efficiency of SiamEFT in diverse and challenging scenes.
融合由高动态范围和时间分辨率事件相机获得的RGB和事件（RGBE）多域信息，已被视为一种用于鲁棒目标跟踪的有效方案。然而，现有的RGBE跟踪方法忽略了不同域上独特的时空特征，导致目标跟踪失败和效率低下，尤其是对于处于复杂背景下的目标。为了解决这个问题，我们提出了一种基于自适应时间特征提取混合网络的新型跟踪器，即暹罗事件帧跟踪器（SiamEFT），它专注于RGBE多样时空特征的有效表示和利用。我们首先设计了一个自适应时间注意力模块，基于自适应时间权重将事件数据聚合到帧中，以增强信息表示。随后，设计了暹罗事件帧跟踪（SiamEF）模块和结合人工神经网络与脉冲神经网络的跨网络融合模块的混合网络，以有效提取和融合RGBE的时空特征。在两个RGBE数据集（VisEvent和COESOT）上进行的大量实验表明，SiamEFT的成功率分别为0.456和0.574，优于当前最先进的竞争方法，并且效率提高了2.3倍。这些结果验证了SiamEFT在多样且具有挑战性的场景中的卓越准确性和效率。
