IEEE Trans Image Process. 2019 Apr;28(4):1837-1850. doi: 10.1109/TIP.2018.2879249. Epub 2018 Nov 2.
The usage of both off-the-shelf and end-to-end trained deep networks have significantly improved the performance of visual tracking on RGB videos. However, the lack of large labeled datasets hampers the usage of convolutional neural networks for tracking in thermal infrared (TIR) images. Therefore, most state-of-the-art methods on tracking for TIR data are still based on handcrafted features. To address this problem, we propose to use image-to-image translation models. These models allow us to translate the abundantly available labeled RGB data to synthetic TIR data. We explore both the usage of paired and unpaired image translation models for this purpose. These methods provide us with a large labeled dataset of synthetic TIR sequences, on which we can train end-to-end optimal features for tracking. To the best of our knowledge, we are the first to train end-to-end features for TIR tracking. We perform extensive experiments on the VOT-TIR2017 dataset. We show that a network trained on a large dataset of synthetic TIR data obtains better performance than one trained on the available real TIR data. Combining both data sources leads to further improvement. In addition, when we combine the network with motion features, we outperform the state of the art with a relative gain of over 10%, clearly showing the efficiency of using synthetic data to train end-to-end TIR trackers.
现成的和端到端训练的深度网络的使用显著提高了 RGB 视频视觉跟踪的性能。然而，缺乏大型标记数据集阻碍了卷积神经网络在热红外 (TIR) 图像跟踪中的使用。因此，大多数最先进的 TIR 数据跟踪方法仍然基于手工制作的特征。为了解决这个问题，我们提出使用图像到图像翻译模型。这些模型允许我们将大量可用的标记 RGB 数据转换为合成的 TIR 数据。我们探索了使用配对和未配对图像翻译模型的方法。这些方法为我们提供了大量的合成 TIR 序列标记数据集，我们可以在这些数据集上训练端到端的最优跟踪特征。据我们所知，我们是第一个为 TIR 跟踪训练端到端特征的人。我们在 VOT-TIR2017 数据集上进行了广泛的实验。我们表明，在大型合成 TIR 数据集中训练的网络比在可用的真实 TIR 数据中训练的网络具有更好的性能。结合这两种数据源可以进一步提高性能。此外，当我们将网络与运动特征结合使用时，我们的表现优于最先进的方法，相对增益超过 10%，这清楚地表明了使用合成数据来训练端到端 TIR 跟踪器的效率。
