Dutta Abhishek, Pérez-Campanero Natalia, Taylor Graham K, Zisserman Andrew, Newport Cait
Department of Engineering Science, University of Oxford, Oxford, UK.
Department of Biology, University of Oxford, Oxford, UK.
R Soc Open Sci. 2025 Jul 23;12(7):242086. doi: 10.1098/rsos.242086. eCollection 2025 Jul.
We introduce a novel video processing method called Detect+Track that combines a deep learning-based object detector with a template-based object agnostic tracker to significantly enhance the accuracy and robustness of animal tracking. Applied to a behavioural experiment involving Picasso triggerfish () navigating a randomized array of cylindrical obstacles, the method accurately localizes fish centroids across challenging conditions including occlusion, variable lighting, body deformation and surface ripples. Virtual gates between adjacent obstacles and between obstacles and tank boundaries are computed using Voronoi tessellation and planar homology, enabling detailed analysis of gap selection behaviour. Fish speed, movement direction and a more precise estimate of body centroid-key metrics for behavioural analyses-are estimated using optical flow method. The modular workflow is adaptable to new experimental designs, supports manual correction and retraining for new object classes and allows efficient large-scale batch processing. By addressing key limitations of existing tracking tools, Detect+Track provides a flexible and generalizable solution for researchers studying movement and decision-making in complex environments. A detailed tutorial is provided, together with all the data and code required to reproduce our results and enable future innovations in behavioural tracking and analysis.
我们介绍了一种名为Detect+Track的新型视频处理方法，该方法将基于深度学习的目标检测器与基于模板的目标无关跟踪器相结合，以显著提高动物跟踪的准确性和鲁棒性。将该方法应用于一项行为实验，实验中毕加索扳机鱼在一组随机排列的圆柱形障碍物中穿梭，该方法能够在包括遮挡、光照变化、身体变形和水面涟漪等具有挑战性的条件下准确地定位鱼的质心。利用Voronoi镶嵌和平面同调计算相邻障碍物之间以及障碍物与水箱边界之间的虚拟门，从而能够对间隙选择行为进行详细分析。使用光流法估计鱼的速度、运动方向以及对行为分析至关重要的身体质心的更精确估计。模块化工作流程适用于新的实验设计，支持针对新对象类别的手动校正和重新训练，并允许进行高效的大规模批处理。通过解决现有跟踪工具的关键局限性，Detect+Track为研究复杂环境中运动和决策的研究人员提供了一种灵活且通用的解决方案。我们提供了详细的教程，以及重现我们的结果并推动行为跟踪和分析未来创新所需的所有数据和代码。
