Liu XiaoLe, Yu Si-Yang, Flierman Nico A, Loyola Sebastián, Kamermans Maarten, Hoogland Tycho M, De Zeeuw Chris I
Faculty of Mathematics, University of Waterloo, Waterloo, ON, Canada.
Department of Neuroscience, Erasmus MC, Rotterdam, Netherlands.
Front Cell Neurosci. 2021 May 28;15:621252. doi: 10.3389/fncel.2021.621252. eCollection 2021.
Animal pose estimation tools based on deep learning have greatly improved animal behaviour quantification. These tools perform pose estimation on individual video frames, but do not account for variability of animal body shape in their prediction and evaluation. Here, we introduce a novel multi-frame animal pose estimation framework, referred to as OptiFlex. This framework integrates a flexible base model (i.e., FlexibleBaseline), which accounts for variability in animal body shape, with an OpticalFlow model that incorporates temporal context from nearby video frames. Pose estimation can be optimised using multi-view information to leverage all four dimensions (3D space and time). We evaluate FlexibleBaseline using datasets of four different lab animal species (mouse, fruit fly, zebrafish, and monkey) and introduce an intuitive evaluation metric-adjusted percentage of correct key points (aPCK). Our analyses show that OptiFlex provides prediction accuracy that outperforms current deep learning based tools, highlighting its potential for studying a wide range of behaviours across different animal species.
基于深度学习的动物姿态估计工具极大地改进了动物行为量化。这些工具对单个视频帧进行姿态估计，但在预测和评估中没有考虑动物体型的变异性。在此，我们引入了一种新颖的多帧动物姿态估计框架，称为OptiFlex。该框架将一个灵活的基础模型（即FlexibleBaseline，它考虑了动物体型的变异性）与一个光流模型相结合，该光流模型整合了来自附近视频帧的时间上下文信息。姿态估计可以使用多视图信息进行优化，以利用所有四个维度（三维空间和时间）。我们使用四种不同实验动物物种（小鼠、果蝇、斑马鱼和猴子）的数据集对FlexibleBaseline进行评估，并引入了一种直观的评估指标——调整后的关键点正确百分比（aPCK）。我们的分析表明，OptiFlex提供的预测准确性优于当前基于深度学习的工具，突出了其在研究不同动物物种的广泛行为方面的潜力。
