Center for Biophysics and Quantitative Biology, University of Illinois at Urbana-Champaign, Urbana, Illinois, United States of America.
Center for the Physics of Living Cells, University of Illinois at Urbana-Champaign, Urbana, Illinois, United States of America.
PLoS Comput Biol. 2023 Oct 23;19(10):e1011566. doi: 10.1371/journal.pcbi.1011566. eCollection 2023 Oct.
Quantitative ethology requires an accurate estimation of an organism's postural dynamics in three dimensions plus time. Technological progress over the last decade has made animal pose estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a fast automated method to record and track the pose of individual larval zebrafish in a 3-D environment, applicable when accurate human labeling is not possible; (ii) a rich annotated dataset of 3-D larval poses for ethologists and the general zebrafish and machine learning community; and (iii) a technique to generate realistic, annotated larval images in different behavioral contexts. Using a three-camera system calibrated with refraction correction, we record diverse larval swims under free swimming conditions and in response to acoustic and optical stimuli. We then employ a convolutional neural network to estimate 3-D larval poses from video images. The network is trained against a set of synthetic larval images rendered using a 3-D physical model of larvae. This 3-D model samples from a distribution of realistic larval poses that we estimate a priori using a template-based pose estimation of a small number of swim bouts. Our network model, trained without any human annotation, performs larval pose estimation three orders of magnitude faster and with accuracy comparable to the template-based approach, capturing detailed kinematics of 3-D larval swims. It also applies accurately to other datasets collected under different imaging conditions and containing behavioral contexts not included in our training.
定量行为学需要准确估计生物体在三维空间加时间内的姿势动态。过去十年的技术进步使得在具有挑战性的场景中对动物姿势进行估计成为可能，并且具有前所未有的细节。在这里，我们提出了 (i) 一种快速自动化的方法，用于记录和跟踪 3-D 环境中个体幼鱼的姿势，适用于无法进行准确人工标记的情况；(ii) 为行为学家以及普通斑马鱼和机器学习社区提供了丰富的 3-D 幼鱼姿势注释数据集；以及 (iii) 一种在不同行为背景下生成逼真、注释的幼鱼图像的技术。我们使用带有折射校正的三相机系统进行记录，记录了在自由游动条件下以及对声和光刺激的反应下的各种幼鱼游动。然后，我们使用卷积神经网络从视频图像中估计 3-D 幼鱼姿势。该网络是针对使用幼虫的 3-D 物理模型渲染的一组合成幼虫图像进行训练的。该 3-D 模型从我们使用少数游泳回合的基于模板的姿势估计来预先估计的真实幼虫姿势分布中进行采样。我们的网络模型在没有任何人工注释的情况下进行训练，其执行幼虫姿势估计的速度快三个数量级，并且与基于模板的方法的准确性相当，能够捕获 3-D 幼虫游动的详细运动学。它还可以准确地应用于在不同成像条件下收集的其他数据集，并且包含不在我们训练范围内的行为背景。
