Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, EPFL, Lausanne, Switzerland.
Computer Vision Laboratory, EPFL, Lausanne, Switzerland.
Nat Methods. 2021 Aug;18(8):975-981. doi: 10.1038/s41592-021-01226-z. Epub 2021 Aug 5.
Markerless three-dimensional (3D) pose estimation has become an indispensable tool for kinematic studies of laboratory animals. Most current methods recover 3D poses by multi-view triangulation of deep network-based two-dimensional (2D) pose estimates. However, triangulation requires multiple synchronized cameras and elaborate calibration protocols that hinder its widespread adoption in laboratory studies. Here we describe LiftPose3D, a deep network-based method that overcomes these barriers by reconstructing 3D poses from a single 2D camera view. We illustrate LiftPose3D's versatility by applying it to multiple experimental systems using flies, mice, rats and macaques, and in circumstances where 3D triangulation is impractical or impossible. Our framework achieves accurate lifting for stereotypical and nonstereotypical behaviors from different camera angles. Thus, LiftPose3D permits high-quality 3D pose estimation in the absence of complex camera arrays and tedious calibration procedures and despite occluded body parts in freely behaving animals.
无标记三维（3D）姿势估计已成为实验室动物运动学研究不可或缺的工具。目前大多数方法通过基于深度网络的二维（2D）姿势估计的多视图三角测量来恢复 3D 姿势。然而，三角测量需要多个同步摄像机和精细的校准协议，这阻碍了它在实验室研究中的广泛采用。在这里，我们描述了 LiftPose3D，这是一种基于深度网络的方法，通过从单个 2D 摄像机视图重建 3D 姿势来克服这些障碍。我们通过将其应用于使用苍蝇、老鼠、大鼠和猕猴的多个实验系统，并在 3D 三角测量不切实际或不可能的情况下，说明了 LiftPose3D 的多功能性。我们的框架可以从不同的摄像机角度对典型和非典型行为进行精确的提升。因此，即使在自由活动的动物中存在部分遮挡的身体部位，LiftPose3D 也可以在没有复杂的摄像机阵列和繁琐的校准程序的情况下进行高质量的 3D 姿势估计。
