Chaumeil Anaïs, Puchaud Pierre, Muller Antoine, Dumas Raphaël, Robert Thomas
Univ Eiffel, Univ Lyon 1, LBMC UMR T_9406, Lyon, France.
Laboratoire de Simulation et Modélisation du Mouvement, Département de Kinésiologie, Université de Montréal, Laval, Canada.
Int J Numer Method Biomed Eng. 2025 Aug;41(8):e70079. doi: 10.1002/cnm.70079.
Multi-camera markerless motion capture commonly triangulates 3D points from 2D keypoint positions in multiple camera views, then applies a multibody kinematics optimization (MKO) to incorporate biomechanical constraints. However, standard pipelines neglect the 2D confidence heatmaps generated by human pose estimation networks. We hypothesized that performing MKO in 2D camera planes would make it more robust to missing keypoints and allow us to obtain better accuracy. 2D confidence heatmaps were used to maximize available information. To test this, we first model each network-derived heatmap as a 2D Gaussian function characterized by its center, amplitude, and standard deviation. Second, we maximize the sum of these modeled confidences after projecting the biomechanical model into the camera planes. To demonstrate feasibility, we evaluated our method on data from two participants performing sit-to-stand, walking, and manual material handling, captured by a two-camera setup, and simultaneously collected marker-based data. Our Gaussian modeling of the heatmaps demonstrated a mean absolute difference of 0.011 compared to the original discrete maps, confirming its validity. In terms of 3D joint positions and angles, the confidence-based MKO produced results similar to classical distance-based methods. Notably, the confidence-based approach overcame occultations: 89.3% of frames could only be obtained with the distance-based MKO due to missing keypoints, while the confidence-based MKO computed 100% of frames. These findings underscore the potential of using full 2D confidence heatmaps in markerless motion capture, especially under challenging conditions such as sparse camera setups.
多相机无标记运动捕捉通常从多个相机视图中的二维关键点位置对三维点进行三角测量，然后应用多体运动学优化（MKO）来纳入生物力学约束。然而，标准流程忽略了人体姿态估计网络生成的二维置信度热图。我们假设在二维相机平面中执行MKO将使其对缺失的关键点更具鲁棒性，并使我们能够获得更高的精度。二维置信度热图被用于最大化可用信息。为了验证这一点，我们首先将每个网络生成的热图建模为一个二维高斯函数，其特征由中心、幅度和标准差表示。其次，在将生物力学模型投影到相机平面后，我们最大化这些建模置信度的总和。为了证明其可行性，我们在由双相机设置捕捉的两名参与者进行从坐起到站立、行走和手动搬运材料的数据上评估了我们的方法，并同时收集了基于标记的数据。我们对热图的高斯建模与原始离散图相比，平均绝对差为0.011，证实了其有效性。在三维关节位置和角度方面，基于置信度的MKO产生的结果与传统的基于距离的方法相似。值得注意的是，基于置信度的方法克服了遮挡问题：由于关键点缺失，89.3%的帧只能通过基于距离的MKO获得，而基于置信度的MKO计算了100%的帧。这些发现强调了在无标记运动捕捉中使用完整二维置信度热图的潜力，特别是在诸如稀疏相机设置等具有挑战性的条件下。
