IEEE Trans Cybern. 2013 Oct;43(5):1370-82. doi: 10.1109/TCYB.2013.2272321. Epub 2013 Jul 22.
We present an algorithm for creating free-viewpoint video of interacting humans using three handheld Kinect cameras. Our method reconstructs deforming surface geometry and temporal varying texture of humans through estimation of human poses and camera poses for every time step of the RGBZ video. Skeletal configurations and camera poses are found by solving a joint energy minimization problem, which optimizes the alignment of RGBZ data from all cameras, as well as the alignment of human shape templates to the Kinect data. The energy function is based on a combination of geometric correspondence finding, implicit scene segmentation, and correspondence finding using image features. Finally, texture recovery is achieved through jointly optimization on spatio-temporal RGB data using matrix completion. As opposed to previous methods, our algorithm succeeds on free-viewpoint video of human actors under general uncontrolled indoor scenes with potentially dynamic background, and it succeeds even if the cameras are moving.
我们提出了一种使用三个手持 Kinect 相机生成交互人体自由视点视频的算法。我们的方法通过对每个 RGBZ 视频时间步的人体姿势和相机姿势进行估计，重建变形的表面几何形状和随时间变化的纹理。骨骼配置和相机姿势是通过求解联合能量最小化问题找到的，该问题优化了来自所有相机的 RGBZ 数据的对齐，以及人体形状模板与 Kinect 数据的对齐。能量函数基于几何对应关系发现、隐式场景分割以及使用图像特征进行对应关系发现的组合。最后，通过使用矩阵完成对时空 RGB 数据进行联合优化来实现纹理恢复。与以前的方法不同，我们的算法在具有潜在动态背景的一般不受控制的室内场景下的人体演员的自由视点视频中取得了成功，即使相机在移动，它也能成功。
