IEEE Trans Pattern Anal Mach Intell. 2022 Jun;44(6):3170-3184. doi: 10.1109/TPAMI.2021.3050505. Epub 2022 May 5.
Modeling 3D humans accurately and robustly from a single image is very challenging, and the key for such an ill-posed problem is the 3D representation of the human models. To overcome the limitations of regular 3D representations, we propose Parametric Model-Conditioned Implicit Representation (PaMIR), which combines the parametric body model with the free-form deep implicit function. In our PaMIR-based reconstruction framework, a novel deep neural network is proposed to regularize the free-form deep implicit function using the semantic features of the parametric model, which improves the generalization ability under the scenarios of challenging poses and various clothing topologies. Moreover, a novel depth-ambiguity-aware training loss is further integrated to resolve depth ambiguities and enable successful surface detail reconstruction with imperfect body reference. Finally, we propose a body reference optimization method to improve the parametric model estimation accuracy and to enhance the consistency between the parametric model and the implicit function. With the PaMIR representation, our framework can be easily extended to multi-image input scenarios without the need of multi-camera calibration and pose synchronization. Experimental results demonstrate that our method achieves state-of-the-art performance for image-based 3D human reconstruction in the cases of challenging poses and clothing types.
从单张图像中准确且稳健地对 3D 人体进行建模极具挑战性，而此类不适定问题的关键在于人体模型的 3D 表示。为了克服常规 3D 表示的局限性，我们提出了参数模型条件隐式表示（PaMIR），它将参数体模型与自由形态深度隐式函数相结合。在我们基于 PaMIR 的重建框架中，提出了一种新颖的深度神经网络，通过参数模型的语义特征对自由形态深度隐式函数进行正则化，从而提高了在挑战性姿势和各种服装拓扑场景下的泛化能力。此外，进一步集成了一种新颖的深度歧义感知训练损失，以解决深度歧义问题，并能够在身体参考不完美的情况下成功进行表面细节重建。最后，我们提出了一种身体参考优化方法，以提高参数模型估计的准确性，并增强参数模型和隐式函数之间的一致性。通过 PaMIR 表示，我们的框架可以轻松扩展到多图像输入场景，而无需进行多相机标定和姿势同步。实验结果表明，我们的方法在挑战性姿势和服装类型的基于图像的 3D 人体重建中达到了最先进的性能。
