Chen Zheng, Wang Chen, Guo Yuan-Chen, Zhang Song-Hai
IEEE Trans Pattern Anal Mach Intell. 2023 Dec;45(12):15694-15705. doi: 10.1109/TPAMI.2023.3305295. Epub 2023 Nov 3.
Neural Radiance Fields (NeRF) achieve photo-realistic view synthesis with densely captured input images. However, the geometry of NeRF is extremely under-constrained given sparse views, resulting in significant degradation of novel view synthesis quality. Inspired by self-supervised depth estimation methods, we propose StructNeRF, a solution to novel view synthesis for indoor scenes with sparse inputs. StructNeRF leverages the structural hints naturally embedded in multi-view inputs to handle the unconstrained geometry issue in NeRF. Specifically, it tackles the texture and non-texture regions respectively: a patch-based multi-view consistent photometric loss is proposed to constrain the geometry of textured regions; for non-textured ones, we explicitly restrict them to be 3D consistent planes. Through the dense self-supervised depth constraints, our method improves both the geometry and the view synthesis performance of NeRF without any additional training on external data. Extensive experiments on several real-world datasets demonstrate that StructNeRF shows superior or comparable performance compared to state-of-the-art methods (e.g. NeRF, DSNeRF, RegNeRF, Dense Depth Priors, MonoSDF, etc.) for indoor scenes with sparse inputs both quantitatively and qualitatively.
神经辐射场（NeRF）通过密集采集的输入图像实现逼真的视图合成。然而，在稀疏视图的情况下，NeRF的几何结构受到极大的约束，导致新视图合成质量显著下降。受自监督深度估计方法的启发，我们提出了StructNeRF，这是一种针对具有稀疏输入的室内场景进行新视图合成的解决方案。StructNeRF利用多视图输入中自然嵌入的结构线索来处理NeRF中无约束的几何问题。具体来说，它分别处理纹理区域和非纹理区域：提出了一种基于块的多视图一致光度损失来约束纹理区域的几何结构；对于非纹理区域，我们明确将它们限制为3D一致平面。通过密集的自监督深度约束，我们的方法在无需对外部数据进行任何额外训练的情况下，提高了NeRF的几何结构和视图合成性能。在多个真实世界数据集上进行的大量实验表明，对于具有稀疏输入的室内场景，StructNeRF在定量和定性方面均表现出优于或可比于现有方法（如NeRF、DSNeRF、RegNeRF、Dense Depth Priors、MonoSDF等）的性能。
