Tschernezki Vadim, Laina Iro, Larlus Diane, Vedaldi Andrea
Visual Geometry Group, University of Oxford.
NAVER LABS Europe.
Proc Int Conf 3D Vis. 2023 Feb 22;2022. doi: 10.1109/3DV57658.2022.00056. eCollection 2022 Sep 16.
We present Neural Feature Fusion Fields (N3F), a method that improves dense 2D image feature extractors when the latter are applied to the analysis of multiple images reconstructible as a 3D scene. Given an image feature extractor, for example pre-trained using self-supervision, N3F uses it as a teacher to learn a student network defined in 3D space. The 3D student network is similar to a neural radiance field that distills said features and can be trained with the usual differentiable rendering machinery. As a consequence, N3F is readily applicable to most neural rendering formulations, including vanilla NeRF and its extensions to complex dynamic scenes. We show that our method not only enables semantic understanding in the context of scene-specific neural fields without the use of manual labels,but also consistently improves over the self-supervised 2D baselines. This is demonstrated by considering various tasks, such as 2D object retrieval, 3D segmentation, and scene editing, in diverse sequences, including long egocentric videos in the EPIC-KITCHENS benchmark. Project page: https://www.robots.ox.ac.uk/~vadim/n3f/.
我们提出了神经特征融合场（N3F），这是一种在将密集二维图像特征提取器应用于可重建为三维场景的多幅图像分析时对其进行改进的方法。给定一个图像特征提取器，例如使用自监督进行预训练的提取器，N3F将其用作教师来学习在三维空间中定义的学生网络。三维学生网络类似于一个神经辐射场，它提炼上述特征，并且可以使用常见的可微渲染机制进行训练。因此，N3F很容易应用于大多数神经渲染公式，包括普通的神经辐射场（NeRF）及其对复杂动态场景的扩展。我们表明，我们的方法不仅能够在不使用手动标签的情况下在特定场景神经场的背景下实现语义理解，而且始终优于自监督二维基线。这在考虑各种任务时得到了证明，例如二维物体检索、三维分割和场景编辑，这些任务在不同的序列中进行，包括EPIC-KITCHENS基准中的长第一人称视角视频。项目页面：https://www.robots.ox.ac.uk/~vadim/n3f/ 。
