Yang Zijiang, Qiu Zhongwei, Xu Chang, Fu Dongmei
IEEE Trans Vis Comput Graph. 2025 Sep;31(9):5842-5853. doi: 10.1109/TVCG.2024.3476331.
3D style transfer aims to generate stylized views of 3D scenes with specified styles, which requires high-quality generating and keeping multi-view consistency. Existing methods still suffer the challenges of high-quality stylization with texture details and stylization with multimodal guidance. In this paper, we reveal that the common training method of stylization with NeRF, which generates stylized multi-view supervision by 2D style transfer models, causes the same object in supervision to show various states (color tone, details, etc.) in different views, leading NeRF to tend to smooth the texture details, further resulting in low-quality rendering for 3D multi-style transfer. To tackle these problems, we propose a novel Multimodal-guided 3D Multi-style transfer of NeRF, termed MM-NeRF. First, MM-NeRF projects multimodal guidance into a unified space to keep the multimodal styles consistency and extracts multimodal features to guide the 3D stylization. Second, a novel multi-head learning scheme is proposed to relieve the difficulty of learning multi-style transfer, and a multi-view style consistent loss is proposed to track the inconsistency of multi-view supervision data. Finally, a novel incremental learning mechanism is proposed to generalize MM-NeRF to any new style with small costs. Extensive experiments on several real-world datasets show that MM-NeRF achieves high-quality 3D multi-style stylization with multimodal guidance, and keeps multi-view consistency and style consistency between multimodal guidance.
三维风格迁移旨在生成具有特定风格的三维场景的风格化视图，这需要高质量的生成并保持多视图一致性。现有方法在带有纹理细节的高质量风格化以及多模态引导的风格化方面仍面临挑战。在本文中，我们揭示了使用神经辐射场（NeRF）进行风格化的常见训练方法，即通过二维风格迁移模型生成风格化的多视图监督，会导致监督中的同一物体在不同视图中呈现出不同状态（色调、细节等），使得NeRF倾向于平滑纹理细节，进而导致三维多风格迁移的渲染质量较低。为了解决这些问题，我们提出了一种新颖的神经辐射场多模态引导三维多风格迁移方法，称为MM-NeRF。首先，MM-NeRF将多模态引导投影到统一空间以保持多模态风格的一致性，并提取多模态特征以指导三维风格化。其次，提出了一种新颖的多头学习方案来缓解学习多风格迁移的困难，并提出了一种多视图风格一致损失来跟踪多视图监督数据的不一致性。最后，提出了一种新颖的增量学习机制，以低成本将MM-NeRF推广到任何新风格。在多个真实世界数据集上进行的大量实验表明，MM-NeRF在多模态引导下实现了高质量的三维多风格化，并保持了多视图一致性以及多模态引导之间的风格一致性。
