Zhu Zhe, Nan Liangliang, Xie Haoran, Chen Honghua, Wang Jun, Wei Mingqiang, Qin Jing
IEEE Trans Vis Comput Graph. 2024 Jul;30(7):3545-3563. doi: 10.1109/TVCG.2023.3236061. Epub 2024 Jun 27.
How will you repair a physical object with some missings? You may imagine its original shape from previously captured images, recover its overall (global) but coarse shape first, and then refine its local details. We are motivated to imitate the physical repair procedure to address point cloud completion. To this end, we propose a cross-modal shape-transfer dual-refinement network (termed CSDN), a coarse-to-fine paradigm with images of full-cycle participation, for quality point cloud completion. CSDN mainly consists of "shape fusion" and "dual-refinement" modules to tackle the cross-modal challenge. The first module transfers the intrinsic shape characteristics from single images to guide the geometry generation of the missing regions of point clouds, in which we propose IPAdaIN to embed the global features of both the image and the partial point cloud into completion. The second module refines the coarse output by adjusting the positions of the generated points, where the local refinement unit exploits the geometric relation between the novel and the input points by graph convolution, and the global constraint unit utilizes the input image to fine-tune the generated offset. Different from most existing approaches, CSDN not only explores the complementary information from images but also effectively exploits cross-modal data in the whole coarse-to-fine completion procedure. Experimental results indicate that CSDN performs favorably against twelve competitors on the cross-modal benchmark.
你将如何修复一个有缺失部分的物理对象？你可以从之前捕获的图像中想象它的原始形状，首先恢复其整体（全局）但粗略的形状，然后细化其局部细节。我们受到启发，模仿物理修复过程来解决点云补全问题。为此，我们提出了一种跨模态形状转移双细化网络（称为CSDN），这是一种具有全周期参与图像的从粗到细的范式，用于高质量的点云补全。CSDN主要由“形状融合”和“双细化”模块组成，以应对跨模态挑战。第一个模块从单张图像中转移内在形状特征，以指导点云缺失区域的几何生成，在这个过程中，我们提出了IPAdaIN，将图像和部分点云的全局特征嵌入到补全过程中。第二个模块通过调整生成点的位置来细化粗输出，其中局部细化单元通过图卷积利用新生成点与输入点之间的几何关系，全局约束单元利用输入图像来微调生成的偏移量。与大多数现有方法不同，CSDN不仅探索图像中的互补信息，还在整个从粗到细的补全过程中有效地利用跨模态数据。实验结果表明，在跨模态基准测试中，CSDN的表现优于十二个竞争对手。
