Masuda Takeshi, Sagawa Ryusuke, Furukawa Ryo, Kawasaki Hiroshi
Annu Int Conf IEEE Eng Med Biol Soc. 2024 Jul;2024:1-6. doi: 10.1109/EMBC53108.2024.10782148.
We present a novel method for synthesising new views of endoscopic images, utilising an innovative 3D shape representation technique known as Gaussian splatting. Structure from Motion (SfM) has traditionally been employed for this purpose to recover 3D shapes of feature points and poses from video sequences, but has only been effective in limited situations due to the featureless, shiny and deformable organ internal surfaces of endoscopic images.To address this challenge, we propose a hybrid method that combines a single depth map estimation using deep neural networks (DNNs) with a Gaussian splatting approach. Our method incorporates 4D Gaussian splatting, optimising the shape and colour Gaussians of the point cloud to align with the input image and synthesising new views by considering deformations. Experiments demonstrate that the proposed method produces less degraded results compared to Gaussian splatting using SfM, a conventional method, using real endoscopic images, and is useful in cases where a conventional method, e.g. SfM, cannot be used.
我们提出了一种合成内窥镜图像新视图的新颖方法，该方法利用了一种名为高斯平铺的创新3D形状表示技术。传统上，运动结构（SfM）已被用于此目的，从视频序列中恢复特征点的3D形状和姿态，但由于内窥镜图像中无特征、有光泽且可变形的器官内表面，该方法仅在有限的情况下有效。为应对这一挑战，我们提出了一种混合方法，该方法将使用深度神经网络（DNN）的单深度图估计与高斯平铺方法相结合。我们的方法结合了4D高斯平铺，优化点云的形状和颜色高斯以与输入图像对齐，并通过考虑变形来合成新视图。实验表明，与使用SfM（一种传统方法）的高斯平铺相比，该方法在使用真实内窥镜图像时产生的结果退化程度更低，并且在无法使用传统方法（例如SfM）的情况下很有用。
