Chen Lin-Zhuo, Lin Zheng, Wang Ziqin, Yang Yong-Liang, Cheng Ming-Ming
IEEE Trans Image Process. 2021;30:2313-2324. doi: 10.1109/TIP.2021.3049332. Epub 2021 Jan 27.
3D spatial information is known to be beneficial to the semantic segmentation task. Most existing methods take 3D spatial data as an additional input, leading to a two-stream segmentation network that processes RGB and 3D spatial information separately. This solution greatly increases the inference time and severely limits its scope for real-time applications. To solve this problem, we propose Spatial information guided Convolution (S-Conv), which allows efficient RGB feature and 3D spatial information integration. S-Conv is competent to infer the sampling offset of the convolution kernel guided by the 3D spatial information, helping the convolutional layer adjust the receptive field and adapt to geometric transformations. S-Conv also incorporates geometric information into the feature learning process by generating spatially adaptive convolutional weights. The capability of perceiving geometry is largely enhanced without much affecting the amount of parameters and computational cost. Based on S-Conv, we further design a semantic segmentation network, called Spatial information Guided convolutional Network (SGNet), resulting in real-time inference and state-of-the-art performance on NYUDv2 and SUNRGBD datasets.
已知3D空间信息对语义分割任务有益。大多数现有方法将3D空间数据作为额外输入，导致一个双流分割网络，该网络分别处理RGB和3D空间信息。这种解决方案大大增加了推理时间，并严重限制了其在实时应用中的范围。为了解决这个问题，我们提出了空间信息引导卷积（S-Conv），它允许高效地整合RGB特征和3D空间信息。S-Conv能够推断由3D空间信息引导的卷积核的采样偏移，帮助卷积层调整感受野并适应几何变换。S-Conv还通过生成空间自适应卷积权重将几何信息纳入特征学习过程。在不影响参数数量和计算成本的情况下，感知几何的能力得到了很大增强。基于S-Conv，我们进一步设计了一个语义分割网络，称为空间信息引导卷积网络（SGNet），在NYUDv2和SUNRGBD数据集上实现了实时推理和领先的性能。
