School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China.
Peng Cheng Laboratory, Shenzhen 518055, China.
Sensors (Basel). 2021 Jan 20;21(3):690. doi: 10.3390/s21030690.
Semantic segmentation is one of the most widely studied problems in computer vision communities, which makes a great contribution to a variety of applications. A lot of learning-based approaches, such as Convolutional Neural Network (CNN), have made a vast contribution to this problem. While rich context information of the input images can be learned from multi-scale receptive fields by convolutions with deep layers, traditional CNNs have great difficulty in learning the geometrical relationship and distribution of objects in the RGB image due to the lack of depth information, which may lead to an inferior segmentation quality. To solve this problem, we propose a method that improves segmentation quality with depth estimation on RGB images. Specifically, we estimate depth information on RGB images via a depth estimation network, and then feed the depth map into the CNN which is able to guide the semantic segmentation. Furthermore, in order to parse the depth map and RGB images simultaneously, we construct a multi-branch encoder-decoder network and fuse the RGB and depth features step by step. Extensive experimental evaluation on four baseline networks demonstrates that our proposed method can enhance the segmentation quality considerably and obtain better performance compared to other segmentation networks.
语义分割是计算机视觉领域研究最广泛的问题之一，它为各种应用做出了巨大贡献。许多基于学习的方法，如卷积神经网络 (CNN)，对这个问题做出了巨大的贡献。虽然通过深层卷积可以从多尺度感受野中学习到输入图像的丰富上下文信息，但由于缺乏深度信息，传统的 CNN 在学习 RGB 图像中物体的几何关系和分布方面存在很大困难，这可能导致分割质量下降。为了解决这个问题，我们提出了一种利用 RGB 图像深度估计来提高分割质量的方法。具体来说，我们通过深度估计网络对 RGB 图像进行深度估计，然后将深度图输入到能够指导语义分割的 CNN 中。此外，为了同时解析深度图和 RGB 图像，我们构建了一个多分支编解码器网络，并逐步融合 RGB 和深度特征。在四个基线网络上的广泛实验评估表明，我们提出的方法可以显著提高分割质量，并获得比其他分割网络更好的性能。
