National Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China.
Sensors (Basel). 2020 Mar 11;20(6):1560. doi: 10.3390/s20061560.
We designed an end-to-end dual-branch residual network architecture that inputs a low-resolution (LR) depth map and a corresponding high-resolution (HR) color image separately into the two branches, and outputs an HR depth map through a multi-scale, channel-wise feature extraction, interaction, and upsampling. Each branch of this network contains several residual levels at different scales, and each level comprises multiple residual groups composed of several residual blocks. A short-skip connection in every residual block and a long-skip connection in each residual group or level allow for low-frequency information to be bypassed while the main network focuses on learning high-frequency information. High-frequency information learned by each residual block in the color image branch is input into the corresponding residual block in the depth map branch, and this kind of channel-wise feature supplement and fusion can not only help the depth map branch to alleviate blur in details like edges, but also introduce some depth artifacts to feature maps. To avoid the above introduced artifacts, the channel interaction fuses the feature maps using weights referring to the channel attention mechanism. The parallel multi-scale network architecture with channel interaction for feature guidance is the main contribution of our work and experiments show that our proposed method had a better performance in terms of accuracy compared with other methods.
我们设计了一个端到端的双分支残差网络架构，该架构分别将低分辨率（LR）深度图和相应的高分辨率（HR）彩色图像输入到两个分支中，并通过多尺度、通道式特征提取、交互和上采样输出 HR 深度图。该网络的每个分支都包含几个不同尺度的残差层，每个层都由多个由几个残差块组成的残差组。每个残差块中的短跳跃连接和每个残差组或层中的长跳跃连接允许低频信息绕过，而主网络专注于学习高频信息。彩色图像分支中每个残差块学习到的高频信息被输入到深度图分支中的相应残差块中，这种通道式特征补充和融合不仅可以帮助深度图分支减轻边缘等细节的模糊，还可以为特征图引入一些深度伪影。为了避免引入上述伪影，通道交互使用参考通道注意力机制的权重来融合特征图。具有通道交互的并行多尺度网络架构是我们工作的主要贡献，实验表明，与其他方法相比，我们提出的方法在准确性方面具有更好的性能。
