Deng Yong, Xiao Jimin, Zhou Steven Zhiying, Feng Jiashi
IEEE Trans Image Process. 2021;30:5835-5847. doi: 10.1109/TIP.2021.3088635. Epub 2021 Jun 24.
The Coarse-To-Fine (CTF) matching scheme has been widely applied to reduce computational complexity and matching ambiguity in stereo matching and optical flow tasks by converting image pairs into multi-scale representations and performing matching from coarse to fine levels. Despite its efficiency, it suffers from several weaknesses, such as tending to blur the edges and miss small structures like thin bars and holes. We find that the pixels of small structures and edges are often assigned with wrong disparity/flow in the upsampling process of the CTF framework, introducing errors to the fine levels and leading to such weaknesses. We observe that these wrong disparity/flow values can be avoided if we select the best-matched value among their neighborhood, which inspires us to propose a novel differentiable Neighbor-Search Upsampling (NSU) module. The NSU module first estimates the matching scores and then selects the best-matched disparity/flow for each pixel from its neighbors. It effectively preserves finer structure details by exploiting the information from the finer level while upsampling the disparity/flow. The proposed module can be a drop-in replacement of the naive upsampling in the CTF matching framework and allows the neural networks to be trained end-to-end. By integrating the proposed NSU module into a baseline CTF matching network, we design our Detail Preserving Coarse-To-Fine (DPCTF) matching network. Comprehensive experiments demonstrate that our DPCTF can boost performances for both stereo matching and optical flow tasks. Notably, our DPCTF achieves new state-of-the-art performances for both tasks - it outperforms the competitive baseline (Bi3D) by 28.8% (from 0.73 to 0.52) on EPE of the FlyingThings3D stereo dataset, and ranks first in KITTI flow 2012 benchmark. The code is available at https://github.com/Deng-Y/DPCTF.
粗到细（CTF）匹配方案已被广泛应用于立体匹配和光流任务中，通过将图像对转换为多尺度表示并从粗到细级别进行匹配，以降低计算复杂度和匹配模糊性。尽管它效率高，但也存在一些弱点，比如容易模糊边缘，错过像细条和孔洞这样的小结构。我们发现，在CTF框架的上采样过程中，小结构和边缘的像素经常被赋予错误的视差/流，从而给精细级别引入误差并导致这些弱点。我们观察到，如果在其邻域中选择最佳匹配值，这些错误的视差/流值是可以避免的，这启发我们提出一种新颖的可微邻居搜索上采样（NSU）模块。NSU模块首先估计匹配分数，然后从其邻居中为每个像素选择最佳匹配的视差/流。它在对视差/流进行上采样时，通过利用精细级别的信息有效地保留了更精细的结构细节。所提出的模块可以直接替代CTF匹配框架中的朴素上采样，并允许神经网络进行端到端训练。通过将所提出的NSU模块集成到基线CTF匹配网络中，我们设计了我们的细节保留粗到细（DPCTF）匹配网络。综合实验表明，我们的DPCTF可以提高立体匹配和光流任务的性能。值得注意的是，我们的DPCTF在这两个任务中都取得了新的最优性能——在FlyingThings3D立体数据集的端点误差（EPE）上比有竞争力的基线（Bi3D）提高了28.8%（从0.73降至0.52），并且在KITTI光流2012基准测试中排名第一。代码可在https://github.com/Deng-Y/DPCTF获取。
