Huang Tao, Li Songjiang, Jia Xu, Lu Huchuan, Liu Jianzhuang
IEEE Trans Image Process. 2022;31:4023-4038. doi: 10.1109/TIP.2022.3176533. Epub 2022 Jun 14.
In recent years, image denoising has benefited a lot from deep neural networks. However, these models need large amounts of noisy-clean image pairs for supervision. Although there have been attempts in training denoising networks with only noisy images, existing self-supervised algorithms suffer from inefficient network training, heavy computational burden, or dependence on noise modeling. In this paper, we proposed a self-supervised framework named Neighbor2Neighbor for deep image denoising. We develop a theoretical motivation and prove that by designing specific samplers for training image pairs generation from only noisy images, we can train a self-supervised denoising network similar to the network trained with clean images supervision. Besides, we propose a regularizer in the perspective of optimization to narrow the optimization gap between the self-supervised denoiser and the supervised denoiser. We present a very simple yet effective self-supervised training scheme based on the theoretical understandings: training image pairs are generated by random neighbor sub-samplers, and denoising networks are trained with a regularized loss. Moreover, we propose a training strategy named BayerEnsemble to adapt the Neighbor2Neighbor framework in raw image denoising. The proposed Neighbor2Neighbor framework can enjoy the progress of state-of-the-art supervised denoising networks in network architecture design. It also avoids heavy dependence on the assumption of the noise distribution. We evaluate the Neighbor2Neighbor framework through extensive experiments, including synthetic experiments with different noise distributions and real-world experiments under various scenarios. The code is available online: https://github.com/TaoHuang2018/Neighbor2Neighbor.
近年来，图像去噪从深度神经网络中受益匪浅。然而，这些模型需要大量的噪声-干净图像对进行监督。尽管已经有人尝试仅使用噪声图像来训练去噪网络，但现有的自监督算法存在网络训练效率低下、计算负担重或依赖噪声建模等问题。在本文中，我们提出了一种名为Neighbor2Neighbor的自监督框架用于深度图像去噪。我们给出了理论动机，并证明通过设计特定的采样器从仅有的噪声图像中生成训练图像对，我们可以训练一个类似于在干净图像监督下训练的网络的自监督去噪网络。此外，我们从优化的角度提出了一种正则化方法，以缩小自监督去噪器和监督去噪器之间的优化差距。基于这些理论理解，我们提出了一种非常简单而有效的自监督训练方案：训练图像对由随机邻居子采样器生成，去噪网络使用正则化损失进行训练。此外，我们提出了一种名为BayerEnsemble的训练策略，以使Neighbor2Neighbor框架适用于原始图像去噪。所提出的Neighbor2Neighbor框架可以在网络架构设计中借鉴当前最优的监督去噪网络的进展。它还避免了对噪声分布假设的严重依赖。我们通过广泛的实验对Neighbor2Neighbor框架进行了评估，包括不同噪声分布的合成实验和各种场景下的真实世界实验。代码可在网上获取：https://github.com/TaoHuang2018/Neighbor2Neighbor。
