Cho Jaehoon, Kim Seungryong, Sohn Kwanghoon
IEEE Trans Image Process. 2022;31:4090-4103. doi: 10.1109/TIP.2022.3180561. Epub 2022 Jun 20.
This paper addresses the problem of single image de-raining, that is, the task of recovering clean and rain-free background scenes from a single image obscured by a rainy artifact. Although recent advances adopt real-world time-lapse data to overcome the need for paired rain-clean images, they are limited to fully exploit the time-lapse data. The main cause is that, in terms of network architectures, they could not capture long-term rain streak information in the time-lapse data during training owing to the lack of memory components. To address this problem, we propose a novel network architecture combining the time-lapse data and, the memory network that explicitly helps to capture long-term rain streak information. Our network comprises the encoder-decoder networks and a memory network. The features extracted from the encoder are read and updated in the memory network that contains several memory items to store rain streak-aware feature representations. With the read/update operation, the memory network retrieves relevant memory items in terms of the queries, enabling the memory items to represent the various rain streaks included in the time-lapse data. To boost the discriminative power of memory features, we also present a novel background selective whitening (BSW) loss for capturing only rain streak information in the memory network by erasing the background information. Experimental results on standard benchmarks demonstrate the effectiveness and superiority of our approach.
本文探讨单图像去雨问题，即从被雨迹遮挡的单张图像中恢复干净无雨背景场景的任务。尽管最近的进展采用真实世界的延时数据来克服对成对雨 - 干净图像的需求，但它们在充分利用延时数据方面仍存在局限。主要原因在于，就网络架构而言，由于缺乏记忆组件，它们在训练期间无法捕捉延时数据中的长期雨痕信息。为解决此问题，我们提出一种新颖的网络架构，将延时数据与明确有助于捕捉长期雨痕信息的记忆网络相结合。我们的网络由编码器 - 解码器网络和记忆网络组成。从编码器提取的特征在包含多个记忆项以存储雨痕感知特征表示的记忆网络中被读取和更新。通过读取/更新操作，记忆网络根据查询检索相关记忆项，使记忆项能够表示延时数据中包含的各种雨痕。为增强记忆特征的判别能力，我们还提出一种新颖的背景选择性白化（BSW）损失，通过擦除背景信息来仅在记忆网络中捕捉雨痕信息。在标准基准上的实验结果证明了我们方法的有效性和优越性。
