Wu Xinyi, Wu Zhenyao, Ju Lili, Wang Song
IEEE Trans Pattern Anal Mach Intell. 2023 Jan;45(1):58-72. doi: 10.1109/TPAMI.2021.3138829. Epub 2022 Dec 5.
In this paper, we tackle the problem of semantic segmentation for nighttime images that plays an equally important role as that for daytime images in autonomous driving, but is also much more challenging due to very poor illuminations and scarce annotated datasets. It can be treated as an unsupervised domain adaptation (UDA) problem, i.e., applying other labeled dataset taken in the daytime to guide the network training meanwhile reducing the domain shift, so that the trained model can generalize well to the desired domain of nighttime images. However, current general-purpose UDA approaches are insufficient to address the significant appearance difference between the day and night domains. To overcome such a large domain gap, we propose a novel domain adaptation network "DANIA" for nighttime semantic image segmentation by leveraging a labeled daytime dataset (the source domain) and an unlabeled dataset that contains coarsely aligned day-night image pairs (the target daytime and nighttime domains). These three domains are used to perform a multi-target adaptation via adversarial training in the network. Specifically, for the unlabeled day-night image pairs, we use the pixel-level predictions of static object categories on a daytime image as a pseudo supervision to segment its counterpart nighttime image. We also include a step of image alignment to relieve the inaccuracy caused by the misalignment between day-night image pairs by estimating a flow to refine the pseudo supervision produced by daytime images. Finally, a re-weighting strategy is applied to further improve the predictions, especially boosting the prediction accuracy of small objects. The proposed DANIA is a one-stage adaptation framework for nighttime semantic segmentation, which does not train additional day-night image transfer models as a separate pre-processing stage. Extensive experiments on Dark Zurich and Nighttime Driving datasets show that our DANIA achieves state-of-the-art performance for nighttime semantic segmentation.
在本文中，我们解决夜间图像语义分割问题。该问题在自动驾驶中与白天图像语义分割起着同等重要的作用，但由于光照条件极差且标注数据集稀缺，其挑战性也更大。它可被视为一个无监督域适应（UDA）问题，即应用白天拍摄的其他标注数据集来指导网络训练，同时减少域偏移，以使训练好的模型能很好地推广到所需的夜间图像域。然而，当前的通用UDA方法不足以解决白天和夜间域之间显著的外观差异。为了克服如此大的域差距，我们提出了一种新颖的域适应网络“DANIA”用于夜间语义图像分割，利用一个标注的白天数据集（源域）和一个包含粗略对齐的日夜图像对的未标注数据集（目标白天和夜间域）。这三个域用于在网络中通过对抗训练进行多目标适应。具体而言，对于未标注的日夜图像对，我们将白天图像上静态物体类别的像素级预测用作伪监督来分割其对应的夜间图像。我们还包括图像对齐步骤，通过估计流来细化白天图像产生的伪监督，以减轻日夜图像对之间未对齐导致的不准确。最后，应用一种重新加权策略来进一步改进预测，特别是提高小物体的预测准确性。所提出的DANIA是用于夜间语义分割的单阶段适应框架，它不将额外的日夜图像转移模型作为单独的预处理阶段进行训练。在Dark Zurich和Nighttime Driving数据集上进行的大量实验表明，我们的DANIA在夜间语义分割方面取得了领先的性能。
