Rainy Night Scene Understanding With Near Scene Semantic Adaptation

Abstract

Deep networks have been used for semantic segmentation tasks on scenes of outdoor environments with increasing popularity. However, the majority of existing work centers on daytime scenes with favorable illumination and weather conditions, and relies on supervision with pixel-level annotations. This paper seeks to address the problem of semantic segmentation for rainy, night-time scenes without using pixel-level annotations. We introduce a near scene semantic approach that uses images of daytime scenes as a bridge for transferring knowledge from pre-trained segmentation models to rainy night images. Specifically, we first present near scene oriented Representation Adaptation (RA) to reduce the domain shift on the representation level. Next, we adapt the segmentation model from the daytime scenario, under varying weather conditions, to the rainy night scenario by using near scene oriented Segmentation Space Adaptation (SSA). Consequently, this further reduces the impact of the domain shift on the segmentation space level. For evaluation, we created a new dataset containing 7000 distinct daytime–night-time image pairs of near scenes obtained by a webcam, and 5266 daytime–rainy night image pairs collected by a car-mounted camera. In addition, we carefully annotated 226 rainy night images with classes defined in Cityscapes. The experimental results clearly demonstrate the advantage of the proposed algorithm.