Seeing Through Darkness: Visual Localization at Night via Weakly Supervised Learning of Domain Invariant Features

Abstract

Long term visual localization has to conquer the problem of robust matching images with dramatic photometric changes caused by different seasons, weather conditions, natural and man-made illumination changes, etc. Due to the practical demand of applying visual localization at night, either for autonomous driving or for augmented reality related applications, how to extract keypoints and descriptors with robustness to day-night illumination changes on images has became the bottleneck. This paper proposes an adversarial learning based solution to harvest from the weakly domain labels of day and night images, along with the point level correspondences among day time images, to achieve robust local feature extraction and description across day-night images. The key idea is to learn a discriminator to distinguish whether a feature map is generated from the day image or night image, and simultaneously to adjust the parameters of feature extraction network so as to fool the discriminator. After adversarial training of the discriminator and feature extraction network, the feature extraction network finally reaches a stable status so that the extracted feature maps are robust to day-night photometric changes, based on which day-night domain invariant keypoints and descriptors can be extracted. Compared to existing local feature learning methods, it only requires an additional set of easily captured night images to improve the domain invariance of learned features. Experiments on two challenging day-night visual localization benchmarks show the effectiveness of the proposed method. In addition, this paper revisits the widely used image matching metrics on HPatches and finds that recall of different methods reflects their relative localization performance, while previously used mean matching accuracy is not essentially related to downstream tasks like visual localization.