CNN-Based Dense Image Matching for Aerial Remote Sensing Images

Abstract

Dense stereo matching plays a key role in 3D reconstruction. The capability of using deep learning in the stereo matching of remote sensing data is currently uncertain. This article investigated the application of deep learning–based stereo methods in aerial image series and proposed a deep learning–based multi-view dense matching framework. First, we applied three typical convolutional neural network models, MC-CNN, GC-Net, and DispNet, to aerial stereo pairs and compared the results with those of the SGM and a commercial software, SURE. Second, on different data sets, the generalization ability of each network is evaluated by using direct transfer learning with models pretrained on other data sets and by fine-tuning with a small number of target training data. Third, we present a deep learning–based multi-view dense matching framework where the multi-view geometry is introduced to further refine matching results. Three sets of aerial images as the main data sets and two open-source sets of street images as auxiliary data sets are used for testing. Experiments show that, first, the performance of deep learning–based stereo methods is slightly better than traditional methods. Second, both the GC-Net and the MC-CNN have demonstrated good generalization ability and can obtain satisfactory results on aerial images using a pretrained model on several available stereo benchmarks. Third, multi-view geometry constraints can further improve the performance of deep learning–based methods, which is better than that of the multi-view–based SGM and SURE.