Two-view correspondence learning using graph neural network with reciprocal neighbor attention

Abstract

Recent advances in two-view correspondence learning consider aggregating local contextual information from k-nearest neighbors by exploring local geometric extractors using convolution, graph or self-attention. However, there is no guarantee that the extracted local context is conducive to correspondence pruning, since there are typically over 90% outliers in the putative set which would inevitably result in contaminated neighbors and hence contaminated local context. To address this issue, in this paper, we propose the RNA-GNN, an attentional graph neural network (GNN) architecture, which can explicitly model interactions among relevant neighboring correspondences. Specifically, a reciprocal neighbor attention (RNA) module is designed to operate sparse attention on relevant neighbors only rather than all neighbors, mitigating the interference of irrelevant and redundant connectivity, and learning more compact and robust representation. In addition, considering that seeking reliable correspondences requires both local and global context, we introduce a context-aware attention module to extract global and channel-wise context by means of global weighted average pooling with weights that are estimated within the network, excluding outliers from this pooling. Extensive experiments on publicly available datasets demonstrate that RNA-GNN substantially advances current state-of-the-arts on camera pose estimation, homography estimation, and remote sensing image registration. Our code is publicly available at https://github.com/ZizhuoLi/RNA-GNN.