LifelongGlue: Keypoint matching for 3D reconstruction with continual neural networks

Abstract

Human beings acquire knowledge by a continually learning process. They learn through experience, accumulate knowledge, and employ it to perform the task at hand. The main aim of an artificial intelligence-based system is to incur the ability of continual learning of a human brain. The current artificial intelligence-based autonomous systems perform well on properly regulated, well-adjusted and homogenized data. However, for most state-of-the-art systems, performance is subdued when presented with multiple task-based incremental data. Motivated by the learning of the brain, this paper introduces LifelongGlue, a continual learning neural network for keypoint association between images for 3D reconstruction. 3D reconstruction of a scene from video or sequential images plays a vital role in augmented reality (AR) applications. Keypoint association is crucial to the accurate pose estimation of a scene from multiple views. The present developed methods do not take into account the relation among sequential frames of the video and estimate the keypoints for each pair independently. Our proposed network enhances the expressiveness of local features through continual self and cross attentions, thus, enabling accurate point matching among sequential images by utilizing previously learned knowledge. In comparison to traditional and previous deep learning-based methods, our methodology achieves higher results for pose estimation in challenging indoor and outdoor scenes. The performance of our methodology is validated on multiple datasets. Results demonstrate that the proposed method outperforms state-of-the-art matching approaches while gaining substantial improvement.