Split-Attention Multiframe Alignment Network for Image Restoration

Abstract

Image registration (or image alignment), the problem of aligning multiple images with relative displacement, is a crucial step in many multiframe image restoration algorithms. To solve the problem that most existing image registration approaches can only align two images in one inference, we propose a split-attention multiframe alignment network (SAMANet). Pixel-level displacements between multiple images are first estimated at low-resolution scales and then refined gradually with the increase in feature resolution. To better integrate the interframe information, we present a split-attention module (SAM) and a dot-product attention module (DPAM), which can adaptively rescale the cost volume features and optical flow features according to the similarity between features from different images. The experimental results demonstrate the superiority of our SAMANet over state-of-the-art image registration methods in terms of both accuracy and robustness. To solve the “ghosting effect” caused by pixelwise registration, we designed two “ghost” removal modules: warping repetition detection module (WRDM) and attention fusion module (AFM). WRDM detects “ghost” regions during the image warping process without increasing the time complexity of the registration algorithm. AFM uses an attention mechanism to rescale the aligned images and enables the registration network and the subsequent image restoration networks to be trained jointly. To validate the strengths of the proposed approaches, we apply SAMANet, WRDM and AFM to three image/video restoration tasks. Extensive evaluations demonstrate that the proposed methods can enhance the performance of image restoration algorithms and outperform the other compared registration algorithms.