A multi-view references image super-resolution framework for generating the large-FOV and high-resolution image

Abstract

This paper investigates the generation of large field-of-view (FOV) and high-resolution (HR) panorama images for smartphones. Existing techniques like image stitching struggle to produce satisfactory results due to geometry misalignment and inconsistent appearance. To circumvent the inherent challenges of image stitching methods and generate high-quality panoramas, we treat the image stitching problem as a multiple-reference-based super-resolution problem. Specifically, one large-FOV low-resolution (LR) image and several overlapped small-FOV HR images are taken as inputs, where the LR image acts as a base and multi-view references provide rich HR information. Building on this foundation, a novel multi-view references image super-resolution framework (MVRefSR) is proposed. Within this framework, to address the residual geometric misalignment between the LR-Ref image pairs after coarse alignment, a flow-based RefSR network (FlowSRNet) is proposed, which super-resolves LR patches with corresponding HR references. To facilitate adaptive feature fusion and minimize distortion in structured regions, a fusion weight estimation module and a gradient branch are introduced in FlowSRNet. Finally, the large-FOV HR image is generated by combining these SR patches together. Furthermore, the lack of real-world RefSR datasets for smartphones is addressed by designing an innovative dataset construction pipeline. Extensive experiments demonstrate the superior performance of FlowSRNet and MVRefSR over the compared SR methods and image stitching software, which proves the effectiveness of generating panoramas from a new perspective.