High-resolution Reference Image Research Articles

Learned high-resolution cardiac CT imaging from ultra-high-resolution PCD-CT.

Coronary computed tomography angiography (cCTA) is a widely used non-invasive diagnostic exam for patients with coronary artery disease (CAD). However, most clinical CT scanners are limited in spatial resolution from use of energy-integrating detectors (EIDs). Radiological evaluation of CAD is challenging, as coronary arteries are small (3-4 mm diameter) and calcifications within them are highly attenuating, leading to blooming artifacts. As such, this is a task well suited for high spatial resolution. Recently, photon-counting-detector (PCD) CT became commercially available, allowing for ultra-high resolution (UHR) data acquisition. However, PCD-CTs are costly, restricting widespread accessibility. To address this problem, we propose a super resolution convolutional neural network (CNN): ILUMENATE (Improved LUMEN visualization through Artificial super-resoluTion imagEs), creating a high resolution (HR) image simulating UHR PCD-CT. The network was trained and validated using patches extracted from 8 patients with a modified U-Net architecture. Training input and labels consisted of UHR PCD-CT images reconstructed with a smooth kernel degrading resolution (LR input) and sharp kernel (HR label). The network learned the resolution difference and was tested on 5 unseen LR patients. We evaluated network performance quantitatively and qualitatively through visual inspection, line profiles to assess spatial resolution improvements, ROIs for CT number stability and noise assessment, structural similarity index (SSIM), and percent diameter luminal stenosis. Overall, ILUMENATE improved images quantitatively and qualitatively, creating sharper edges more closely resembling reconstructed HR reference images, maintained stable CT numbers with less than 4% difference, reduced noise by 28%, maintained structural similarity (average SSIM = 0.70), and reduced percent diameter stenosis with respect to input images. ILUMENATE demonstrates potential impact for CAD patient management, improving the quality of LR CT images bringing them closer to UHR PCD-CT images.

Nonreference object-based pansharpening quality assessment

Pansharpening involves the fusion of panchromatic (PAN) and multispectral (MS) images to obtain a high-resolution image with enhanced spectral and spatial information. Assessing the quality of the resulting fused image poses a challenge due to the absence of a high-resolution reference image. Numerous methods have been proposed to address this, from assessing quality at reduced resolution to full-resolution evaluations. Many existing approaches are pixel-based, where quality metrics are applied and averaged on individual pixels. In this article, we introduce a novel object-based method for assessing the quality of pansharpened images at full resolution. In object-based quality assessment methods, the reaction of different areas of the fused image to the fusion process is reflected. Our approach revolves around extracting objects from the given image and evaluating extracted objects. By doing so, the distinct responses of different objects within the fused image to the fusion process are captured. The proposed method leverages a unique object extraction technique known as segmentation by nearest neighbor (SNN) to extract objects of the MS image. This method extracts the objects based on the image’s characteristics without any requirement for parameter tuning. These extracted objects are then mapped onto both PAN and fused images. The proposed spectral index measures the spectral homogeneity of the fused image’s objects and the proposed spatial index measures the injected spatial content from the PAN image to the fused image’s objects. Experimental results underscore the robustness and reliability of the proposed method. Additionally, by visualizing distortion values on object-maps, we gain insights into fusion quality across diverse areas within the scene.

Hyperspectral Image Super Resolution With Real Unaligned RGB Guidance.

Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference (Ref-RGB) image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this article, we explore the fusion-based HSI super-resolution with real Ref-RGB images that have both rigid and nonrigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network (HSIFN) with heterogeneous feature extractions, multistage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multiscale features with an HSI encoder and an RGB encoder, respectively. The features of Ref-RGB images are then processed by a multistage alignment module to explicitly align the features of Ref-RGB with the LR HSI. Finally, the aligned features of Ref-RGB are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned Ref-RGB, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison. The code and dataset are publicly available at https://zeqiang-lai.github.io/HSI-RefSR/.

AEFormer: Zoom Camera Enables Remote Sensing Super-Resolution via Aligned and Enhanced Attention

Reference-based super-resolution (RefSR) has achieved remarkable progress and shows promising potential applications in the field of remote sensing. However, previous studies heavily rely on existing and high-resolution reference image (Ref), which is hard to obtain in remote sensing practice. To address this issue, a novel structure based on a zoom camera structure (ZCS) together with a novel RefSR network, namely AEFormer, is proposed. The proposed ZCS provides a more accessible way to obtain valid Ref than traditional fixed-length camera imaging or external datasets. The physics-enabled network, AEFormer, is proposed to super-resolve low-resolution images (LR). With reasonably aligned and enhanced attention, AEFormer alleviates the misalignment problem, which is challenging yet common in RefSR tasks. Herein, it contributes to maximizing the utilization of spatial information across the whole image and better fusion between Ref and LR. Extensive experimental results on benchmark dataset RRSSRD and real-world prototype data both verify the effectiveness of the proposed method. Hopefully, ZCS and AEFormer can enlighten a new model for future remote sensing imagery super-resolution.

Super-resolution biomedical imaging via reference-free statistical implicit neural representation

Objective. Supervised deep learning for image super-resolution (SR) has limitations in biomedical imaging due to the lack of large amounts of low- and high-resolution image pairs for model training. In this work, we propose a reference-free statistical implicit neural representation (INR) framework, which needs only a single or a few observed low-resolution (LR) image(s), to generate high-quality SR images. Approach. The framework models the statistics of the observed LR images via maximum likelihood estimation and trains the INR network to represent the latent high-resolution (HR) image as a continuous function in the spatial domain. The INR network is constructed as a coordinate-based multi-layer perceptron, whose inputs are image spatial coordinates and outputs are corresponding pixel intensities. The trained INR not only constrains functional smoothness but also allows an arbitrary scale in SR imaging. Main results. We demonstrate the efficacy of the proposed framework on various biomedical images, including computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, and ultrasound images, across different SR magnification scales of 2×, 4×, and 8×. A limited number of LR images were used for each of the SR imaging tasks to show the potential of the proposed statistical INR framework. Significance. The proposed method provides an urgently needed unsupervised deep learning framework for numerous biomedical SR applications that lack HR reference images.

Accelerating Volumetric CT and MRI Imaging by Reference-Free Deep Learning Transformation from Low-Resolution to High-Resolution

High-resolution (HR) images are important in precision radiation oncology. However, acquiring HR volumetric CT and MRI images is often time consuming; also, the resolution in some direction(s) (e.g., z-direction in the case of CT) is often limited by imaging hardware or fundamental imaging principle. Super-resolution (SR) imaging, i.e., the low-resolution (LR) to HR image transformation, is widely used to improve image resolution. Data-driven deep learning (DL) methods have achieved great success in SR imaging, yet they can hardly be applied to medical imaging as they require large amount of LR-HR image pairs to train the model. We therefore propose a reference-free DL method to increase resolutions of volumetric medical images in an efficient way. We propose a maximum likelihood estimation (MLE)-based implicit neural representation (INR) network for SR imaging. The INR network aims to represent an image as a continuous function parameterized by a coordinate-based multi-layer perceptron. The INR network takes image coordinates as input and outputs corresponding pixel intensities. To train the network without using any HR images, we use a MLE framework to model LR observations' statistics and their relation to the latent HR image. The predicted HR image from the INR's output is transformed to LR images based on the MLE, and the network parameters are then optimized by minimizing the distance between the transformed LR images and actual LR observations. We demonstrate the efficacy of the proposed method on CT and MRI images for 2x, 4x, and 8x SR using only one or two LR image(s). The performance is compared with a conventional SR method named plain MLE, in terms of visual quality and numerical qualities of PSNR and SSIM. Our method outperformed the plain MLE method in the experiment. Table 1 reports the numerical improvements of our method over the compared plain MLE method. For 2x SR with a single LR image, our method achieved significant improvements in both PSNR and SSIM. When using two LR images, the better structural restoration capability of our method became more obvious with higher SR magnifications, as indicated by the increased SSIM differences. Better noise suppression capability of our method is observed in all our studies, as indicated by the PSNR values. In visual quality evaluation, we observed sharper image details with less noise in SR images generated by the proposed method, compared with the plain MLE method. The proposed novel reference-free DL method can efficiently provide high-quality HR images with only one or two LR images for CT and MRI imaging. This method can be easily generalized to many other radiation therapy related applications without the requirement for HR reference images.

Double-Layer Search and Adaptive Pooling Fusion for Reference-Based Image Super-Resolution

Reference-based image super-resolution (RefSR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) images by introducing HR reference images. The key step of RefSR is to transfer reference features to LR features. However, existing methods still lack an efficient transfer mechanism, resulting in blurry details in the generated image. In this article, we propose a double-layer search module and an adaptive pooling fusion module group for reference-based image super-resolution, called DLASR. Based on the re-search strategy, the double-layer search module can produce an accurate index map and score map. These two maps are used to filter out accurate reference features, which greatly increases the efficiency of feature transfer in the later stage. Through two continuous feature-enhancement steps, the adaptive pooling fusion module group can transfer more valuable reference features to the corresponding LR features. In addition, a structure reconstruction module is proposed to recover the geometric information of the images, which further improves the visual quality of the generated image. We conduct comparative experiments on a variety of datasets, and the results prove that DLASR achieves significant improvements over other state-of-the-art methods, in terms of quantitative accuracy and qualitative visual effect. The code is available at https://github.com/clttyou/DLASR.

Image quality enhancement of 4D light field microscopy via reference impge propagation-based one-shot learning

Four-dimensional (4D) light-field (LF) microscopes can acquire 3D information about target objects using a microlens array (MLA). However, the resolution and quality of sub-images in the LF images are reduced because of the spatial multiplexing of rays by the element lenses of the MLA. To overcome these limitations, this study proposes an LF one-shot learning technique that can convert LF sub-images into high-quality images similar to the 2D images of conventional optical microscopes obtained without any external training datasets for image enhancement. The proposed convolutional neural network model was trained using only one training dataset comprising a high-resolution reference image captured without an MLA as the ground truth. Further, its input was the central view of the LF image. After LF one-shot learning, the trained model should be able to convert well the other LF sub-images of various directional views that were not used in the main training process. Therefore, novel learning techniques were designed for LF one-shot learning. These novel techniques include an autoencoder-based model initialization method, a feature map-based learning algorithm to prevent the overfitting of the model, and cut loss to prevent saturation. The experimental results verified that the proposed technique effectively enhances the LF image quality and resolution using a reference image. Moreover, this method enhances the resolution by up to 13 times, decreases the noise amplification effect, and restores the lost details of microscopic objects. The proposed technique is stable and yields superior experimental results compared with those of the existing resolution-enhancing methods.

Semi-supervised super-resolution of diffusion-weighted images based on multiple references.

Spatial resolution of diffusion tensor images is usually compromised to accelerate the acquisitions, and the state-of-the-art (SOTA) image super-resolution (SR) reconstruction methods are commonly based on supervised learning models. Considering that matched low-resolution (LR) and high-resolution (HR) diffusion-weighted (DW) image pairs are not readily available, we propose a semi-supervised DW image SR reconstruction method based on multiple references (MRSR) extracted from other subjects. In MRSR, the prior information of multiple HR reference images is migrated into a residual-like network to assist SR reconstruction of DW images, and a CycleGAN-based semi-supervised strategy is used to train the network with 30% matched and 70% unmatched LR-HR image pairs. We evaluate the performance of the MRSR by comparing against SOTA methods on an HCP dataset in terms of the quality of reconstructed DW images and diffusion metrics. MRSR achieves the best performance, with the mean PSNR/SSIM of DW images being improved by at least 14.3%/28.8% and 1%/1.4% respectively relative to SOTA unsupervised and supervised learning methods, and with the fiber orientations deviating from the ground truth by about 6.28° on average, the RMSEs of fractional anisotropy, mean diffusivity, axial diffusivity and radial diffusivity being 3.0%, 4.6%, 5.7% and 4.5% respectively relative to the ground truth. We validate the effectiveness of the proposed network structure, multiple-reference and CycleGAN-based semi-supervised learning strategies for SR reconstruction of diffusion tensor images through the ablation studies. The proposed method allows us to achieve SR reconstruction for diffusion tensor images with a limited number of matched image pairs.

WDBSTF: A Weighted Dual-Branch Spatiotemporal Fusion Network Based on Complementarity between Super-Resolution and Change Prediction

Spatiotemporal fusion (STF) is a solution to generate satellite images with both high-spatial and high-temporal resolutions. The deep learning-based STF algorithms focus on spatial dimensions to build a super-resolution (SR) model or the temporal dimensions to build a change prediction (CP) model, or the task itself to build a data-driven end-to-end model. The multi-source images used for STF usually have large spatial scale gaps and temporal spans. The large spatial scale gaps lead to poor spatial details based on a SR model; the large temporal spans make it difficult to accurately reconstruct changing areas based on a CP model. We propose a weighted dual-branch spatiotemporal fusion network based on complementarity between super-resolution and change prediction (WDBSTF), which includes the SR branch and CP branch, and a weight module representing the complementarity of the two branches. The SR branch makes full use of edge information and high-resolution reference images to obtain high-quality spatial features for image reconstruction. The CP branch decomposes complex problems via a two-layer cascaded network, changes features from the difference image, and selects high-quality spatial features through the attention mechanism. The fusion result of the CP branch has rich image details, but the fusion accuracy in the changing area is low due to the lack of detail. The SR branch has consistent and excellent fusion performances in the changing and no-changing areas, but the image details are not rich enough compared with the CP branch due to the large amplification factor. Next, a weighted network was designed to combine the advantages of the two branches to produce improved fusion results. We evaluated the performance of the WDBSTF in three representative scenarios, and both visual and quantitative evaluations demonstrate the state-of-the-art performance of our algorithm. (On the LGC dataset, our method outperforms the suboptimal method by 2.577% on SSIM. On the AHB dataset, our method outperforms the suboptimal method by 1.684% on SSIM. On the CIA dataset, our method outperforms the suboptimal method by 5.55% on SAM).

Ref‐ZSSR: Zero‐Shot Single Image Superresolution with Reference Image

AbstractSingle image superresolution (SISR) has achieved substantial progress based on deep learning. Many SISR methods acquire pairs of low‐resolution (LR) images from their corresponding high‐resolution (HR) counterparts. Being unsupervised, this kind of method also demands large‐scale training data. However, these paired images and a large amount of training data are difficult to obtain. Recently, several internal, learning‐based methods have been introduced to address this issue. Although requiring a large quantity of training data pairs is solved, the ability to improve the image resolution is limited if only the information of the LR image itself is applied. Therefore, we further expand this kind of approach by using similar HR reference images as prior knowledge to assist the single input image. In this paper, we proposed zero‐shot single image superresolution with a reference image (Ref‐ZSSR). First, we use an unconditional generative model to learn the internal distribution of the HR reference image. Second, a dual‐path architecture that contains a downsampler and an upsampler is introduced to learn the mapping between the input image and its downscaled image. Finally, we combine the reference image learning module and dual‐path architecture module to train a new generative model that can generate a superresolution (SR) image with the details of the HR reference image. Such a design encourages a simple and accurate way to transfer relevant textures from the reference high‐definition (HD) image to LR image. Compared with using only the image itself, the HD feature of the reference image improves the SR performance. In the experiment, we show that the proposed method outperforms previous image‐specific network and internal learning‐based methods.

Dual Projection Fusion for Reference-Based Image Super-Resolution.

Reference-based image super-resolution (RefSR) methods have achieved performance superior to that of single image super-resolution (SISR) methods by transferring texture details from an additional high-resolution (HR) reference image to the low-resolution (LR) image. However, existing RefSR methods simply add or concatenate the transferred texture feature with the LR features, which cannot effectively fuse the information of these two independently extracted features. Therefore, this paper proposes a dual projection fusion for reference-based image super-resolution (DPFSR), which enables the network to focus more on the different information between feature sources through inter-residual projection operations, ensuring effective filling of detailed information in the LR feature. Moreover, this paper also proposes a novel backbone called the deep channel attention connection network (DCACN), which is capable of extracting valuable high-frequency components from the LR space to further facilitate the effectiveness of image reconstruction. Experimental results show that we achieve the best peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) performance compared with the state-of-the-art (SOTA) SISR and RefSR methods. Visual results demonstrate that the proposed method in this paper recovers more natural and realistic texture details.

Online Learning for Reference-Based Super-Resolution

Online learning is a method for exploiting input data to update deep networks in the test stage to derive potential performance improvement. Existing online learning methods for single-image super-resolution (SISR) utilize an input low-resolution (LR) image for the online adaptation of deep networks. Unlike SISR approaches, reference-based super-resolution (RefSR) algorithms benefit from an additional high-resolution (HR) reference image containing plenty of useful features for enhancing the input LR image. Therefore, we introduce a new online learning algorithm, using several reference images, which is applicable to not only RefSR but also SISR networks. Experimental results show that our online learning method is seamlessly applicable to many existing RefSR and SISR models, and that improves performance. We further present the robustness of our method to non-bicubic degradation kernels with in-depth analyses.

Reference-based Image and Video Super-Resolution via C2-Matching.

Reference-based Super-Resolution (Ref-SR) has recently emerged as a promising paradigm to enhance a low-resolution (LR) input image or video by introducing an additional high-resolution (HR) reference image. Existing Ref-SR methods mostly rely on implicit correspondence matching to borrow HR textures from reference images to compensate for the information loss in input images. However, performing local transfer is difficult because of two gaps between input and reference images: the transformation gap (e.g. scale and rotation) and the resolution gap (e.g. HR and LR). To tackle these challenges, we propose C2-Matching in this work, which performs explicit robust matching crossing transformation and resolution. 1) To bridge the transformation gap, we propose a contrastive correspondence network, which learns transformation-robust correspondences using augmented views of the input image. 2) To address the resolution gap, we adopt teacher-student correlation distillation, which distills knowledge from the easier HR-HR matching to guide the more ambiguous LR-HR matching. 3) Finally, we design a dynamic aggregation module to address the potential misalignment issue between input images and reference images. In addition, to faithfully evaluate the performance of Reference-based Image Super-Resolution (Ref Image SR) under a realistic setting, we contribute the Webly-Referenced SR (WR-SR) dataset, mimicking the practical usage scenario. We also extend C2-Matching to Reference-based Video Super-Resolution (Ref VSR) task, where an image taken in a similar scene serves as the HR reference image. Extensive experiments demonstrate that our proposed C2-Matching significantly outperforms state of the arts by up to 0.7dB on the standard CUFED5 benchmark and also boosts the performance of video super-resolution by incorporating the C2-Matching component into Video SR pipelines. Notably, C2-Matching also shows great generalizability on WR-SR dataset as well as robustness across large scale and rotation transformations. Codes and datasets are available at https://github.com/yumingj/C2-Matching.

Learning Discrete Representations From Reference Images for Large Scale Factor Image Super-Resolution.

Image super-resolution (SR) task aims to recover high-resolution (HR) images from degraded low-resolution (LR) images, which has achieved great progress due to the recent advances of deep neural networks. Due to severe information loss of the LR images, it is more challenging to reconstruct high quality HR images at large scale factors, i. e., higher than 4× . Traditional reference image based SR methods usually perform patch matching to locate detailed texture from HR reference images which could provide fine details from similar image contents. But it suffers from difficulties in achieving good matching in the largely downscaled image space or feature space due to the ill-posed nature between LR and HR mapping. In this paper, we tackle this problem by exploiting fine details contained in reference HR images. Inspired by vector quantization (VQ), we propose a simple yet effective auto-encoder convolutional neural network (CNN) module to learn discrete representations of images. Furthermore, we propose to progressively learn pairs of cross-scale discrete feature representations using paired LR and HR reference images. The coarser scale of the discrete representation is responsible for encoding the global image structure while the paired finer scale of the discrete representation takes charge of capturing missing details in the finer image scale. During inference, continuous features of the test LR image are used as queries to retrieve finer scale discrete representations (value) by searching the nearest coarser scale discrete representations (key). Then, the queries and retrieved values are combined to progressively recover the HR image. Experimental results indicate that when compared with the state-of-the-art image SR models, the proposed method can achieve advanced performance in terms of both objective quality and subjective quality. The code will be available on URL: https://github.com/sunwj/refsr.

DSMA: Reference-Based Image Super-Resolution Method Based on Dual-View Supervised Learning and Multi-Attention Mechanism

Reference image based super-resolution methods (RefSR) have made rapid and remarkable progress in the field of image super-resolution (SR) in recent years by introducing additional high-resolution (HR) images to enhance the recovery of low-resolution (LR) images. The existing RefSR methods can rely on implicit correspondence matching to transfer the HR texture from the reference image (Ref) to compensate for the information loss in the input image. However, the differences between low-resolution input images and high-resolution reference images still affects the effective utilization of Ref images, so it is an important challenge to make full use of the information in Ref images to improve the SR performance. In this paper, we propose an image super-resolution method based on dual-view supervised learning and multi-attention mechanism (DSMA). It enhances the learning of important detail features of Ref images and weakens the interference of noisy information by introducing the multi-attention mechanism, while employing dual-view supervision to motivate the network to learn more accurate feature representations. Quantitative and qualitative experiments on these benchmarks, i.e., CUFED5, Urban100 and Manga109, show that DSMA outperforms the state-of-the-art baselines with significant improvements.

Undersampled magnetic resonance image reconstruction based on support prior and deep image prior without pre-training

Magnetic resonance imaging (MRI) method based on deep learning needs large-quantity and high-quality patient-based datasets for pre-training. However, this is a challenge to the clinical applications because it is difficult to obtain a sufficient quantity of patient-based MR datasets due to the limitation of equipment and patient privacy concerns. In this paper, we propose a novel undersampled MRI reconstruction method based on deep learning. This method does not require any pre-training procedures and does not depend on training datasets. The proposed method is inspired by the traditional deep image prior (DIP) framework, and integrates the structure prior and support prior of the target MR image to improve the efficiency of learning. Based on the similarity between the reference image and the target image, the high-resolution reference image obtained in advance is used as the network input, thereby incorporating the structural prior information into network. By taking the coefficient index set of the reference image with large amplitude in the wavelet domain as the known support of the target image, the regularization constraint term is constructed, and the network training is transformed into the optimization process of network parameters. Experimental results show that the proposed method can obtain more accurate reconstructions from undersampled <i>k</i>-space data, and has obvious advantages in preserving tissue features and detailed texture.

Pixelwise Dynamic Convolution Neural Network for LiDAR Depth Data Interpolation

A light detecting and ranging (LiDAR) system is an important means that takes an omni-directional view to collect precise surrounding 3D information in high sampling frequency. However, due to the architecture of a LiDAR sensor, LiDAR data typically contains much less information in the vertical direction compared to the horizontal direction. In addition, LiDAR data are often defected during measurement. There have been previous researches about LiDAR data interpolation using a high-resolution reference image. However, they suffer from the calibration error due to vibration or shock during operation and do not consider defected data. Here, we propose a reference-free interpolation method with considering the defected LiDAR data using a two-step network architecture. Our proposed method is designed first to exploit the rich information in the horizontal direction of the LiDAR data to restore the defected data and then increase the vertical resolution of LiDAR data. Since the proposed interpolation method is a reference-free approach, it if free from the calibration issue. We confirm that our proposed method shows better performance than existing interpolation methods and is effective as a pre-processing function in other applications. For example, our proposed method can improve the performance of the human segmentation. We also propose a pixelwise dynamic convolution layer as a component of the proposed interpolation method. The proposed dynamic convolution layer adaptively processes each pixel with variable kernels, while existing dynamic layers only use fixed kernels. Experimental results show that the proposed dynamic layer is more effective than existing dynamic layers for LiDAR data interpolation.

Reference guided image super-resolution via efficient dense warping and adaptive fusion

Due to the limited improvement of single-image based super-resolution (SR) methods in recent years, the reference based image SR (RefSR) methods, which super-resolve the low-resolution (LR) input with the guidance of similar high-resolution (HR) reference images are emerging. There are two main challenges in RefSR, i.e. reference image warping and exploring the guidance information from the warped references. For reference warping, we propose an efficient dense warping method to deal with large displacements, which is much faster than traditional patch (or texture) matching strategy. For the SR process, since different reference images complement each other, and have different similarities with the LR image, we further propose a similarity based feature fusion strategy to take advantage of the most similar reference regions. The SR process is realized by an encoder–decoder network and trained with pixel-level reconstruction loss, degradation loss and feature-level perceptual loss. Extensive experiments on three benchmark datasets demonstrate that the proposed method outperforms state-of-the-art SR methods in both subjective and objective measurements.

Mitigation of truncation effects in elongated Shack-Hartmann laser guide star wavefront sensor images.

Laser guide star Shack-Hartmann wavefront sensor images on extremely large telescopes (ELT) will be significantly elongated due to the off-axis projection of the laser relative to the subapertures. The finite number of pixels of the wavefront sensor detector means the most elongated images will be truncated, introducing errors in the centroid measurements. In this paper, we propose appending to the truncated wavefront sensor image the most likely missing tails from a high-resolution nontruncated reference image, which can be calculated from all of the low-resolution images. We show, via numerical simulation, that we can improve the centroid estimate for the most elongated subapertures on an ELT in the presence of read and photon noise.