High-quality High-resolution Research Articles

Lightweight image super-resolution with sliding Proxy Attention Network

Recently, image super-resolution (SR) models using window-based Transformers have demonstrated superior performance compared to SR models based on convolutional neural networks. Nevertheless, Transformer-based SR models often entail high computational demands. This is due to the adoption of shifted window self-attention following the window self-attention layer to model long-range relationships, resulting in additional computational overhead. Moreover, extracting local image features only with the self-attention mechanism is insufficient to reconstruct rich high-frequency image content. To overcome these challenges, we propose the Sliding Proxy Attention Network (SPAN), capable of recovering high-quality High-Resolution (HR) images from Low-Resolution (LR) inputs with substantially fewer model parameters and computational operations. The primary innovation of SPAN lies in the Sliding Proxy Transformer Block (SPTB), integrating the local detail sensitivity of convolution with the long-range dependency modeling of self-attention mechanism. Key components within SPTB include the Enhanced Local Feature Extraction Block (ELFEB) and the Sliding Proxy Attention Block (SPAB). ELFEB is designed to enhance the local receptive field with lightweight parameters for high-frequency details compensation. SPAB optimizes computational efficiency by implementing intra-window and cross-window attention in a single operation through leveraging window overlap. Experimental results demonstrate that SPAN can produce high-quality SR images while effectively managing computational complexity. The code is publicly available at: https://github.com/zononhzy/SPAN.

Frame Selection Using Spatiotemporal Dynamics and Key Features as Input Pre-processing for Video Super-Resolution Models

This paper presents a novel approach to video super-resolution (VSR) by focusing on the selection of input frames, a process critical to VSR. VSR methods typically rely on deep learning techniques, those that are able to learn features from a large dataset of low-resolution (LR) and corresponding high-resolution (HR) videos and generate high-quality HR frames from any new LR input frames using the learned features. However, these methods often use as input the immediate neighbouring frames to a given target frame without considering the importance and dynamics of the frames across the temporal dimension of a video. This work aims to address the limitations of the conventional sliding-window mechanisms by developing input frame selection algorithms. By dynamically selecting the most representative neighbouring frames based on content-aware selection measures, our proposed algorithms enable VSR models to extract more informative and accurate features that are better aligned with the target frame, leading to improved performance and higher-quality HR frames. Through an empirical study, we demonstrate that the proposed dynamic content-aware selection mechanism improves super-resolution results without any additional architectural overhead, offering a counter-intuitive yet effective alternative to the long-established trend of increasing architectural complexity to improve VSR results.

KGSR: A kernel guided network for real-world blind super-resolution

In recent years, deep learning-based methods have emerged as dominant players in the field of super-resolution (SR), owing to their exceptional reconstruction performance. The primary driver of their effectiveness lies in their utilization of extensive sets of paired low-resolution and high-resolution images for training deep learning models. This training enables the models to effectively replicate the intricate mapping relationship between low-resolution and high-resolution images. Nevertheless, at present, acquiring a sufficient quantity of such image pairs that satisfy the requirements remains a formidable obstacle. Therefore, in order to break the restriction of limited training sets, self-supervised learning has been introduced to train a model for each low-quality image, without requiring pairwise ground-truths. However, they generally presuppose the generation of low-resolution (LR) images from their high-resolution (HR) counterparts using a pre-defined kernel, such as Bicubic downscaling. Such an assumption is seldom valid for real-world LR images, where degradation processes in practical applications are diverse, intricate, and often undisclosed. Therefore, when the presumed downscaling kernel does not match the actual one, the outcomes of state-of-the-art approaches degrade substantially. In this paper, we introduce KGSR, a kernel-guided network for addressing real-world blind SR, effectively avoiding requiring large training image pairs and transforming the blind image super-resolution problem into a supervised learning and non-blind scenario. Specifically, KGSR trains two networks, namely Upscaling and Downscaling, utilizing only patches extracted from the input test image. On one hand, owing to the cross-scale recurrence property of the SR kernel within a single image, the Downscaling network acquires knowledge of the image-specific degradation process through a generative adversarial network. Consequently, the Downscaling network is capable of generating a downsampled version of the LR test image even when the acquisition process is unknown or less than ideal. Additionally, we employ a dedicated discriminator to compel the Downscaling network to prioritize the characterization of kernel orientations. Conversely, a precise blur kernel has the potential to yield superior performance. Guided by the accurate image-specific SR kernel acquired from the Downscaling network and the downsampled LR input, the Upscaling network is capable of producing a high-quality HR image from the LR input. Within the Upscaling network, we additionally introduce an effective module for harnessing the acquired image-specific SR kernel. KGSR operates as a fully unsupervised approach, yet it can concurrently produce both the image-specific SR kernel and high-quality HR images. Comprehensive experiments conducted on standard benchmarks validate the efficacy of the proposed approach compared to state-of-the-art methodologies. Moreover, the suggested method can deliver visually appealing SR outcomes while exhibiting shorter processing times when applied to real-world LR images.

Accelerating Volumetric CT and MRI Imaging by Reference-Free Deep Learning Transformation from Low-Resolution to High-Resolution

High-resolution (HR) images are important in precision radiation oncology. However, acquiring HR volumetric CT and MRI images is often time consuming; also, the resolution in some direction(s) (e.g., z-direction in the case of CT) is often limited by imaging hardware or fundamental imaging principle. Super-resolution (SR) imaging, i.e., the low-resolution (LR) to HR image transformation, is widely used to improve image resolution. Data-driven deep learning (DL) methods have achieved great success in SR imaging, yet they can hardly be applied to medical imaging as they require large amount of LR-HR image pairs to train the model. We therefore propose a reference-free DL method to increase resolutions of volumetric medical images in an efficient way. We propose a maximum likelihood estimation (MLE)-based implicit neural representation (INR) network for SR imaging. The INR network aims to represent an image as a continuous function parameterized by a coordinate-based multi-layer perceptron. The INR network takes image coordinates as input and outputs corresponding pixel intensities. To train the network without using any HR images, we use a MLE framework to model LR observations' statistics and their relation to the latent HR image. The predicted HR image from the INR's output is transformed to LR images based on the MLE, and the network parameters are then optimized by minimizing the distance between the transformed LR images and actual LR observations. We demonstrate the efficacy of the proposed method on CT and MRI images for 2x, 4x, and 8x SR using only one or two LR image(s). The performance is compared with a conventional SR method named plain MLE, in terms of visual quality and numerical qualities of PSNR and SSIM. Our method outperformed the plain MLE method in the experiment. Table 1 reports the numerical improvements of our method over the compared plain MLE method. For 2x SR with a single LR image, our method achieved significant improvements in both PSNR and SSIM. When using two LR images, the better structural restoration capability of our method became more obvious with higher SR magnifications, as indicated by the increased SSIM differences. Better noise suppression capability of our method is observed in all our studies, as indicated by the PSNR values. In visual quality evaluation, we observed sharper image details with less noise in SR images generated by the proposed method, compared with the plain MLE method. The proposed novel reference-free DL method can efficiently provide high-quality HR images with only one or two LR images for CT and MRI imaging. This method can be easily generalized to many other radiation therapy related applications without the requirement for HR reference images.

A lightweight distillation CNN-transformer architecture for remote sensing image super-resolution

ABSTRACT Remote sensing images exhibit rich texture features and strong autocorrelation. Although the super-resolution (SR) method of remote sensing images based on convolutional neural networks (CNN) can capture rich local information, the limited perceptual field prevents it from establishing long-distance dependence on global information, leading to the low accuracy of remote sensing image reconstruction. Furthermore, it is difficult for existing SR methods to be deployed in mobile devices due to their large network parameters and high computational demand. In this study, we propose a lightweight distillation CNN-Transformer SR architecture, named DCTA, for remote sensing SR, addressing the aforementioned issues. Specifically, the proposed DCTA first extracts the coarse features through the coarse feature extraction layer and then learns the deep features of remote sensing at different scales by fusing the feature distillation extraction module of CNN and Transformer. In addition, we introduce the feature fusion module at the end of the feature distillation extraction module to control the information propagation, aiming to select the informative components for better feature fusion. The extracted low-resolution (LR) feature maps are reorganized through the up-sampling module to obtain high-resolution (HR) feature maps with high accuracy to generate high-quality HR remote sensing images. The experiments comparing different methods demonstrate that the proposed approach performs well on multiple datasets, including NWPU-RESISC45, Draper, and UC Merced. This is achieved by balancing reconstruction performance and network complexity, resulting in both competitive subjective and objective results.

Med-SRNet: GAN-Based Medical Image Super-Resolution via High-Resolution Representation Learning.

High-resolution (HR) medical imaging data provide more anatomical details of human body, which facilitates early-stage disease diagnosis. But it is challenging to get clear HR medical images because of the limiting factors, such as imaging systems, imaging environments, and human factors. This work presents a novel medical image super-resolution (SR) method via high-resolution representation learning based on generative adversarial network (GAN), namely, Med-SRNet. We use GAN as backbone of SR considering the advantages of GAN that can significantly reconstruct the visual quality of the images, and the high-frequency details of the images are more realistic in the image SR task. Furthermore, we employ the HR network (HRNet) in GAN generator to maintain the HR representations and repeatedly use multi-scale fusions to strengthen HR representations for facilitating SR. Moreover, we adopt deconvolution operations to recover high-quality HR representations from all the parallel lower resolution (LR) streams with the aim to yield richer aggregated features, instead of simple bilinear interpolation operations used in HRNetV2. When evaluated on a home-made medical image dataset and two public COVID-19 CT datasets, the proposed Med-SRNet outperforms other leading edge methods, which obtains higher peak signal to noise ratio (PSNR) values and structural similarity (SSIM) values, i.e., maximum improvement of 1.75 and minimum increase of 0.433 on the PSNR metric for “Brain” test sets under 8× and maximum improvement of 0.048 and minimum increase of 0.016 on the SSIM metric for “Lung” test sets under 8× compared with other methods.

Classifying Facial Regions for Face Hallucination

Recently, convolutional neural networks (CNNs) have dominated the face hallucination task due to their powerful feature representation capability. However, most of them simply use the same weights to treat different facial regions without considering the reconstruction difficulty of different facial regions, resulting in the component regions (e.g., eyes, nose, mouth) of the reconstructed faces tending to be blurred. In this paper, we propose a novel facial region classification network (FRCN) to address this problem. The proposed method first divides the input low-resolution (LR) facial image into several patch blocks, then classifies them into three categories according to their reconstruction difficulty, and finally inputs the three types of patch blocks into three networks with different weights for reconstruction and combining, thereby recovering high-quality high-resolution (HR) facial image. Experimental results show that FRCN can remarkably improve face reconstruction's performance.

Adaptive optimal multi-features learning based representation for face hallucination

Face hallucination (FH) is a classical problem to reconstruct a high-resolution (HR) face image for an observed low-resolution (LR) one. The existing methods represent LR facial images though the spatial pixel domain or by combining confined image features with this spatial pixel information. However, the uncertainty in stipulating the optimal proportion for such multiple image features may lead to unexpected results as the optimal proportion for each LR input face image may vary for obtaining the desired HR result. Additionally, they suffer from degraded performance when the observed LR images are contaminated with higher noise. For addressing such problems, this paper proposes an adaptive optimal multi-features proportion learning (OMFPL) scheme, which adopts the Grey Wolf Optimization (GWO) approach for determining the optimum proportion of each feature to represent a particular LR face image. Moreover, an appropriate threshold is applied on different feature samples in the training data for representing the LR patches with their nearest examples. The optimal proportion of these relevant features helps to reconstruct the high-quality HR faces for both noise-free and noisy LR faces. The performance of OMFPL is validated on widely used public databases, real-world images, and surveillance faces, where it achieves the superior results concerning the several competitive state-of-the-art FH methods.

Semantic-Driven Face Hallucination Based on Residual Network

State-of-the-art face hallucination methods utilize convolutional neural networks to restore high-quality high-resolution (HR) face images from low-resolution (LR) face images by employing appearance knowledge. However, most of these solutions only deploy the appearance knowledge (such as face attributes) to modulate the first layer of convolution feature with channel-wise shifting and ignore the face-identity information while emphasizing the vision quality of the generated result. To address the above issues, we propose a semantic-driven residual network based on a generative adversarial network to restore the HR face image with proper identity from the LR face image. Firstly, precise semantic information is explored based on improved U-net. Secondly, the semantic information is concatenated into the residual blocks of the reconstruction module, which is exceptionally efficient in modulating the extracted feature and guiding the generation of HR face images. Finally, the proposed methods can obtain the pleasuring SR face image with a similar identity of LR face image, after adversarially optimizing the designed loss functions. Extensive experiments demonstrate that the proposed semantic-driven residual network performs against the state-of-the-arts both quantitatively and qualitatively.

Image Super-Resolution Using Multi-Scale Space Feature and Deformable Convolutional Network

Recent researches on image super-resolution (SR) have achieved great progressing with the great development of convolutional neural networks (CNNs). However, existing CNNs usually adopt fixed filter structures and the convolutions just rely on the local information contained in the fixed receptive field. Above phenomena prevent high-level convolution layers from encoding semantics over spatial locations and largely limits the learning capacity of CNNs. What’s more, many methods simply used a single-size feature map and failed to consider the spatial information, thereby these results also are unsatisfactory. To address these problems, in this paper, a network with multi-scale space features and deformable convolutional (MulSSD) is presented to further improve the reconstruction accuracy. Specifically, a multi-scale space features compressed block containing the deformable convolutional layer is proposed, which can augment the spatial sampling locations and incorporate the multi-scale space compression features and adaptively adjust the sampling grid and receptive fields. In addition, the design of symmetrical combinations make the information can be smoothly propagated through multiple channels during the training, which effectively improves the training efficiency. Extensive experiments on benchmark datasets validate that the proposed method achieves outperforming quantitative and qualitative performance. And the experimental results also proved that our proposed MulSSD can reconstruct high-quality high-resolution (HR) images at a relatively fast speed and outperform other methods by a large margin.

PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening

This article addresses the problem of remote sensing image pan-sharpening from the perspective of generative adversarial learning. We propose a novel deep neural network-based method named pansharpening GAN (PSGAN). To the best of our knowledge, this is one of the first attempts at producing high-quality pan-sharpened images with generative adversarial networks (GANs). The PSGAN consists of two components: a generative network (i.e., generator) and a discriminative network (i.e., discriminator). The generator is designed to accept panchromatic (PAN) and multispectral (MS) images as inputs and maps them to the desired high-resolution (HR) MS images, and the discriminator implements the adversarial training strategy for generating higher fidelity pan-sharpened images. In this article, we evaluate several architectures and designs, namely, two-stream input, stacking input, batch normalization layer, and attention mechanism to find the optimal solution for pan-sharpening. Extensive experiments on QuickBird, GaoFen-2, and WorldView-2 satellite images demonstrate that the proposed PSGANs not only are effective in generating high-quality HR MS images and superior to state-of-the-art methods but also generalize well to full-scale images.

MRI image synthesis with dual discriminator adversarial learning and difficulty-aware attention mechanism for hippocampal subfields segmentation

Background and objectiveHippocampal subfields (HS) segmentation accuracy on high resolution (HR) MRI images is higher than that on low resolution (LR) MRI images. However, HR MRI data collection is more expensive and time-consuming. Thus, we intend to generate HR MRI images from the corresponding LR MRI images for HS segmentation. Methods and resultsTo generate high-quality HR MRI hippocampus region images, we use a dual discriminator adversarial learning model with difficulty-aware attention mechanism in hippocampus regions (da-GAN). A local discriminator is applied in da-GAN to evaluate the visual quality of hippocampus region voxels of the synthetic images. And the difficulty-aware attention mechanism based on the local discriminator can better model the generation of hard-to-synthesis voxels in hippocampus regions. Additionally, we design a SemiDenseNet model with 3D Dense CRF postprocessing and an Unet-based model to perform HS segmentation. The experiments are implemented on Kulaga-Yoskovitz dataset. Compared with conditional generative adversarial network (c-GAN), the PSNR of generated HR T2w images acquired by our da-GAN achieves 0.406 and 0.347 improvement in left and right hippocampus regions. When using two segmentation models to segment HS, the DSC values achieved on the generated HR T1w and T2w images are both improved than that on LR T1w images. ConclusionExperimental results show that da-GAN model can generate higher-quality MRI images, especially in hippocampus regions, and the generated MRI images can improve HS segmentation accuracy.

Joint Face Super-Resolution and Deblurring Using Generative Adversarial Network

Facial image super-resolution (SR) is an important aspect of facial analysis, and it can contribute significantly to tasks such as face alignment, face recognition, and image-based 3D reconstruction. Recent convolutional neural network (CNN) based models have exhibited significant advancements by learning mapping relations using pairs of low-resolution (LR) and high-resolution (HR) facial images. However, because these methods are conventionally aimed at increasing the PSNR and SSIM metrics, the reconstructed HR images might be blurry and have an overall unsatisfactory perceptual quality even when state-of-the-art quantitative results are achieved. In this study, we address this limitation by proposing an adversarial framework intended to reconstruct perceptually high-quality HR facial images while simultaneously removing blur. To this end, a simple five-layer CNN is employed to extract feature maps from LR facial images, and this feature information is provided to two-branch encoder-decoder networks that generate HR facial images with and without blur. In addition, local and global discriminators are combined to focus on the reconstruction of HR facial structures. Both qualitative and quantitative results demonstrate the effectiveness of the proposed method for generating photorealistic HR facial images from a variety of LR inputs. Moreover, it was also verified, through a use case scenario that the proposed method can contribute more to the field of face recognition than existing approaches.

Deformable Non-Local Network for Video Super-Resolution

The video super-resolution (VSR) task aims to restore a high-resolution (HR) video frame by using its corresponding low-resolution (LR) frame and multiple neighboring frames. At present, many deep learning-based VSR methods rely on optical flow to perform frame alignment. The final recovery results will be greatly affected by the accuracy of optical flow. However, optical flow estimation cannot be completely accurate, and there are always some errors. In this paper, we propose a novel deformable non-local network (DNLN) which is a non-optical-flow-based method. Specifically, we apply the deformable convolution and improve its ability of adaptive alignment at the feature level. Furthermore, we utilize a non-local structure to capture the global correlation between the reference frame and the aligned neighboring frames, and simultaneously enhance desired fine details in the aligned frames. To reconstruct the final high-quality HR video frames, we use residual in residual dense blocks to take full advantage of the hierarchical features. Experimental results on benchmark datasets demonstrate that the proposed DNLN can achieve state-of-the-art performance on VSR task.

New single-image super-resolution reconstruction using MRF model

Abstract Learning-based single-image super-resolution reconstruction (LSISRR) problem using Markov Random Field (LSISRR-MRF) is a robust and integrated framework for both natural and face image super-resolution. The spatial relationships between patches are learned from the training images using the Markov network, and the missing details in the unknown high-resolution (HR) images are predicted using the learned relationship. However, selection of efficient training images with better structural similarity to the input low-resolution (LR) image and appropriate selection of compatibility constraints to measure the correlation between patches affect a lot to the reconstruction results. Besides these, selection of optimal candidate patches for each input LR patch from a pool of training patches is a time-consuming process. To avoid these pitfalls, an improved and fast LSISRR-MRF model is proposed here. A statistical framework, i.e., spatiogram based matching is used to choose efficient training images. Searching of candidate patches is performed in a compact and well-structured search space known as epitomic representation. Smoothness preserving compatibility functions are utilized to well constraint the correlation between patches. Efficacy of the suggested method for generating a high-quality HR image is validated via several experimental analyses on both synthetic and real-time images over some of the state of art methods.

Fast super-resolution algorithm using rotation-invariant ELBP classifier and hierarchical pattern matching

This paper proposes a fast super-resolution (SR) algorithm using content-adaptive two-dimensional (2D) finite impulse response (FIR) filters based on a rotation-invariant classifier. The proposed algorithm consists of a learning stage and an inference stage. In the learning stage, we cluster a sufficient number of low-resolution (LR) and high-resolution (HR) patch pairs into a specific number of groups using the rotation-invariant classifier, and choose a specific number of dominant clusters. Then, we compute the optimal 2D FIR filter(s) to synthesize a high-quality HR patch from an LR patch per cluster, and finally store the patch-adaptive 2D FIR filters in a dictionary. Also, we present a smart hierarchical addressing method for effective dictionary exploration in the inference stage. In the inference stage, the ELBP of each input LR patch is extracted in the same way as the learning stage, and the best matched FIR filter(s) to the input LR patch is found from the dictionary by the hierarchical addressing. Finally, we synthesize the HR patch by using the optimal 2D FIR filter. The experimental results show that the proposed algorithm produces better HR images than the existing SR methods, while providing fast running time.

Learning-based superresolution algorithm using quantized pattern and bimodal postprocessing for text images

This paper proposes a learning-based superresolution algorithm using text characteristics for text images. The proposed algorithm consists of a learning stage and an inference stage. In the learning stage, a sufficient number of low-resolution (LR) to high-resolution (HR) block pairs are first extracted from various LR–HR image pairs that are composed of texts. Then, we classify those block pairs into 512 clusters and, for each cluster, calculate the optimal two-dimensional (2-D) finite impulse response (FIR) filter to synthesize a high-quality HR block from an LR block and store the block-adaptive 2-D FIR filters in a dictionary with their associated index. In the inference stage, we find the best-matched candidate to each input LR block from the dictionary and synthesize the HR block using the optimal 2-D FIR filter. Finally, an HR image is produced via proper postprocessing. Experimental results show that the proposed algorithm provides superior visual quality to images from previous works and outperforms previous processes in terms of computational complexity.

High performance super-resolution reconstruction of multi-frame degraded images with local weighted anisotropy and successive regularization

Multi-frame image super-resolution (SR) is a procedure which takes several observed degraded low-resolution (LR) images, and processes them together to synthesize one or more high-quality high-resolution (HR) images. Due to an insufficient number of LR images and ill-conditioned blur operators, SR image reconstruction is a typical ill-posed inverse problem. However, the regularization approach can be used as an effective means to solve this kind of problems. In this paper, we propose an algorithm for multi-frame SR reconstruction with two new types of regularization terms, termed as local weighted anisotropy regularization and successive regularization toward iteration process, which are characterized by strong ability of suppressing noise and preserving edge information in HR image reconstruction. Furthermore, an iterative refinement procedure obtained superior result by employing Bregman iteration. Experiment results show that the effectiveness of the proposed regularization method and corresponding algorithm for SR reconstruction.

Eigentransformation-based face super-resolution in the wavelet domain

In this paper, we propose a wavelet-based eigentransformation method for human face hallucination. Our algorithm uses the wavelet transform to decompose interpolated low-resolution (LR) images in the wavelet domain to obtain high-frequency information in three different directions, and employs the eigentransformation method to reconstruct the corresponding finer high-frequency content of the high-resolution (HR) images. The low-frequency content of the HR images in the wavelet domain is estimated based on the interpolated images directly. The resulting high-quality HR faces can be synthesized by using the inverse wavelet transform, with all the estimated coefficients. By combining interpolation and eigentransformation, the reconstructed images are less dependent on the training set selected, and can better preserve the low-frequency content. Thus, the reconstructed images look more like the ground-true HR images, as compared to the original eigentransformation method. Experimental results show that our proposed algorithm outperforms the original eigentransformation and other existing methods for face hallucination in terms of both visual quality and objective measurements.