CNN-based Super-resolution Methods Research Articles

Learning knowledge representation with meta knowledge distillation for single image super-resolution

Although the deep CNN-based super-resolution methods have achieved outstanding performance, their memory cost and computational complexity severely limit their practical employment. Knowledge distillation (KD), which can efficiently transfer knowledge from a cumbersome network (teacher) to a compact network (student), has demonstrated its advantages in some computer vision applications. The representation of knowledge is vital for knowledge transferring and student learning, which is generally defined in hand-crafted manners or uses the intermediate features directly. In this paper, we propose a model-agnostic meta knowledge distillation method under the teacher–student architecture for the single image super-resolution task. It provides a more flexible and accurate way to help teachers transmit knowledge in accordance with the abilities of students via knowledge representation networks (KRNets) with learnable parameters. Specifically, the texture-aware dynamic kernels are generated from local information to decompose the distillation problem into texture-wise supervision for further promoting the recovery quality of high-frequency details. In addition, the KRNets are optimized in a meta-learning manner to ensure the knowledge transferring and the student learning are beneficial to improving the reconstructed quality of the student. Experiments conducted on various single image super-resolution datasets demonstrate that our proposed method outperforms existing defined knowledge representation-related distillation methods and can help super-resolution algorithms achieve better reconstruction quality without introducing any extra inference complexity.

Deep learning architecture for 3D image super-resolution of late gadolinium enhanced cardiac MRI.

High-resolution late gadolinium enhanced (LGE) cardiac magnetic resonance imaging (MRI) volumes are difficult to acquire due to the limitations of the maximal breath-hold time achievable by the patient. This results in anisotropic 3D volumes of the heart with high in-plane resolution, but low-through-plane resolution. Thus, we propose a 3D convolutional neural network (CNN) approach to improve the through-plane resolution of the cardiac LGE-MRI volumes. We present a 3D CNN-based framework with two branches: a super-resolution branch to learn the mapping between low-resolution and high-resolution LGE-MRI volumes, and a gradient branch that learns the mapping between the gradient map of low-resolution LGE-MRI volumes and the gradient map of high-resolution LGE-MRI volumes. The gradient branch provides structural guidance to the CNN-based super-resolution framework. To assess the performance of the proposed CNN-based framework, we train two CNN models with and without gradient guidance, namely, dense deep back-projection network (DBPN) and enhanced deep super-resolution network. We train and evaluate our method on the 2018 atrial segmentation challenge dataset. Additionally, we also evaluate these trained models on the left atrial and scar quantification and segmentation challenge 2022 dataset to assess their generalization ability. Finally, we investigate the effect of the proposed CNN-based super-resolution framework on the 3D segmentation of the left atrium (LA) from these cardiac LGE-MRI image volumes. Experimental results demonstrate that our proposed CNN method with gradient guidance consistently outperforms bicubic interpolation and the CNN models without gradient guidance. Furthermore, the segmentation results, evaluated using Dice score, obtained using the super-resolved images generated by our proposed method are superior to the segmentation results obtained using the images generated by bicubic interpolation () and the CNN models without gradient guidance (). The presented CNN-based super-resolution method with gradient guidance improves the through-plane resolution of the LGE-MRI volumes and the structure guidance provided by the gradient branch can be useful to aid the 3D segmentation of cardiac chambers, such as LA, from the 3D LGE-MRI images.

A Study on the Super Resolution Combining Spatial Attention and Channel Attention

Existing CNN-based super resolution methods have low emphasis on high-frequency features, resulting in poor performance for contours and textures. To solve this problem, this paper proposes single image super resolution using an attention mechanism that emphasizes high-frequency features and a feature extraction process with different depths. In order to emphasize the high-frequency features of the channel and space, it is composed of CSBlock that combines channel attention and spatial attention. Attention block using 10 CSBlocks was used for high-frequency feature extraction. In order to extract various features with different degrees of feature emphasis from insufficient low-resolution features, features were extracted from structures connected with different numbers of attention blocks. The extracted features were expanded through sub-pixel convolution to create super resolution images, and learning was performed through L1 loss. Compared to the existing deep learning method, it showed improved results in several high-frequency features such as small object outlines and line patterns. In PSNR and SSIM, it showed about 11% to 26% improvement over the existing Bicubic interpolation and about 1 to 2% improvement over VDSR and EDSR.

An Enhanced Residual Feature Fusion Network Integrated with a Terrain Weight Module for Digital Elevation Model Super-Resolution

The scale of digital elevation models (DEMs) is vital for terrain analysis, surface simulation, and other geographic applications. Compared to traditional super-resolution (SR) methods, deep convolutional neural networks (CNNs) have shown great success in DEM SR. However, in terms of these CNN-based SR methods, the features extracted by the stackable residual modules cannot be fully utilized as the depth of the network increases. Therefore, our study proposes an enhanced residual feature fusion network (ERFFN) for DEM SR. The designed residual fusion module groups four residual modules to make better use of the local residual features. Meanwhile, the residual structure is refined by inserting a lightweight enhanced spatial residual attention module into each basic residual block to further strengthen the efficiency of the network. Considering the continuity of terrain features, terrain weight modules are integrated into the loss module. Based on two large-scale datasets, our ERFFN shows a 10–20% reduction in the mean absolute error and the lowest error in terrain features, such as slope, demonstrating the superiority of an ERFFN-based DEM SR over state-of-the-art methods. Finally, to demonstrate potential value in real-world applications, we deploy the ERFFN to reconstruct a large geographic area covering 44,000 km2 which contains missing parts.

MLWAN: Multi-Scale Learning Wavelet Attention Module Network for Image Super Resolution

Image super resolution (SR) is an important image processing technique in computer vision to improve the resolution of images and videos. In recent years, deep convolutional neural network (CNN) has made significant progress in the field of image SR; however, the existing CNN-based SR methods cannot fully search for background information in the measurement of feature extraction. In addition, in most cases, different scale factors of image SR are assumed to be different assignments and completed by training different models, which does not meet the actual application requirements. To solve these problems, we propose a multi-scale learning wavelet attention network (MLWAN) model for image SR. Specifically, the proposed model consists of three parts. In the first part, low-level features are extracted from the input image through two convolutional layers, and then a new channel-spatial attention mechanism (CSAM) block is concatenated. In the second part, CNN is used to predict the highest-level low-frequency wavelet coefficients, and the third part uses recursive neural networks (RNN) with different scales to predict the wavelet coefficients of the remaining subbands. In order to further achieve lightweight, an effective channel attention recurrent module (ECARM) is proposed to reduce network parameters. Finally, the inverse discrete wavelet transform (IDWT) is used to reconstruct HR image. Experimental results on public large-scale datasets demonstrate the superiority of the proposed model in terms of quantitative indicators and visual effects.

Multi-Attention Residual Network for Image Super Resolution

Recently, many studies have shown that deep convolutional neural network can achieve superior performance in image super resolution (SR). The majority of current CNN-based SR methods tend to use deeper architecture to get excellent performance. However, with the growing depth and width of network, the hierarchical features from low-resolution (LR) images cannot be exploited effectively. On the other hand, most models lack the ability of discriminating different types of information and treating them equally, which results in limiting the representational capacity of the models. In this study, we propose the multi-attention residual network (MARN) to address these problems. Specifically, we propose a new multi-attention residual block (MARB), which is composed of attention mechanism and multi-scale residual network. At the beginning of each residual block, the channel importance of image features is adaptively recalibrated by attention mechanism. Then, we utilize convolutional kernels of different sizes to adaptively extract the multi-attention features on different scales. At the end of blocks, local multi-attention features fusion is applied to get more effective hierarchical features. After obtaining the outputs of each MARB, global hierarchical feature fusion jointly fuses all hierarchical features for reconstructing images. Our extensive experiments show that our model outperforms most of the state-of-the-art methods.

Dense channel splitting network for MR image super-resolution

High spatial resolution is desirable in magnetic resonance imaging (MRI) as it can provide detailed anatomical information, facilitating radiologists with accurate quantitative analysis. Super-resolution (SR) algorithms are effective approaches to enhance MR images' spatial resolution. In the past few years, convolutional neural network (CNN)-based SR methods have significantly improved and outperformed conventional ones. However, existing CNN-based SR methods usually do not explicitly consider the frequency property of images, leading to the limited representation of high-frequency components reflecting image details. To alleviate this problem, a dense channel splitting network (DCSN) algorithm is proposed to process the frequency bands for better feature detection. Specifically, a channel splitting module, a cascaded multi-branch dilation module, and a dense-in and recursive-out mechanism are designed to separate frequency bands of MR images and forward the high-frequency information to deeper layers for reconstruction. Several experiments are performed on real T2 brain and PD (proton density) knee images. The results indicate that the proposed network is superior to conventional CNN-based SR methods.

A Cost-Effective Interpolation for Multi-Magnification Super-Resolution

Super-Resolution (SR) was an important research topic, and SR methods based on Convolutional Neural Network (CNN) confirmed its groundbreaking performance. However, notably implementing the CNN model into resource-limited hardware devices is a great challenge. Therefore, we present a hardware-friendly and low-cost interpolation for Multi-Magnification SR image reconstruction. We follow our previous work, which is a learning-based interpolation (LCDI) with a self-defined classifier of image texture, and extend its original x2 architecture to x3 and x4 architecture. Besides, the required pre-trained weights are reduced by the fusion scheme. Experimentally, the proposed method has only 75% lower pre-trained weights than LCDI. Compared to the related work OLM-SI (One linear learning mapping-SI), the run-time and quantity of pre-trained weights of the x2 proposed method are at least 90% lower. Compared to CNN-based SR methods, the proposed method loses a little lower performance, but the evaluation of computational cost is much lower. In conclusion, the proposed method is cost-effective and a practical solution for resource-limited hardware and device.

Deep Learning Approaches to Spatial Downscaling of GRACE Terrestrial Water Storage Products Using EALCO Model Over Canada

Estimating terrestrial water storage (TWS) with high spatial resolution is crucial for hydrological and water resource management. Comparing to traditional in-situ data measurement, observation from space borne sensor such as Gravity Recovery and Climate Experiment (GRACE) satellites is quite effective to obtain a large-scale TWS data. However, the coarse resolution of the GRACE data restricts its application at a local scale. This paper presents three novel convolutional neural network (CNN) based approaches including the Super-Resolution CNN (SRCNN), the Very Deep Super-Resolution (VDSR), and the Residual Channel Attention Networks (RCAN) to spatial downscaling of the monthly GRACE TWS products using the outputs of the Ecological Assimilation of Land and Climate Observations (EALCO) model over Canada. We also compare the performance of CNN-based methods with the empirical linear regression-based downscaling method. All comparison results were evaluated by root mean square error (RMSE) between the reconstructed GRACE TWS and the original one. RMSEs over the matched pixels are 22.3, 14.4, 18.4 and 71.6 mm of SRCNN, VDSR, RCAN and linear regression-based method respectively. Obviously, VDSR shows the best accuracy among all methods. The result shows all CNN-based super resolution methods preform much better than traditional method in spatial downscaling.

Super-resolution of remotely sensed data using channel attention based deep learning approach

ABSTRACT Remote Sensing image super-resolution aims to improve the spectral and/or spatial resolution of the satellite imageries. In order to improve the performance of the CNN-based super-resolution methods, increasing the depth of the network is commonly used. However, this increases computational complexity and training difficulties only with small improvement of the performance. Meanwhile, the CNN kernels treat all the channels equally and cannot take the advantage of the abundant high-frequency information contained in the low-resolution images. To address these problems, Channel attention is one of the mechanisms and has been proven to be useful in many tasks. In this research, we proposed a channel attention-based framework for Remote Sensing Image Super-resolution (CARS) by constructing a novel residual channel attention block (RCAB) to further extract the features. In addition, a densely residual channel attention block (RCAB+) and densely residual spatial attention block (RSAB) were proposed to improve the performance. We adopted a post-upsampling architecture to reduce the computational complexity and time cost. Moreover, transfer learning strategy (CARS+T) was introduced to further improve the SR performance and proved to generate finer edge details. Experimentally, our proposed CARS, CARS_SA and CARS+T achieved competitive quantitative and qualitative results both on Data Fusion Contest Dataset and Pleiades Dataset that we created.

Enhanced Single Image Super Resolution Method Using Lightweight Multi-Scale Channel Dense Network.

Super resolution (SR) enables to generate a high-resolution (HR) image from one or more low-resolution (LR) images. Since a variety of CNN models have been recently studied in the areas of computer vision, these approaches have been combined with SR in order to provide higher image restoration. In this paper, we propose a lightweight CNN-based SR method, named multi-scale channel dense network (MCDN). In order to design the proposed network, we extracted the training images from the DIVerse 2K (DIV2K) dataset and investigated the trade-off between the SR accuracy and the network complexity. The experimental results show that the proposed method can significantly reduce the network complexity, such as the number of network parameters and total memory capacity, while maintaining slightly better or similar perceptual quality compared to the previous methods.

Wide Weighted Attention Multi-Scale Network for Accurate MR Image Super-Resolution

High-quality magnetic resonance (MR) images afford more detailed information for reliable diagnoses and quantitative image analyses. Given low-resolution (LR) images, the deep convolutional neural network (CNN) has shown its promising ability for image super-resolution (SR). The LR MR images usually share some visual characteristics: structural textures of different sizes, edges with high correlation, and less informative background. However, multi-scale structural features are informative for image reconstruction, while the background is more smooth. Most previous CNN-based SR methods use a single receptive field and equally treat the spatial pixels (including the background). It neglects to sense the entire space and get diversified features from the input, which is critical for high-quality MR image SR. We propose a wide weighted attention multi-scale network ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{W}^{2}$ </tex-math></inline-formula> AMSN) for accurate MR image SR to address these problems. On the one hand, the features of varying sizes can be extracted by the wide multi-scale branches. On the other hand, we design a non-reduction attention mechanism to recalibrate feature responses adaptively. Such attention preserves continuous cross-channel interaction and focuses on more informative regions. Meanwhile, the learnable weighted factors fuse extracted features selectively. The encapsulated wide weighted attention multi-scale block ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{W}^{2}$ </tex-math></inline-formula> AMSB) is integrated through a recurrent framework and global attention mechanism. Extensive experiments and diversified ablation studies show the effectiveness of our proposed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{W}^{2}$ </tex-math></inline-formula> AMSN, which surpasses state-of-the-art methods on most popular MR image SR benchmarks quantitatively and qualitatively. And our method still offers superior accuracy and adaptability on real MR images.

A Novel SRTSR Model for Cross-Resolution Person Re-Identification

Compared to the general person re-identification (re-id), pedestrian images in cross-resolution person re-id tasks always have variant resolutions, thus the resolution mismatching problem between low-resolution (LR) images and high-resolution (HR) images significantly limits the performance of neural networks. The general solution is to introduce interpolation methods or super-resolution (SR) module to normalize all the images to the same resolution. Unfortunately, current SR methods can only upscale the entire image to a certain scale, but fail to unify the resolution of images with different width-to-height ratio. To solve this problem, we propose a Specific-Resolution-Targeted Super-Resolution (SRTSR) embedded person re-id method. Instead of using deep CNN-based SR methods, our SRTSR network consists of a fast and light-weight feature learning module with a self-adaptive Swish (SA-Swish) as its activation function to improve its performance, and a specific-resolution-targeted upscale module to upscale the images to any specific resolution. Compared to the traditional upscale module which can only accept the upscale factor for the entire image, our upscale module can either accept two independent and arbitrary upscale factors for either height and width, e.g., $2.4\times $ for height and $3.1\times $ for width, or any specific resolution, e.g., $256\times 128$ , helping to unify the resolution of images with different width-to-height ratio. Extensive experiments are conducted and the result shows that the performance of our method is over the SR-based state-of-the-art person re-id methods on common re-id benchmarks.

Inception residual attention network for remote sensing image super-resolution

ABSTRACT How to enhance the spatial resolution for a remote sensing image is an important issue that we face. Many image super-resolution (SR) techniques have been proposed for this purpose and deep convolutional neural network (CNN) is the most effective approach in recent years. However, we observe that most CNN-based SR methods treat low-frequency areas and high-frequency areas equally, hence hindering the recovery of high-frequency information. In this paper, we propose a network named inception residual attention network (IRAN) to address this problem. Specifically, we propose a spatial attention module to make the network adaptively learn the importance of different spatial areas, so as to pay more attention to the areas with high-frequency information. Furthermore, we present an inception module to fuse local multilevel features, so as to provide richer information for reconstructing detailed textures. In order to evaluate the effectiveness of the proposed method, a large number of experiments are performed on UCMerced-LandUse data set and the results show that the proposed method is superior to the current state-of-the-art methods in both visual effects and objective indicators.

Brain MRI Super-Resolution Using 3D Dilated Convolutional Encoder–Decoder Network

The spatial resolution of magnetic resonance images (MRI) is limited by the hardware capacity, sampling time, signal-to-noise ratio (SNR), and patient comfort. Recently, deep convolutional neural networks (CNN) have achieved impressive success in MRI super-resolution (SR) reconstruction. Increasing network depth or width can enlarge the receptive field to improve SR accuracy, however, it is impractical for MRI reconstruction in clinical applications because of high computational loads. To address this issue, we propose a novel dilated convolutional encoder-decoder (DCED) network to improve the resolution of MRI. We exploit three-dimensional (3D) dilated convolutions as encoders to extract high-frequency features. The dilated encoders capture wider contextual information by exponentially enlarging the receptive field, without introducing additional parameters or layers. Then we decode the features using deconvolution operations to alleviate gridding artifacts and restore fine details. To improve information flow, the encoders and decoders are aggregated into symmetrically connected blocks. The output of each block is passed to the final convolution layer, which facilitates to extract hierarchical features. In addition, we also exploit a geometric self-ensemble 3D wavelet fusion method to improve the potential performance of MRI SR. Experimental results on four public available brain datasets show that our proposed method outperforms NLM (non-local means), LRTV (low-rank and total variation) and current CNN-based SR methods, which demonstrates that our method achieves a new state-of-the-art performance in MRI SR task.

A comparative study of CNN-based super-resolution methods in MRI reconstruction and its beyond

The progress of convolution neural network (CNN) based super-resolution has shown its potential in image processing community. Meanwhile, Compressed Sensing MRI (CS-MRI) provides the possibility to accelerate the traditional acquisition process of MRI. In this work, on the basis of decomposing the cascade network to be a series of alternating CNN-based sub-network and data-consistency sub-network, we investigate the performance of the cascade networks in CS-MRI by employing various CNN-based super-resolution methods in the CNN-based sub-network. Furthermore, recognizing that existing methods of exploring dense connection in the CNN-based sub-network are insufficient to utilize the feature information, we propose a dense connected cascade network for more accurate MRI reconstruction. Specifically, the proposed network densely connects both CNN-based sub-network and data-consistency sub-network, thus takes advantage of the data-consistency of k-space data in a densely connected fashion. Experimental results on various MR data demonstrated that the proposed network is superior to current cascade networks in reconstruction quality.

A Real-Time Convolutional Neural Network for Super-Resolution on FPGA With Applications to 4K UHD 60 fps Video Services

In this paper, we present a novel hardware-friendly super-resolution (SR) method based on a convolutional neural network (CNN) and its dedicated hardware (HW) on field programmable gate array (FPGA). Although CNN-based SR methods have shown very promising results for SR, their computational complexities are prohibitive for hardware implementation. To the best of our knowledge, we are the first to implement a real-time CNN-based SR HW that upscales 2K full high-definition video to 4K ultra high-definition (UHD) video at 60 frames per second (fps). In our dedicated CNN-based SR HW, low-resolution input frames are processed line-by-line, and the number of convolutional filter parameters is reduced significantly by incorporating depth-wise separable convolutions with a residual connection. Our CNN-based SR HW incorporates a cascade of 1D convolutions having large receptive fields along horizontal lines while keeping vertical receptive fields minimal, which allows us to save required line memory space in achieving comparable SR performance against full 2D convolution operations. For efficient HW implementation, we use a simple and effective quantization method with little peak signal-to-noise ratio (PSNR) degradation. Also, we propose a compression method to efficiently store intermediate feature map data to reduce the number of line memories used in HW. Our HW implementation on the FPGA generates 4K UHD frames of higher PSNR values at 60 fps and shows better visual quality, compared with conventional CNN-based SR methods that are trained and tested in software.

A Comparable Study of CNN-Based Single Image Super-Resolution for Space-Based Imaging Sensors.

In the case of space-based space surveillance (SBSS), images of the target space objects captured by space-based imaging sensors usually suffer from low spatial resolution due to the extremely long distance between the target and the imaging sensor. Image super-resolution is an effective data processing operation to get informative high resolution images. In this paper, we comparably study four recent popular models for single image super-resolution based on convolutional neural networks (CNNs) with the purpose of space applications. We specially fine-tune the super-resolution models designed for natural images using simulated images of space objects, and test the performance of different CNN-based models in different conditions that are mainly considered for SBSS. Experimental results show the advantages and drawbacks of these models, which could be helpful for the choice of proper CNN-based super-resolution method to deal with image data of space objects.

GUN: Gradual Upsampling Network for Single Image Super-Resolution

In this paper, an efficient super-resolution (SR) method based on deep convolutional neural network (CNN) is proposed, namely gradual upsampling network (GUN). Recent CNN-based SR methods often preliminarily magnify the low-resolution (LR) input to high-resolution (HR) input and then reconstruct the HR input, or directly reconstruct the LR input and then recover the HR result at the last layer. The proposed GUN utilizes a gradual process instead of these two commonly used frameworks. The GUN consists of an input layer, multiple upsampling and convolutional layers, and an output layer. By means of the gradual process, the proposed network can simplify the direct SR problem to multistep easier upsampling tasks with very small magnification factor in each step. Furthermore, a gradual training strategy is presented for the GUN. In the proposed training process, an initial network can be easily trained with edgelike samples, and then, the weights are gradually tuned with more complex samples. The GUN can recover fine and vivid results and is easy to be trained. The experimental results on several image sets demonstrate the effectiveness of the proposed network.