Super-resolution Network Research Articles

CVGSR: Stereo image Super-Resolution with Cross-View guidance

The ability to capture the complementary information of stereo images is critical to the development of stereo image super-resolution. Most existing studies have attempted to integrate reliable stereo correspondence along the polar directions through parallax attention and various fusion strategies. However, most of these approaches ignore the large parallax differences in stereo images, resulting in poor performance of convolutional-based parallax attention in capturing the long-range dependencies between images. In this paper, we propose a novel cross-view guided stereo image super-resolution network (CVGSR) for reconstructing high-resolution stereo image pairs with rich texture details by fully exploiting the complementary nature of stereo image pairs. Specifically, we first deploy a cross-view interaction module (CVIM) to explore intra/cross-view dependencies from local to global to compensate for the incomplete compatibility of information between the left and right views. This module uses a progressive cross-guiding strategy to better merge features from occluded and non-occluded regions. Based on this, an efficient attention Transformer (EAT) is improved to activate more input information and further mine the cross-view complementarity. Furthermore, we design a texture loss to optimize the visual perceptual quality of reconstructed images with sharp boundaries and rich texture details. Extensive experiments on four stereo image datasets demonstrate that the proposed CVGSR achieves a competitive and excellent performance.

Multi-level feature interaction image super-resolution network based on convolutional nonlinear spiking neural model

Image super-resolution (ISR) is designed to recover lost detail information from low-resolution images, resulting in high-quality and high-definition high-resolution images. In the existing single ISR (SISR) methods based on convolutional neural networks (CNN), however, most of the models cannot effectively combine global and local information and are also easy to ignore the correlation between different hierarchical feature information. To address these problems, this study proposes a multi-level feature interactive image super-resolution network, which is constructed by the convolutional units inspired by nonlinear spiking mechanism in nonlinear spiking neural P systems, including shallow feature processing, deep feature extraction and fusion, and reconstruction modules. The different omni domain self-attention blocks are introduced to extract global information in the deep feature extraction and fusion stage and formed a feature enhancement module having a Transformer structure using a novel convolutional unit for extracting local information. Furthermore, to adaptively fuse features between different hierarchies, we design a multi-level feature fusion module, which not only can adaptively fuse features between different hierarchies, but also can better interact with contextual information. The proposed model is compared with 16 state-of-the-art or baseline models on five benchmark datasets. The experimental results show that the proposed model not only achieves good reconstruction performance, but also strikes a good balance between model parameters and performance.

A High-Performance Accelerator for Real-Time Super-Resolution on Edge FPGAs

In the digital era, the prevalence of low-quality images contrasts with the widespread use of high-definition displays, primarily due to low-resolution cameras and compression technologies. Image super-resolution (SR) techniques, particularly those leveraging deep learning, aim to enhance these images for high-definition presentation. However, real-time execution of deep neural network (DNN)-based SR methods at the edge poses challenges due to their high computational and storage requirements. To address this, field-programmable gate arrays (FPGAs) have emerged as a promising platform, offering flexibility, programmability, and adaptability to evolving models. Previous FPGA-based SR solutions have focused on reducing computational and memory costs through aggressive simplification techniques, often sacrificing the quality of the reconstructed images. This paper introduces a novel SR network specifically designed for edge applications, which maintains reconstruction performance while managing computation costs effectively. Additionally, we propose an architectural design that enables the real-time and end-to-end inference of the proposed SR network on embedded FPGAs. Our key contributions include a tailored SR algorithm optimized for embedded FPGAs, a DSP-enhanced design that achieves a significant four-fold speedup, a novel scalable cache strategy for handling large feature maps, optimization of DSP cascade consumption, and a constraint optimization approach for resource allocation. Experimental results demonstrate that our FPGA-specific accelerator surpasses existing solutions, delivering superior throughput, energy efficiency, and image quality.

Multi-scale gated network for efficient image super-resolution

Multi-scale gated network for efficient image super-resolution

Sub-RENet: a wavelet-based network for super resolution of diagnostic ultrasound

Sub-RENet: a wavelet-based network for super resolution of diagnostic ultrasound

Refining computer tomography data with super-resolution networks to increase the accuracy of respiratory flow simulations

Accurately computing the flow in the nasal cavity with computational fluid dynamics (CFD) simulations requires highly resolved computational meshes based on anatomically realistic geometries. Such geometries can only be obtained from computer tomography (CT) data with high spatial resolution, i.e., featuring a ≤1mm slice thickness. In practice, CT images are, however, recorded at a lower resolution to not expose patients to high radiation and to reduce the overall costs. To overcome this problem and to provide patients with a detailed physics-based diagnosis, e.g., for surgery planning, the potential of super-resolution networks (SRNs) to increase the CT resolution is analyzed. Therefore, an SRN is developed and trained on CT data. Its predictive performance is improved by an automated hyperparameter optimization technique. The training time is further reduced without predictive accuracy degradation by oversampling images with challenging regions. The performance of the SRN is assessed by an analysis of the reconstructed 3D surfaces of the human upper airway and by comparing results of CFD simulations. That is, surfaces and simulation results based on SRN-generated CT data at 1mm resolution are compared to those obtained from unmodified CT data-sets at low (3mm) and high (1mm) resolution, as well as from CT data interpolated to a 1mm resolution from coarse data. The findings reveal the SRN-based approach to have the lowest deviations in the surfaces and CFD results when compared to those based on the original high-resolution data. The pressure loss between the inflow (nostrils) and outflow (pharynx) regions averaged for three test patients differs by only 1.3%, compared to 8.7% and 8.8% in the coarse and interpolated cases. It is concluded that the SRN-based method is a promising tool to enhance underresolved CT data to yield reliable numerical results of respiratory flows.

Local Means Binary Networks for Image Super-Resolution.

The success of modern single image super-resolution (SISR) algorithms is inspired by the development of deep convolutional neural networks (CNNs). However, these CNN-based methods require considerable computation and complexity, making it impossible for these methods to perform real-time calculations in edge devices. Thus, lightweight model design has become a development trend in the super-resolution field, including pruning, quantization, and other methods. The 1-bit quantization is an extreme lightweight method which can reduce the calculation amount of the model in an extreme manner and is friendly to hardware such as edge devices. Most existing binary quantization approaches lead to a large information loss during forward propagation, especially in detailed color information (e.g., edge, texture, and contrast). The loss of color information makes modern binary methods unsuitable for SISR tasks. We think the loss occurs because these methods typically utilize a uniform threshold to quantize the weights and activations. Thus, in this article, we thoroughly analyze the difference between normal classification tasks and SISR tasks, and present a binarization scheme based on local means. The proposed method can maintain more detailed information in feature maps using dynamic thresholds during quantization. Specifically, each value in the full precision activations has a corresponding threshold during the quantization process, and those thresholds are determined by the full precision values of the surroundings. In addition, a gradient approximator is introduced to adaptively optimize the gradient for updating binary weights. We then verify the effectiveness of our method for training binary networks on several SISR benchmarks including VDSR and SRResNet. Experimental results show that the proposed method can outperform the state-of-the-art algorithms to obtain binary networks for image super-resolution with better peak signal-to-noise ratio (PSNR) values and visual quality.

Image Super-resolution Networks based on Structure Reparameterized Convolution

The traditional image super resolution method based on convolutional neural network generally brings two problems: First, the model has only a single scale of receptive field, which cannot use the image information in a smaller range; Second, the size of the feature map will become smaller and smaller in the process of continuous convolution, and it is necessary to continuously carry out edge zeroing operation to maintain the original size, which leads to the loss of part of the edge information. To solve the above two problems, we propose a network structure based on structure re-parameterized convolution, which sets a small convolution kernel in parallel beside a large convolution kernel in the same layer, trains the two cores simultaneously, and finally merges the two cores. The experimental results show that in this way, we make the large convolution kernel could capture smaller information, which not only improves the high-frequency details of the reconstructed image but also avoids frequent edge filling to reduce the information density, and effectively speeds up the reasoning speed after the model is re-parameterized. Compared with the advanced methods, our method achieves good results.

Why Not Both? An Attention-Guided Transformer with Pixel-Related Deconvolution Network for Face Super-Resolution

Transformer-based encoder-decoder networks for face super-resolution (FSR) have achieved promising success in delivering stunningly clear and detailed facial images by capturing local and global dependencies. However, these methods have certain limitations. Specifically, the deconvolution in upsampling layers neglects the relationship between adjacent pixels, which is crucial in facial structure reconstruction. Additionally, raw feature maps are fed to the transformer blocks directly without mining their potential feature information, resulting in suboptimal face images. To circumvent these problems, we propose an attention-guided transformer with pixel-related deconvolution network for FSR. Firstly, we devise a novel Attention-Guided Transformer Module (AGTM), which is composed of an Attention-Guiding Block (AGB) and a Channel-wise Multi-head Transformer Block (CMTB). AGTM at the top of the encoder-decoder network (AGTM-T) promotes both local facial details and global facial structures, while AGTM at the bottleneck side (AGTM-B) optimizes the encoded features. Secondly, a Pixel-Related Deconvolution (PRD) layer is specially designed to establish direct relationships among adjacent pixels in the upsampling process. Lastly, we develop a Multi-scale Feature Fusion Module (MFFM) to fuse multi-scale features for better network flexibility and reconstruction results. Quantitative and qualitative experimental results on various datasets demonstrate that the proposed method outperforms other state-of-the-art FSR methods.

CGIHE-VDSR: Color global image histogram equalization with very deep super resolution networks for color image super resolution

Medical and satellite image analysis require incredibly high resolution. Super-resolution combines several low-resolution images of the same scene to generate a high-resolution image. The Super resolution employing deep learning techniques still has an illumination issue. This paper proposes a novel CGIHE-VDSR algorithm that integrates the Very Deep Super Resolution (VDSR) Network with Color Global Image Histogram Equalization (CGIHE) to improve image resolution. In the proposed method, the low-resolution image is first histogram equalized using the CGIHE algorithm. Then, the VDSR network is applied to the histogram equalized image for super-resolution. The comparison of real-time data with the benchmark images is done using the proposed algorithm in the MATLAB platform. The PSNR and SSIM metrics demonstrate that the super resolution image obtained using the proposed method is significantly better than the existing methods.

Physics-informed deep-learning applications to experimental fluid mechanics

High-resolution reconstruction of flow-field data from low-resolution and noisy measurements is of interest due to the prevalence of such problems in experimental fluid mechanics, where the measurement data are in general sparse, incomplete and noisy. Deep-learning approaches have been shown suitable for such super-resolution tasks. However, a high number of high-resolution examples is needed, which may not be available for many cases. Moreover, the obtained predictions may lack in complying with the physical principles, e.g. mass and momentum conservation. Physics-informed deep learning provides frameworks for integrating data and physical laws for learning. In this study, we apply physics-informed neural networks (PINNs) for super-resolution of flow-field data both in time and space from a limited set of noisy measurements without having any high-resolution reference data. Our objective is to obtain a continuous solution of the problem, providing a physically-consistent prediction at any point in the solution domain. We demonstrate the applicability of PINNs for the super-resolution of flow-field data in time and space through three canonical cases: Burgers’ equation, two-dimensional vortex shedding behind a circular cylinder and the minimal turbulent channel flow. The robustness of the models is also investigated by adding synthetic Gaussian noise. Furthermore, we show the capabilities of PINNs to improve the resolution and reduce the noise in a real experimental dataset consisting of hot-wire-anemometry measurements. Our results show the adequate capabilities of PINNs in the context of data augmentation for experiments in fluid mechanics.

A blind image super-resolution network guided by kernel estimation and structural prior knowledge

The goal of blind image super-resolution (BISR) is to recover the corresponding high-resolution image from a given low-resolution image with unknown degradation. Prior related research has primarily focused effectively on utilizing the kernel as prior knowledge to recover the high-frequency components of image. However, they overlooked the function of structural prior information within the same image, which resulted in unsatisfactory recovery performance for textures with strong self-similarity. To address this issue, we propose a two stage blind super-resolution network that is based on kernel estimation strategy and is capable of integrating structural texture as prior knowledge. In the first stage, we utilize a dynamic kernel estimator to achieve degradation presentation embedding. Then, we propose a triple path attention groups consists of triple path attention blocks and a global feature fusion block to extract structural prior information to assist the recovery of details within images. The quantitative and qualitative results on standard benchmarks with various degradation settings, including Gaussian8 and DIV2KRK, validate that our proposed method outperforms the state-of-the-art methods in terms of fidelity and recovery of clear details. The relevant code is made available on this link as open source.

UPAMNet: A unified network with deep knowledge priors for photoacoustic microscopy

Photoacoustic microscopy (PAM) has gained increasing popularity in biomedical imaging, providing new opportunities for tissue monitoring and characterization. With the development of deep learning techniques, convolutional neural networks have been used for PAM image resolution enhancement and denoising. However, there exist several inherent challenges for this approach. This work presents a Unified PhotoAcoustic Microscopy image reconstruction Network (UPAMNet) for both PAM image super-resolution and denoising. The proposed method takes advantage of deep image priors by incorporating three effective attention-based modules and a mixed training constraint at both pixel and perception levels. The generalization ability of the model is evaluated in details and experimental results on different PAM datasets demonstrate the superior performance of the method. Experimental results show improvements of 0.59 dB and 1.37 dB, respectively, for 1/4 and 1/16 sparse image reconstruction, and 3.9 dB for image denoising in peak signal-to-noise ratio.

A novel spectral super-resolution network with dominant information between spatial and spectral domains

Existing spectral super-resolution (SSR) methods have achieved satisfactory performance by designing complicated deep convolution neural networks (DCNNs) to extract spectral and spatial features. However, these methods ignore the fact that the significance of spatial and spectral information in each hyperspectral image (HSI) is different, and most of them directly fuse two kinds of information with concatenation and convolution operation, which resulting in generating redundant information and have negative effects on reconstruction. To address such inadequacies, this paper proposes a novel adaptive spatial–spectral modulation network (ASSM-Net). Specifically, we propose a new adaptive feature fusion module (AFFM) to replace traditional convolutional fusion schemes. Through explicitly measuring the weights of spatial information and spectral features of HSI, AFFM can select dominant features for each pixel to modulate spatial and spectral information. Additionally, we develop a pixel-weighted aware attention (PAA) mechanism to enhance the feature interdependences for recovering finer structure information. Finally, a large number of quantitative and qualitative experiments reveal that the proposed network achieves competitive reconstruction results on three benchmark datasets (NTIRE 2022, CAVE and Harvard).

Enhanced Video Super-Resolution Network towards Compressed Data

Video super-resolution (VSR) algorithms aim at recovering a temporally consistent high-resolution (HR) video from its corresponding low-resolution (LR) video sequence. Due to the limited bandwidth during video transmission, most available videos on the internet are compressed. Nevertheless, few existing algorithms consider the compression factor in practical applications. In this paper, we propose an enhanced VSR model towards compressed videos, termed as ECVSR, to simultaneously achieve compression artifacts reduction and SR reconstruction end-to-end. ECVSR contains a motion-excited temporal adaption network (METAN) and a multi-frame SR network (SRNet). The METAN takes decoded LR video frames as input and models inter-frame correlations via bidirectional deformable alignment and motion-excited temporal adaption, where temporal differences are calculated as motion prior to excite the motion-sensitive regions of temporal features. In SRNet, cascaded recurrent multi-scale blocks (RMSB) are employed to learn deep spatio-temporal representations from adapted multi-frame features. Then, we build a reconstruction module for spatio-temporal information integration and HR frame reconstruction, which is followed by a detail refinement module for texture and visual quality enhancement. Extensive experimental results on compressed videos demonstrate the superiority of our method for compressed VSR. Code will be available at https://github.com/lifengcs/ECVSR .

A flow-based multi-scale learning network for single image stochastic super-resolution

Single image super-resolution (SISR) is still an important while challenging task. Existing methods usually ignore the diversity of generated Super-Resolution (SR) images. The fine details of the corresponding high-resolution (HR) images cannot be confidently recovered due to the degradation of detail in low-resolution (LR) images. To address the above issue, this paper presents a flow-based multi-scale learning network (FMLnet) to explore the diverse mapping spaces for SR. First, we propose a multi-scale learning block (MLB) to extract the underlying features of the LR image. Second, the introduced pixel-wise multi-head attention allows our model to map multiple representation subspaces simultaneously. Third, by employing a normalizing flow module for a given LR input, our approach generates various stochastic SR outputs with high visual quality. The trade-off between fidelity and perceptual quality can be controlled. Finally, the experimental results on five datasets demonstrate that the proposed network outperforms the existing methods in terms of diversity, and achieves competitive PSNR/SSIM results. Code is available at https://github.com/qianyuwu/FMLnet.

A boosted degradation representation learning for blind image super-resolution

The significant gap between the assumed and actual degradation model will undoubtedly lead to a serious performance decrease for deep learning based image super-resolution (SR) methods. To solve this shortcoming, this paper presents an enhanced blind image super-resolution (EBSR) network based on unsupervised degradation representation learning. The proposed EBSR mainly is made up of two branching subnetworks. The first subnetwork is a residual-based degradation encoder, which is responsible to learn a high-dimensional abstract degradation representation vector from the input low-resolution (LR) image patches using contrast learning. Another branch is the degradation-aware fusion SR (DAFSR) network that completes the image SR task with the aid of the degradation representation vector learned by the encoder. To obtain an improved SR performance under various degradation settings, the degradation-aware fusion block (DAFB), the core block of DAFSR, has been elaborated, which embeds the degradation model information learned by the encoder into every layer of the network. Additionally, a shallow feature extraction block (SFEB) is constructed for DAFSR to extract global information from the LR image. The proposed EBSR can flexibly adapt to various degradation models. Extensive experimental results on synthetic datasets and real images show that the proposed EBSR is able to achieve a leading reconstruction performance over other blind SR algorithms.

SuperVidConform: Conformation detail-preserving network (CDPN) for video super-resolution

Video Super Resolution (VSR) applications extensively utilize deep learning-based methods. Several VSR methods primarily focus on improving the fine-patterns within reconstructed video frames. It frequently overlooks the crucial aspect of keeping conformation details, particularly sharpness. Therefore, reconstructed video frames often fail to meet expectations. In this paper, we propose a Conformation Detail-Preserving Network (CDPN) named as SuperVidConform. It focuses on restoring local region features and maintaining the sharper details of video frames. The primary focus of this work is to generate the high-resolution (HR) frame from its corresponding low-resolution (LR). It consists of two parts: (i) The proposed model decomposes confirmation details from the ground-truth HR frames to provide additional information for the super-resolution process, and (ii) These video frames pass to the temporal modelling SR network to learn local region features by residual learning that connects the network intra-frame redundancies within video sequences. The proposed approach is designed and validated using VID4, SPMC, and UDM10 datasets. The experimental results show the proposed model presents an improvement of 0.43 dB (VID4), 0.78 dB (SPMC), and 0.84 dB (UDM10) in terms of PSNR. Further, the CDPN model set new standards for the performance of self-generated surveillance datasets.

SwinDPSR: Dual-Path Face Super-Resolution Network Integrating Swin Transformer

Whether to use face priors in the face super-resolution (FSR) methods is a symmetry problem.Various face priors are used to describe the overall and local face features, making the generation of super-resolution face images expensive and laborious. FSR methods that do not require any prior information tend to focus too much on the local features of the face, ignoring the modeling of global information. To solve this problem, we propose a dual-path facial image super-resolution network (SwinDPSR) fused with Swin Transformer. The network does not require additional face priors, and it learns global face shape and local face components through two independent branches. In addition, the channel attention ECA module is used to aggregate the global and local face information in the above dual-path sub-networks, which can generate corresponding high-quality face images. The results of face super-resolution reconstruction experiments on public face datasets and a real-scene face dataset show that SwinDPSR is superior to previous advanced methods both in terms of visual effects and objective indicators. The reconstruction results are evaluated with four evaluation metrics: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), and mean perceptual score (MPS).

Scene recovery: Combining visual enhancement and resolution improvement

Visibility enhancement of outdoor images under complex imaging conditions has been a crucial task for computer vision and received growing attention. However, existing image enhancement methods could result in typical block-like artifacts or color distortion. The undesirable impurities might also be significantly magnified after the enhancement task, further reducing the image quality. For enhancing and super-resolving complex real-world degradation, we propose a simultaneous visual enhancement and resolution improvement (VERI) variational scene recovery model for jointly enhancing image visibility and improving the resolution of the degraded image. Particularly, we estimate the scattering light map for degradation images to achieve clean scene radiance and simultaneously seek a high-quality image through a deep super-resolution network. The semi-proximal alternating direction method of multipliers (sPADMM) algorithm is employed for efficiently solving the minimization problems in the proposed model. Extensive experiments illustrate the effectiveness and robustness of the proposed method in dealing with various scenes, such as haze, sandstorm, underwater or low illumination.