Super-resolution Network Research Articles

Multi-dimensional attention-aided transposed ConvBiLSTM network for hyperspectral image super-resolution

Hyperspectral (HS) image always suffers from the deficiency of low spatial resolution, compared with conventional optical image types, which has limited its further applications in remote sensing areas. Therefore, HS image super-resolution (SR) techniques are broadly employed in order to observe finer spatial structures while preserving the spectra of ground covers. In this paper, a novel multi-dimensional attention-aided transposed convolutional long-short term memory (LSTM) network is proposed for single HS image super-resolution task. The proposed network employs the convolutional bi-directional LSTM for the purpose of local and non-local spatial–spectral feature explorations, and transposed convolution for the purpose of image amplification and reconstruction. Moreover, a multi-dimensional attention module is proposed, aiming to capture the salient features on spectral, channel, and spatial dimensions, simultaneously, to further improve the learning abilities of network. Experiments on four commonly-used HS images demonstrate the effectiveness of this approach, compared with several state-of-the-art deep learning-based SR methods.

Asymmetric convolutional modulation network for efficient image super-resolution

Thanks to the effectiveness of large kernel convolutions in acquiring large receptive fields, lightweight single image super-resolution (SISR) approaches based on large kernel designs have achieved significant performance. However, these state-of-the-art methods still require high computational costs, and their model efficiency remains to be improved. To address these challenges, we propose an asymmetric convolutional modulation network (ACMN) for efficient image super-resolution. In detail, we design an asymmetric convolutional modulation unit (ACMU) that combines large kernel design and modulation mechanism for adaptively selecting representative features from a large receptive field. Furthermore, we adopt depth-wise asymmetric convolutions to further reduce computational burden. To supplement the local context information and promote inter-channel interaction, we further develop a channel shuffle feedforward network (CSFN) to extract local feature information and facilitate inter-channel information interaction. Experimental results demonstrate that our ACMN outperforms other state-of-the-art efficient SR methods with fewer parameters and Multi-Adds.

Optimization of semi-supervised generative adversarial network models: a survey

PurposeWith the development of intelligent technology, deep learning has made significant progress and has been widely used in various fields. Deep learning is data-driven, and its training process requires a large amount of data to improve model performance. However, labeled data is expensive and not readily available.Design/methodology/approachTo address the above problem, researchers have integrated semi-supervised and deep learning, using a limited number of labeled data and many unlabeled data to train models. In this paper, Generative Adversarial Networks (GANs) are analyzed as an entry point. Firstly, we discuss the current research on GANs in image super-resolution applications, including supervised, unsupervised, and semi-supervised learning approaches. Secondly, based on semi-supervised learning, different optimization methods are introduced as an example of image classification. Eventually, experimental comparisons and analyses of existing semi-supervised optimization methods based on GANs will be performed.FindingsFollowing the analysis of the selected studies, we summarize the problems that existed during the research process and propose future research directions.Originality/valueThis paper reviews and analyzes research on generative adversarial networks for image super-resolution and classification from various learning approaches. The comparative analysis of experimental results on current semi-supervised GAN optimizations is performed to provide a reference for further research.

High-resolution enhanced cross-subspace fusion network for light field image superresolution

Light field (LF) images offer abundant spatial and angular information, therefore, the combination of which is beneficial in the performance of LF image superresolution (LF image SR). Currently, existing methods often decompose the 4D LF data into low-dimensional subspaces for individual feature extraction and fusion for LF image SR. However, the performance of these methods is restricted because of lacking effective correlations between subspaces and missing out on crucial complementary information for capturing rich texture details. To address this, we propose a cross-subspace fusion network for LF spatial SR (i.e., CSFNet). Specifically, we design the progressive cross-subspace fusion module (PCSFM), which can progressively establish cross-subspace correlations based on a cross-attention mechanism to comprehensively enrich LF information. Additionally, we propose a high-resolution adaptive enhancement group (HR-AEG), which preserves the texture and edge details in the high resolution feature domain by employing a multibranch enhancement method and an adaptive weight strategy. The experimental results demonstrate that our approach achieves highly competitive performance on multiple LF datasets compared to state-of-the-art (SOTA) methods.

LVTSR: learning visible image texture network for infrared polarization super-resolution imaging

Infrared polarization (IRP) division-of-focal-plane (DoFP) imaging technology has gained attention, but limited resolution due to sensor size hinders its development. High-resolution visible light (VIS) images are easily obtained, making it valuable to use VIS images to enhance IRP super-resolution (SR). However, IRP DoFP SR is more challenging than infrared SR due to the need for accurate polarization reconstruction. Therefore, this paper proposes an effective multi-modal SR network, integrating high-resolution VIS image constraints for IRP DoFP image reconstruction, and incorporating polarization information as a component of the loss function to achieve end-to-end IRP SR. For the multi-modal IRP SR, a benchmark dataset was created, which includes 1559 pairs of registered images. Experiments on this dataset demonstrate that the proposed method effectively utilizes VIS images to restore polarization information in IRP images, achieving a 4x magnification. Results show superior quantitative and visual evaluations compared to other methods.

PSAR-SR: Patches separation and artifacts removal for improving super-resolution networks

The success of the ClassSR has led to a strategy of decomposing images being used for large image SR. The decomposed image patches have different recovery difficulties. Therefore, in ClassSR, image patches are reconstructed by different networks to greatly reduce the computational cost. However, in ClassSR, the training of multiple sub-networks inevitably increases the training difficulty. Furthermore, decomposing images with overlapping not only increases the computational cost but also inevitably produces artifacts. To address these challenges, we propose an end-to-end general framework, named patches separation and artifacts removal SR (PSAR-SR). In PSAR-SR, we propose an image information complexity module (IICM) to efficiently determine the difficulty of recovering image patches. Then, we propose a patches classification and separation module (PCSM), which can dynamically select an appropriate SR path for image patches of different recovery difficulties. Moreover, we propose a multi-attention artifacts removal module (MARM) in the network backend, which can not only greatly reduce the computational cost but also solve the artifacts problem well under the overlapping-free decomposition. Further, we propose two loss functions — threshold penalty loss (TP-Loss) and artifacts removal loss (AR-Loss). TP-Loss can better select appropriate SR paths for image patches. AR-Loss can effectively guarantee the reconstruction quality between image patches. Experiments show that compared to the leading methods, PSAR-SR well eliminates artifacts under the overlapping-free decomposition and achieves superior performance on existing methods (e.g., FSRCNN, CARN, SRResNet, RCAN and CAMixerSR). Moreover, PSAR-SR saves 53%–65% FLOPs in computational cost far beyond the leading methods. The code will be made available: https://github.com/dywang95/PSAR-SR.

Resolution Enhancement of Metabolomic J-Res NMR Spectra Using Deep Learning.

J-Resolved (J-Res) nuclear magnetic resonance (NMR) spectroscopy is pivotal in NMR-based metabolomics, but practitioners face a choice between time-consuming high-resolution (HR) experiments or shorter low-resolution (LR) experiments which exhibit significant peak overlap. Deep learning neural networks have been successfully used in many fields to enhance quality of natural images, especially with regard to resolution, and therefore offer the prospect of improving two-dimensional (2D) NMR data. Here, we introduce the J-RESRGAN, an adapted and modified generative adversarial network (GAN) for image super-resolution (SR), which we trained specifically for metabolomic J-Res spectra to enhance peak resolution. A novel symmetric loss function was introduced, exploiting the inherent vertical symmetry of J-Res NMR spectra. Model training used simulated high-resolution J-Res spectra of complex mixtures, with corresponding low-resolution spectra generated via blurring and down-sampling. Evaluation of peak pair resolvability on J-RESRGAN demonstrated remarkable improvement in resolution across a variety of samples. In simulated plasma data, 100% of peak pairs exhibited enhanced resolution in super-resolution spectra compared to their low-resolution counterparts. Similarly, enhanced resolution was observed in 80.8-100% of peak pairs in experimental plasma, 85.0-96.7% in urine, 94.4-98.9% in full fat milk, and 82.6-91.7% in orange juice. J-RESRGAN is not sample type, spectrometer or field strength dependent and improvements on previously acquired data can be seen in seconds on a standard desktop computer. We believe this demonstrates the promise of deep learning methods to enhance NMR metabolomic data, and in particular, the power of J-RESRGAN to elucidate overlapping peaks, advancing precision in a wide variety of NMR-based metabolomics studies. The model, J-RESRGAN, is openly accessible for download on GitHub at https://github.com/yanyan5420/J-RESRGAN.

Infrared Image Super-Resolution Network Utilizing the Enhanced Transformer and U-Net.

Infrared images hold significant value in applications such as remote sensing and fire safety. However, infrared detectors often face the problem of high hardware costs, which limits their widespread use. Advancements in deep learning have spurred innovative approaches to image super-resolution (SR), but comparatively few efforts have been dedicated to the exploration of infrared images. To address this, we design the Residual Swin Transformer and Average Pooling Block (RSTAB) and propose the SwinAIR, which can effectively extract and fuse the diverse frequency features in infrared images and achieve superior SR reconstruction performance. By further integrating SwinAIR with U-Net, we propose the SwinAIR-GAN for real infrared image SR reconstruction. SwinAIR-GAN extends the degradation space to better simulate the degradation process of real infrared images. Additionally, it incorporates spectral normalization, dropout, and artifact discrimination loss to reduce the potential image artifacts. Qualitative and quantitative evaluations on various datasets confirm the effectiveness of our proposed method in reconstructing realistic textures and details of infrared images.

Enhanced Learning Enriched Features Mechanism Using Deep Convolutional Neural Network for Image Denoising and Super-Resolution

Enhanced Learning Enriched Features Mechanism Using Deep Convolutional Neural Network for Image Denoising and Super-Resolution

Dual path features interaction network for efficient image super-resolution

Image super-resolution (SR) is a crucial task in computer vision that involves reconstructing a low-resolution (LR) image into its high-resolution (HR) counterpart. Transformer-based methods excel at establishing long-range dependency but face challenges with high-complexity computations and capturing fine-grained features. Conversely, CNN-based methods are advantageous in capturing fine-grained features but are limited by fixed receptive fields. Additionally, most SR methods suffer from channel redundancy, leading to higher computational overhead. In this paper, we propose a Dual Path Features Interaction Network (DPFINet) to achieve efficient image SR, which consists of two components: a) To alleviate the issue of feature channel redundancy, a Local–Global Features Modeling (LGFM) method is newly proposed, which concurrently models global and local features by splitting features along different channels. In LGFM, a Shift Window Linear Attention (SWLA) layer is adopted to effectively capture global information through a large shift window based on the split features. Meanwhile, a Multi-Scale Detail Enhancement (MSDE) layer is designed, where the split other features are encoded to facilitate detail reconstruction through an interactive fusion of semantic and local information, thereby addressing the limitations of SWLA in capturing fine-grained features. b) A Cross-Level Features Interaction (CLFI) method is proposed to fuse global and local features modeled by different network structures (SWLA and MSDE), where a novel residual fusion mechanism is designed to preserve both global and local information while complementing each other. Extensive experiments demonstrate that our method outperforms most state-of-the-art SR methods on five benchmark datasets. Notably, during inference, our approach improves performance by 0.41 dB and reduces memory consumption by approximately 79% compared to DiVANet (Behjati et al., 2023) on the Manga109 (×4) dataset.

Digging into Depth and Color Spaces: A Mapping Constraint Network for Depth Super-Resolution

Scene depth super-resolution (DSR) poses an inherently ill-posed problem due to the extremely large space of one-to-many mapping functions from a given low-resolution (LR) depth map, which possesses limited depth information, to multiple plausible high-resolution (HR) depth maps. This characteristic renders the task highly challenging, as identifying an optimal solution becomes significantly intricate amidst this multitude of potential mappings. While simplistic constraints have been proposed to address the DSR task, the relationship between LR and HR depth maps and the color image has not been thoroughly investigated. In this paper, we introduce a novel mapping constraint network (MCNet) that incorporates additional constraints derived from both LR depth maps and color images. This integration aims to optimize the space of mapping functions and enhance the performance of DSR. Specifically, alongside the primary DSR network (DSRNet) dedicated to learning LR-to-HR mapping, we have developed an auxiliary degradation network (ADNet) that operates in reverse, generating the LR depth map from the reconstructed HR depth map to obtain depth features in LR space. To enhance the learning process of DSRNet in LR-to-HR mapping, we introduce two mapping constraints in LR space: 1) the cycle-consistent constraint, which offers additional supervision by establishing a closed loop between LR-to-HR and HR-to-LR mappings, and 2) the region-level contrastive constraint, aimed at reinforcing region-specific HR representations by explicitly modeling the consistency between LR and HR spaces. To leverage the color image effectively, we introduce a feature screening module (FSM) to adaptively fuse color features at different layers, which can simultaneously maintain strong structural context and suppress texture distraction through subspace generation and image projection. Comprehensive experimental results across synthetic and real-world benchmark datasets unequivocally demonstrate the superiority of our proposed method over state-of-the-art DSR methods. Our MCNet achieves an average MAD reduction of 3.7% and 7.5% over state-of-the-art DSR method for \(\times\) 8 and \(\times\) 16 cases on Milddleburry dataset, respectively, without incurring additional costs during inference.

AttGAN: attention gated generative adversarial network for spatio-temporal super-resolution of ocean phenomena

ABSTRACT This study proposes an innovative deep learning-aided approach based on generative adversarial networks named AttGAN, which is specialized for solving the spatio-temporal super-resolution problem of ocean datasets with complex waveforms and turbulent features. The proposed method can efficiently restore the desired fine high-frequency details while maintaining high efficiency. The key idea of the proposed approach is to incorporate an attention gate in the generator, which allows for emphasizing salient features for super-resolution tasks. In addition, residual convolutional blocks are used in the generator and discriminator to extract features. By establishing regression loss, physical constraints, and adversarial loss into a comprehensive loss function, a generator that exhibits high fidelity and strong generalization capabilities is trained. The proposed AttGAN is evaluated by experiments conducted on two datasets, and the experimental results validate its superiority over state-of-the-art physical constraint models in both qualitative and quantitative metrics. Moreover, the AttGAN has faster processing times than the corresponding ocean numerical models and better quality of results. Impact statement

FDE-Net: A memory-efficiency densely connected network inspired from fractional-order differential equations for single image super-resolution

Dense connection has proved an effective way to take full advantage of hierarchical features extracted from low-resolution images using deep neural networks (DNNs) for single image super-resolution (SISR). However, existing densely connected networks are often manually designed and overly depended on practical experience, thus leading to suboptimal performance. Moreover, due to their complicated connections, they are memory-consuming and lack of interpretability. To address all these problems with one stone, we propose to construct a new densely connected network for SISR from a dynamic system perspective. Following this idea, we cast the hierarchical transformation of DNNs into the state evolution of a fractional-order dynamic system, which empowers us to automatically construct two interdependent densely connected modules based on the system solution rather manual design. They are a prediction module that controls the system to iteratively predict the next state, and a correction module that iteratively refines the predicted state to improve the prediction accuracy. With these two modules as backbone, we establish a Fractional-order Differential Equations-based network (FDE-Net) for SISR. Since the iterative computational manner requires both densely connected modules to be of the recurrent structure, FDE-Net is memory-efficiency and good interpretability. In addition, we analyze the existence and uniqueness of the solution for FDE to theoretically guarantee the feasibility of FDE-Net. Experiments on four SISR benchmark datasets demonstrate the superiority of FDE-Net over existing densely connected networks and other baselines in terms of generalization capacity, especially with limited memory.

Multi-directional feature fusion super-resolution network based on nonlinear spiking neural P systems

Single-image super-resolution (SISR) aims to recover lost details and textures from limited information to improve the visual quality and detail clarity of an image. Most of the existing convolutional neural network (CNN)-based SISR methods do not make full use of the intermediate features of the network, and at the same time its processing process can result in the loss of information details. To address these problems, this paper recommends a novel multi-directional feature fusion super-resolution network. The direction-aware mechanism with nonlinear spiking neural P (NSNP) systems is integrated to design a multidirectional feature extraction module, while different NSNP-like convolution modules are constructed for extracting deep features on the basis of nonlinear spiking mechanism. Moreover, in the residual feature enhancement module, three different branches are designed using these NSNP-like convolution modules for feature enhancement and fusion to decrease the loss of information. The preposed network is evaluated on five benchmark datasets, and the experimental results show that the network performs a better reconstruction with good capability of generalization compared to most state-of-the-art methods.

LSRN-AED: lightweight super-resolution network based on asymmetric encoder–decoder

LSRN-AED: lightweight super-resolution network based on asymmetric encoder–decoder

PEFormer: a pixel-level enhanced CNN-transformer hybrid network for face image super-resolution

PEFormer: a pixel-level enhanced CNN-transformer hybrid network for face image super-resolution

Semantic guidance incremental network for efficiency video super-resolution

AbstractIn video streaming, bandwidth constraints significantly affect client-side video quality. Addressing this, deep neural networks offer a promising avenue for implementing video super-resolution (VSR) at the user end, leveraging advancements in modern hardware, including mobile devices. The principal challenge in VSR is the computational intensity involved in processing temporal/spatial video data. Conventional methods, uniformly processing entire scenes, often result in inefficient resource allocation. This is evident in the over-processing of simpler regions and insufficient attention to complex regions, leading to edge artifacts in merged regions. Our innovative approach employs semantic segmentation and spatial frequency-based categorization to divide each video frame into regions of varying complexity: simple, medium, and complex. These are then processed through an efficient incremental model, optimizing computational resources. A key innovation is the sparse temporal/spatial feature transformation layer, which mitigates edge artifacts and ensures seamless integration of regional features, enhancing the naturalness of the super-resolution outcome. Experimental results demonstrate that our method significantly boosts VSR efficiency while maintaining effectiveness. This marks a notable advancement in streaming video technology, optimizing video quality with reduced computational demands. This approach, featuring semantic segmentation, spatial frequency analysis, and an incremental network structure, represents a substantial improvement over traditional VSR methodologies, addressing the core challenges of efficiency and quality in high-resolution video streaming.

Dual residual and large receptive field network for lightweight image super-resolution

In recent years, there has been rapid progress in single-image super-resolution (SISR) reconstruction technology based on deep learning, but many methods face challenges in practical application due to their excessive computational requirements. To address this issue, numerous lightweight super-resolution (SR) methods have been proposed. Convolutional neural networks (CNNs) have demonstrated outstanding performance in lightweight SR reconstruction tasks. However, existing CNN-based lightweight SR networks still suffer from redundancy due to the use of small convolutional kernels and deeper network architectures for feature extraction. To solve these problems, we propose a novel network called Dual Residual and Large Receptive Field Network (DRLN), which includes two main designs. Firstly, a Large Receptive Field Feature Extraction Block (LRFEB) is proposed, which concatenates large kernel depth separable convolutions and large kernel depth separable dilated convolutions, effectively capturing spatial contextual information without increasing the number of parameters or computational burden, and enhancing the overall performance of the model. Secondly, in the Efficient Feature Distillation Block (EFDB), we innovatively propose a dual residual structure, which greatly improve the learning ability and feature expression richness of the network, and significantly improve the reconstruction performance. Extensive experiments demonstrate that DRLN surpasses BSRN which is the champion of the model complexity track of the NTIRE 2022 Efficient SR Challenge on all benchmark datasets, with fewer parameters and less computation.

Spectral Super-Resolution via Model-Guided Cross-Fusion Network.

Spectral super-resolution, which reconstructs a hyperspectral image (HSI) from a single red-green-blue (RGB) image, has acquired more and more attention. Recently, convolution neural networks (CNNs) have achieved promising performance. However, they often fail to simultaneously exploit the imaging model of the spectral super-resolution and complex spatial and spectral characteristics of the HSI. To tackle the above problems, we build a novel cross fusion (CF)-based model-guided network (called SSRNet) for spectral super-resolution. In specific, based on the imaging model, we unfold the spectral super-resolution into the HSI prior learning (HPL) module and imaging model guiding (IMG) module. Instead of just modeling one kind of image prior, the HPL module is composed of two subnetworks with different structures, which can effectively learn the complex spatial and spectral priors of the HSI, respectively. Furthermore, a CF strategy is used to establish the connection between the two subnetworks, which further improves the learning performance of the CNN. The IMG module results in solving a strong convex optimization problem, which adaptively optimizes and merges the two features learned by the HPL module by exploiting the imaging model. The two modules are alternately connected to achieve optimal HSI reconstruction performance. Experiments on both the simulated and real data demonstrate that the proposed method can achieve superior spectral reconstruction results with relatively small model size. The code will be available at https://github.com/renweidian.

Super-resolution of digital rock images with hybrid attention multi-branch neural network

High-resolution digital rock images are vital in studying microstructure and fluid flow characterization of oil and gas reservoirs. Computed tomography (CT) is currently the primary tool to acquire digital rock images because of its non-destructiveness and three-dimensional imaging capability. However, the restriction of imaging capability of CT sets a compromise between image resolution and field of view (FOV). The Hybrid Attention Multi-branch Super-Resolution Network (HAMSR) proposed in this study aims to remedy this deficiency. HAMSR employs a limited discrete distribution to capture all possible high-order information and generate reliable high-resolution images. The network is built to prioritize the recovery of pores, cracks, and textures in digital rock images by introducing channel and spatial attention mechanisms. The structure together with modules of HAMSR have been designed and optimized to yield superior performance with fewer parameters and less training time. HAMSR and several contemporary advanced models are trained on the DeepRock-SR 2D dataset which contains lots of sandstone and carbonate images. It is evident that HAMSR-generated images have minimal noise, clearer edges, and more micropores. Quantitative calculations show that compared to traditional bicubic interpolation, HAMSR acquires a 2.669 dB and 1.376 dB increase in peak signal-to-noise ratio (PSNR) respectively on the test sets (27%–38% reduction in relative error). Furthermore, the high-resolution (HR) images reconstructed by HAMSR are generally consistent with real HR images concerning statistical characteristics such as porosity, pore size distribution, shape factor distribution, and two-point correlation function. This work provides a new super-resolution model to generate high-quality digital rock images for further analysis.