Super-resolution Network Research Articles

PLGNet: Prior-Guided Local and Global Interactive Hybrid Network for Face Super-Resolution

PLGNet: Prior-Guided Local and Global Interactive Hybrid Network for Face Super-Resolution

Information sparsity guided transformer for multi-modal medical image super-resolution

Multi-modal medical image super-resolution (SR) plays a vital role in enhancing the resolution of medical images, providing more detailed visuals that aid in accurate clinical diagnosis. Recently, Transformer-based super-resolution methods have significantly promoted performance improvement in this field due to their capacity to capture global dependencies. They usually process all non-overlapping patches as tokens and densely sample these tokens without screening for calculating attention mechanisms. However, this strategy ignores the spatial sparsity of medical images, resulting in redundant or even detrimental computations on less informative regions. Hence, this paper proposes a novel sparsity-guided medical image SR network, namely SG-SRNet, by exploiting the spatial sparsity characteristics of the medical images. SG-SRNet mainly consists of two components: a sparsity mask (SM) generator for image sparsity estimation, and a sparsity-guided Transformer (SGTrans) for high-resolution image reconstruction. Specially, the SM generator generates a sparsity mask by minimizing our cross-sparsity loss which can respond to the informative positions. SGTrans first screens out the informative patches according to the sparsity mask. Then, SGTrans utilizes the designed cluster-based attention to only calculate attention between information-related tokens. We perform comprehensive experiments on three datasets to show that SG-SRNet brings significant performance enhancements with low computational complexity.

A scalable attention network for lightweight image super-resolution

Modeling long-range dependencies among features has become a consensus to improve the results of single image super-resolution (SISR), which stimulates interest in enlarging the kernel sizes in convolutional neural networks (CNNs). Although larger kernels definitely improve the network performance, network parameters and computational complexities are raised sharply as well. Hence, an optimization of setting the kernel sizes is required to improve the efficiency of the network. In this work, we study the influence of the positions of larger kernels on the network performance, and propose a scalable attention network (SCAN). In SCAN, we propose a depth-related attention block (DRAB) that consists of several multi-scale information enhancement blocks (MIEBs) and resizable-kernel attention blocks (RKABs). The RKAB dynamically adjusts the kernel size concerning the locations of the DRABs in the network. The resizable mechanism allows the network to extract more informative features in shallower layers with larger kernels and focus on useful information in deeper layers with smaller ones, which effectively improves the SR results. Extensive experiments demonstrate that the proposed SCAN outperforms other state-of-the-art lightweight SR methods. Our codes are available at https://github.com/ginsengf/SCAN.

An efficient feature reuse distillation network for lightweight image super-resolution

In recent research, single-image super-resolution (SISR) using deep Convolutional Neural Networks (CNN) has seen significant advancements. While previous methods excelled at learning complex mappings between low-resolution (LR) and high-resolution (HR) images, they often required substantial computational and memory resources. We propose the Efficient Feature Reuse Distillation Network (EFRDN) to alleviate these challenges. EFRDN primarily comprises Asymmetric Convolutional Distillation Modules (ACDM), incorporating the Multiple Self-Calibrating Convolution (MSCC) units for spatial and channel feature extraction. It includes an Asymmetric Convolution Residual Block (ACRB) to enhance the skeleton information of the square convolution kernel and a Feature Fusion Lattice Block (FFLB) to convert low-order input signals into higher-order representations. Introducing a Transformer module for global features, we enhance feature reuse and gradient flow, improving model performance and efficiency. Extensive experimental results demonstrate that EFRDN outperforms existing methods in performance while conserving computing and memory resources.

PreciDBPN: A customized deep learning approach for hourly precipitation downscaling in eastern China

Long-term series of high-resolution gridded precipitation datasets are essential for hydrological and meteorological research. Producing high-resolution precipitation data from regional models demands substantial computational resources and labor. Global Reanalyses offer long-term coverage and effectively capture annual and seasonal precipitation patterns. However, they have inadequate resolution and frequently have difficulties depicting extreme conditions. This study proposes an efficient and accurate approach for generating long-term series of high spatial and temporal resolution precipitation. It is achieved by leveraging deep learning techniques to integrate the fifth generation of the European Centre for Medium-Range Weather Forecasts (ECMWF) global climate reanalysis (ERA5, 0.25°, hourly) data with high-resolution precipitation fusion datasets. Considering the heavy-tailed distribution of precipitation, we developed the PreciDBPN model structure, which combines a classification network with a super-resolution network and incorporates physically relevant indices into the model's input. We trained and evaluated the PreciDBPN and baseline models in eastern China using the China Meteorological Administration Land Data Assimilation System (CLDAS) precipitation dataset (0.0625°, hourly, 2017–2022). When compared to baseline methods and ERA5, our model excels in multiple metrics and provides a more precise representation of relative rainfall frequency. Independent verification was performed using station observations during the period of 2010–2015 when CLDAS data were unavailable. During this verification, the PreciDBPN demonstrated exceptional performance and greater robustness compared to the baseline models. Because our method can efficiently downscale precipitation and bias-correct reanalysis data using minimal computational resources, it can be used to generate high-resolution precipitation datasets (0.0625°, hourly) from 1979 to 2022 while correcting for heavy precipitation underestimations in reanalysis data.

Residual SwinV2 transformer coordinate attention network for image super resolution

Swin Transformers have been designed and used in various image super-resolution (SR) applications. One of the recent image restoration methods is RSTCANet, which combines Swin Transformer with Channel Attention. However, for some channels of images that may carry less useful information or noise, Channel Attention cannot automatically learn the insignificance of these channels. Instead, it tries to enhance their expression capability by adjusting the weights. It may lead to excessive focus on noise information while neglecting more essential features. In this paper, we propose a new image SR method, RSVTCANet, based on an extension of Swin2SR. Specifically, to effectively gather global information for the channel of images, we modify the Residual SwinV2 Transformer blocks in Swin2SR by introducing the coordinate attention for each two successive SwinV2 Transformer Layers (S2TL) and replacing Multi-head Self-Attention (MSA) with Efficient Multi-head Self-Attention version 2 (EMSAv2) to employ the resulting residual SwinV2 Transformer coordinate attention blocks (RSVTCABs) for feature extraction. Additionally, to improve the generalization of RSVTCANet during training, we apply an optimized RandAugment for data augmentation on the training dataset. Extensive experimental results show that RSVTCANet outperforms the recent image SR method regarding visual quality and measures such as PSNR and SSIM.

A novel low-quality Raman hyperspectral image reconstruction method for corn component mapping

Raman imaging used to obtain spatial distribution information of chemical components on the sample surface has been widely applied in the nutritional assessment and quality control of cereal grains. However, obtaining high-quality Raman hyperspectral images of cereal grain based on point scanning is time-consuming due to the limitations of the instrumental imaging principle and Raman hyperspectral images acquired in a short time are low-quality with noise interference, limiting its practical application in the visualization of cereal grain composition. To rapidly and effectively achieve high-quality component distribution mappings of cereal grains based on Raman hyperspectral images, we propose a novel Swin Transformer-based method combining spatial-spectral denoising and image reconstruction (SwinSSID-IR) for low-quality Raman hyperspectral image reconstruction. The novel denoising network named SwinSSID innovatively introduces Swin Transformer blocks (STB) to effectively eliminate noise from raw low-quality Raman hyperspectral images of corn kernels. Next, the super-resolution network named SwinIR rapidly generates high-quality component distribution mappings of corn with exceptional clarity and detail from the denoised low-quality Raman hyperspectral image at characteristic peak. Experimental results show the competitiveness of the proposed method compared with other similar advanced methods and its effective application in non-destructive visual analysis of corn components.

FDDCC-VSR: a lightweight video super-resolution network based on deformable 3D convolution and cheap convolution

FDDCC-VSR: a lightweight video super-resolution network based on deformable 3D convolution and cheap convolution

SRARDA: A lightweight adaptive residual dense attention generative adversarial network for image super-resolution

Image super-resolution (SR) is the task of inferring a high resolution (HR) image from one/multiple single low resolution (LR) input(s). Traditional networks are evaluated by pixel-level metrics such as Peak-Signal-to-Noise Ratio (PSNR) etc., which do not always align with human perception of image quality. They often produce excessively smooth images that lack high-frequency texture and appear unnatural. Therefore, in this paper, we propose a lightweight adaptive residual dense attention generative adversarial network (SRARDA) for image SR. Firstly, our generator adopts the residual in residual (RIR) structure but redesigns the basic module. By using dynamic residual connection (ARC) to dynamically adjust the importance of residual and main paths, we design a novel adaptive residual dense attention block (ARDAB) that enhances the feature extraction capability of the generator. In addition, we build a high-frequency filtering unit (HFU) to extract more high-frequency features from the LR space. Finally, to fully utilize the discriminator, we use WGAN to compute the difference between the HR image and the reconstructed image. Experiments demonstrate that SRARDA effectively addresses the issue of excessive smoothing in reconstructed images, while also enhancing visual quality.

Super-resolution reconstruction of hydrate-bearing CT images for microscopic detection of pore

The pore structure of marine natural gas-hydrate-bearing sediments is a key factor related to the physical properties of reservoirs. However, the resolution of micro-computed tomography (micro-CT) images is unsuitable for the analysis of pore structures in fine-grained sediments. In this regard, super-resolution (SR) reconstruction technology is expected to improve the spatial resolution of micro-CT images. We present a self-supervised learning method that does not require high-resolution datasets as input images to complete the training and reconstruction processes. This method is an end-to-end network consisting of two subnetworks: an SR network and a downscaling network. We trained on a self-built dataset of hydrate samples from three different particle sizes. Compared with typical methods, the SR results indicate that our method provides high resolution while improving clarity. In addition, it has the highest consistency with the liquid saturation method with the subsequent calculation of porosity parameters. This study contributes to the investigation of seepage and energy transfer in sediments containing natural gas hydrates, which is particularly important for the exploration and development of marine natural gas hydrate resources.

HASN: hybrid attention separable network for efficient image super-resolution

HASN: hybrid attention separable network for efficient image super-resolution

Multibeam water column image super-resolution by combining morphological and intensity features

ABSTRACT Multibeam water column images (WCI) play a crucial role in the detection and recognition of underwater targets. However, WCI is not a complete image of underwater space due to the gaps between beams inherent in multibeam systems. In this study, we design the WCI super-resolution network (WCISRN) based on Swin-transformer to adapt to the uneven and sparse distribution of water column data. To enable WCISRN to learn morphological features effectively, we produced simulated datasets for pre-training. Subsequently, we conduct self-supervised training on the measured WCI dataset to fine-tune the network parameters, aiming to better adapt WCISRN to the backscattering intensity features of WCI. Finally, by performing network cross-fusion, the pre-trained network and self-supervised training network are combined to form a new network that achieves a balance between image morphology and intensity features. The proposed method is evaluated through visual analysis and quantitative comparison using simulated and real-world datasets. The experimental results demonstrate that our method exhibits stronger detail restoration ability and clearer image morphology. The average peak signal-to-noise ratio (PSNR) on the test set improved from 23.75–26.14, indicating a significant enhancement in the imaging quality of WCI.

Cross-Dimensional Attention Fusion Network for Simulated Single Image Super-Resolution

Cross-Dimensional Attention Fusion Network for Simulated Single Image Super-Resolution

Super-resolution Imaging of Telescopic Systems based on Optical-neural Network Joint Optimization

Optical telescopes are an important tool for acquiring optical information about distant objects, and resolution is an important indicator that measures the ability to observe object details. However, due to the effects of system aberration, atmospheric seeing, and other factors, the observed image of ground-based telescopes is often degraded, resulting in reduced resolution. This paper proposes an optical-neural network joint optimization method to improve the resolution of the observed image by co-optimizing the point-spread function (PSF) of the telescopic system and the image super-resolution (SR) network. To improve the speed of image reconstruction, we designed a generative adversarial net (LCR-GAN) with light parameters, which is much faster than the latest unsupervised networks. To reconstruct the PSF trained by the network in the optical path, a phase mask is introduced. It improves the image reconstruction effect of LCR-GAN by reconstructing the PSF that best matches the network. The results of simulation and verification experiments show that compared with the pure deep learning method, the SR image reconstructed by this method is rich in detail and it is easier to distinguish stars or stripes.

Accurate recognition of micromorphology images of epoxy coatings for deep-sea environments based on a deep learning super-resolution method

Crack initiation and extension occur in organic coatings during service in deep-sea environments. However, when extracting detailed crack information from SEM images of epoxy mica coatings at different time periods in a simulated deep-sea fluid-hydraulic environment, uninteresting background regions are treated equally, resulting in unnecessary computational redundancy. To address the relatively blurred edges and unclear textures of SEM images, a crack image super-resolution network based on global mixed attention (GMA-net) is proposed for application to SEM images of organic coatings. The recognition results of the images processed with GMA-net were compared with those of the original images and the images processed with bicubic method, respectively. The results show that this method not only refrains from destroying the clarity of the original images but also greatly outperforms bicubic method in terms of precision, recall, mAP50 and mPA50-95, which are improved by approximately 23.1%, 32.4%, 36.4% and 26.7%, respectively. This method effectively highlights the details and improves the recognition accuracy of the edge texture with the aim of providing a good basis for subsequent recognition and even lifetime prediction studies.

The destruction of cytoplasmic skeleton leads to the change of nuclear structure and the looseness of lamin A submicroscopic network

The interaction between lamin A and the cytoplasmic skeleton plays a key role in maintaining nuclear mechanical properties. However, the effect of destruction of the cytoplasmic skeleton on the 3D submicroscopic structure of lamin A has not been elucidated. In this study, we developed an image quantization algorithm to quantify changes in the submicroscopic structure of the intact lamin A 3D network within the nucleus. We used blebbistatin or nocodazole to disrupt the fibrillar structure of F-actin or tubulin, respectively, and then quantified changes in the lamin A super-resolution network structure, the morphological and mechanical properties of the nucleus and the spatial distribution of chromosomes. Ultimately, we found for the first time that disruption of the cytoplasmic skeleton changes the lamin A submicroscopic network and nuclear structural characteristics. In summary, this study contributes to understanding the trans-nuclear membrane interaction characteristics of lamin A and the cytoplasmic skeleton.

Squeeze & Excitation joint with Combined Channel and Spatial Attention for Pathology Image Super-Resolution

Super-resolution (SR) techniques are pivotal in enhancing low-resolution images and crucial in medical diagnosis, where detail and clarity are paramount. Traditional pixel-loss-based SR methods, while adept at producing high-resolution (HR) images, often result in artifice content. This loss of information compromises both the visual experience and the accuracy of subsequent diagnoses. Addressing this, we have developed an innovative SR approach integrating a joint Squeeze and Excitation (SE) mechanism with a Combined Channel and Spatial Attention (CCSA) mechanism. The SE mechanism effectively recalibrates channel-wise feature responses, enhancing the representational capacity of the network. Meanwhile, the CCSA mechanism focuses on extracting spatial and channel-wise features, ensuring that critical high-frequency details are preserved. The dual approach significantly refines the quality of the images, maintaining essential details necessary for accurate medical diagnosis. To validate our proposed approach, we used a benchmark dataset, bcSR, tailored to challenge SR models to focus on broader and more critical regions. Comparative analysis proves that our model excels in performance over existing state-of-the-art methods. In conclusion, our proposed SR Network, with its innovative SE and CCSA mechanisms, offers a potent tool for pathology image SR. It elevates the quality of super-resolved images, which will significantly aid in the accuracy and efficiency of medical diagnoses, providing a valuable asset to medical professionals.

GlobalSR: Global context network for single image super-resolution via deformable convolution attention and fast Fourier convolution

Vision Transformer have achieved impressive performance in image super-resolution. However, they suffer from low inference speed mainly because of the quadratic complexity of multi-head self-attention (MHSA), which is the key to learning long-range dependencies. On the contrary, most CNN-based methods neglect the important effect of global contextual information, resulting in inaccurate and blurring details. If one can make the best of both Transformers and CNNs, it will achieve a better trade-off between image quality and inference speed. Based on this observation, firstly assume that the main factor affecting the performance in the Transformer-based SR models is the general architecture design, not the specific MHSA component. To verify this, some ablation studies are made by replacing MHSA with large kernel convolutions, alongside other essential module replacements. Surprisingly, the derived models achieve competitive performance. Therefore, a general architecture design GlobalSR is extracted by not specifying the core modules including blocks and domain embeddings of Transformer-based SR models. It also contains three practical guidelines for designing a lightweight SR network utilizing image-level global contextual information to reconstruct SR images. Following the guidelines, the blocks and domain embeddings of GlobalSR are instantiated via Deformable Convolution Attention Block (DCAB) and Fast Fourier Convolution Domain Embedding (FCDE), respectively. The instantiation of general architecture, termed GlobalSR-DF, proposes a DCA to extract the global contextual feature by utilizing Deformable Convolution and a Hadamard product as the attention map at the block level. Meanwhile, the FCDE utilizes the Fast Fourier to transform the input spatial feature into frequency space and then extract image-level global information from it by convolutions. Extensive experiments demonstrate that GlobalSR is the key part in achieving a superior trade-off between SR quality and efficiency. Specifically, our proposed GlobalSR-DF outperforms state-of-the-art CNN-based and ViT-based SISR models regarding accuracy-speed trade-offs with sharp and natural details.

Multi-scale transformer network for super-resolution of visible and thermal air images

Reference image-based Super-Resolution (RefSR) is introduced to improve the quality of a Low-resolution (LR) input image by leveraging the additional information provided by a High-Resolution (HR) reference image (Ref). While existing RefSR methods focus on thermal or visible flows separately, they often struggle to enhance the resolution of small objects such as Mini/Micro UAVs (Unmanned Aerial Vehicle) due to the resolution disparities between the input and reference images. To cope with these challenges when dealing with UAV early detection in context of video surveillance, we propose ThermoVisSR, a multiscale texture transformer for enhancing the Super-Resolution (SR) of visible and thermal images of Mini/Micro UAVs. Our approach tries to reconstruct the fine details of these objects while preserving their approximation (the body form and color of the different scene objects) already contained in the LR image. Hence, our model is divided up into two streams dealing separately with approximation and detail reconstruction. In the first one, we introduce a Convolution Neural Network (CNN) fusion backbone to extract the Low-Frequency (LF) approximation from the original LR image pairs. In the second one and to extract the details from the Ref image, our approach involves blending features from both visible and thermal sources to make the most of what each offer. Subsequently, we introduce the High-Frequency Texture Transformer (HFTT) across various resolutions of the merged features to ensure an accurate correspondence matching and significant transfer of High-Frequency (HF) patches from Ref to LR images. Moreover, to adapt the injection to the different bands well, we incorporate the separable software decoder (SSD) into the HFTT allowing to capture channel-specific details during the reconstruction phase. We validated our approach using a newly created dataset of Air images of Mini/Micro UAVs. Experimental results demonstrate that the proposed model consistently outperforms the state-of-the-art approaches on both qualitative and quantitative assessments.

Rectified Binary Network for Single-Image Super-Resolution.

Binary neural network (BNN) is an effective approach to reduce the memory usage and the computational complexity of full-precision convolutional neural networks (CNNs), which has been widely used in the field of deep learning. However, there are different properties between BNNs and real-valued models, making it difficult to draw on the experience of CNN composition to develop BNN. In this article, we study the application of binary network to the single-image super-resolution (SISR) task in which the network is trained for restoring original high-resolution (HR) images. Generally, the distribution of features in the network for SISR is more complex than those in recognition models for preserving the abundant image information, e.g., texture, color, and details. To enhance the representation ability of BNN, we explore a novel activation-rectified inference (ARI) module that achieves a more complete representation of features by combining observations from different quantitative perspectives. The activations are divided into several parts with different quantification intervals and are inferred independently. This allows the binary activations to retain more image detail and yield finer inference. In addition, we further propose an adaptive approximation estimator (AAE) for gradually learning the accurate gradient estimation interval in each layer to alleviate the optimization difficulty. Experiments conducted on several benchmarks show that our approach is able to learn a binary SISR model with superior performance over the state-of-the-art methods. The code will be released at https://github.com/jwxintt/Rectified-BSR.