Super-resolution Network Research Articles

Neutron-Image Super-Resolution based on Convolutional Neural Networks and Novel Parallel Transformer

Enhancing the resolution of neutron imaging can be achieved by upgrading the hardware components of the neutron imaging system, including employing more potent neutron source systems, utilizing high-resolution detectors, and implementing superior collimation techniques. However, such improvements are frequently constrained by challenges such as high cost and limited flexibility. In this paper, a flexible and efficient single-neutron-image super-resolution network is proposed for neutron image super-resolution. We propose a novel multi-scale feature extraction network, which can extract features deeply by fusing features of different types and levels. In order to learn more context information, a novel parallel Transformer is proposed to overcome the long dependency problem of sequences. A high-frequency information enhancement network is proposed to enhance the high-frequency information of neutron image in order to address the blur problem. Quantitative and qualitative analyses of benchmark datasets and real neutron images show that the proposed network performs better on Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) than bicubic interpolation and other methods, and reconstructs high-resolution neutron images that are more consistent with human visual perception, highlighting its application potential in neutron images quality enhancement.

CFDN: cross-scale feature distillation network for lightweight single image super-resolution

CFDN: cross-scale feature distillation network for lightweight single image super-resolution

ITSRN++: Stronger and better implicit transformer network for screen content image continuous super-resolution

Nowadays, online screen sharing and remote cooperation are becoming ubiquitous. However, the screen content may be downsampled and compressed during transmission, while it may be displayed on large screens or the users would zoom in for detail observation at the receiver side. Therefore, developing a strong and effective screen content image (SCI) super-resolution (SR) method is demanded. We observe that the weight-sharing upsampler (such as deconvolution or pixel shuffle) could be harmful to sharp and thin edges in SCIs, and the fixed scale upsampler makes it inflexible to fit screens with various sizes. To solve this problem, we propose an implicit transformer network for SCI continuous SR (termed as ITSRN++). Specifically, we propose a modulation based transformer as the upsampler, which modulates the pixel features in discrete space via a periodic nonlinear function to generate features in continuous space. To better restore the high-frequency details in screen content images, we further propose dual branch block (DBB) as the feature extraction backbone, where convolution and attention branches are utilized parallelly on the same linear transformed value. Besides, we construct a large-scale SCI2K dataset to facilitate the research on SCI SR. Experimental results on nine datasets demonstrate that the proposed method achieves state-of-the-art performance for SCI SR and also works well for natural image SR.

Adding Noise to Super-Resolution Training Set: Method to Denoise Super Resolution for Structure from Motion Preprocessing

The resolution and noise levels of input images directly affect the three-dimensional (3D) structure-from-motion (SfM) reconstruction performance. Conventional super-resolution (SR) methods focus too little on denoising, and latent image noise becomes worse when resolution is improved. This study proposes two SR denoising training algorithms to simultaneously improve resolution and noise: add-noise-before-downsampling and downsample-before-adding-noise. These portable methods preprocess low-resolution training images using real-world noise samples instead of altering the basic neural network. Hence, they concurrently improve resolution while reducing noise for an overall cleaner SfM performance. We applied these methods to the existing SR network: super-resolution convolutional neural network, enhanced deep residual super-resolution, residual channel attention network, and efficient super-resolution transformer, comparing their performances with those of conventional methods. Impressive peak signal-to-noise and structural similarity improvements of 0.12 dB and 0.56 were achieved on the noisy images of Smartphone Image Denoising Dataset, respectively, without altering the network structure. The proposed methods caused a very small loss (<0.01 dB) on clean images. Moreover, using the proposed SR algorithm makes the 3D SfM reconstruction more complete. Upon applying the methods to non-preprocessed and conventionally preprocessed models, the mean projection error was reduced by a maximum of 27% and 4%, respectively, and the number of 3D densified points was improved by 310% and 7%, respectively.

Lightweight image super-resolution network based on dynamic graph message passing and convolution mixer

The single image super-resolution (SISR) task much consideration in the last few years. There are many image SR models that are developed based on advanced deep learning methods. However, these models extract features in image space limiting the ability to extract short and long dependencies. To solve this limitation, a graph mixer block (GMB) is designed based on merging the dynamic graph message passing network (DGMN) with the convolution mixer (ConvMix) for the task of image super-resolution (SR). By utilizing the designed GMB block and the concept of residual learning, we developed a graph mixer network (GMN) for SISR. The idea of this model is the integration to benefit from both types of graph features and traditional convolution features. Combining these features allows the model to capture both global graph structure and local patterns, leading to more comprehensive and expressive representations. The function of the DGMN is used to capture long-range structured relationships with efficient weight modules. This DGMN is conditioned on the input to make adaptively sampling nodes in the graph used for message passing. Then, the ConvMix is used to extract additional features that not is extracted by DGMN modules based on channel and spatial mixing. The ConvMix is designed based on using two depthwise convolutions with one pointwise convolution. The result shows that our GMN attained state-of-the-art results in multiple SR benchmarks

Deep learning enables the use of ultra-high-density array in DNBSEQ

DNBSEQ employs a patterned array to facilitate massively parallel sequencing of DNA nanoballs (DNBs), leading to a considerable boost in throughput. By employing the ultra-high-density (UHD) array with an increased density of DNB binding sites, the throughput of DNBSEQ can be further expanded. However, the typical imaging system of the DNBSEQ sequencer is unable to resolve adjacent DNBs spaced smaller than the resolution limit, resulting in poor base-calling performance of the UHD array and hindering its practical application. In this study, we propose a deep-learning-based DNB image super-resolution network named DNBSRN to address this problem. DNBSRN has a specifically designed structure for DNB images and employs a histogram-matching-based preprocessing approach. For the eight DNB image datasets generated from the DNBSEQ sequencer using UHD arrays with 360 nm pitch, the base-calling performances are significantly improved after super-resolution reconstruction by DNBSRN and reached a comparable level to those of the regular density array. In terms of reconstruction speed, DNBSRN takes only 7.61 ms for an input image with 500 × 500 pixels, which minimizes its influence on throughput. Furthermore, compared with state-of-the-art super-resolution networks, DNBSRN demonstrates superior performance in terms of both the quality and speed of DNB image reconstruction. DNBSRN successfully addresses the DNB image super-resolution task. Integrating DNBSRN into the image analysis workflow of DNBSEQ will allow for the application of UHD array, hence enabling a considerable improvement in throughput as well as tremendous savings in unit reagent cost.

Asymmetric Large Kernel Distillation Network for efficient single image super-resolution.

Recently, significant advancements have been made in the field of efficient single-image super-resolution, primarily driven by the innovative concept of information distillation. This method adeptly leverages multi-level features to facilitate high-resolution image reconstruction, allowing for enhanced detail and clarity. However, many existing approaches predominantly emphasize the enhancement of distilled features, often overlooking the critical aspect of improving the feature extraction capabilities of the distillation module itself. In this paper, we address this limitation by introducing an asymmetric large-kernel convolution design. By increasing the size of the convolution kernel, we expand the receptive field, which enables the model to more effectively capture long-range dependencies among image pixels. This enhancement significantly improves the model's perceptual ability, leading to more accurate reconstructions. To maintain a manageable level of model complexity, we adopt a lightweight architecture that employs asymmetric convolution techniques. Building on this foundation, we propose the Lightweight Asymmetric Large Kernel Distillation Network (ALKDNet). Comprehensive experiments conducted on five widely recognized benchmark datasets-Set5, Set14, BSD100, Urban100, and Manga109-indicate that ALKDNet not only preserves efficiency but also demonstrates performance enhancements relative to existing super-resolution methods. The average PSNR and SSIM values show improvements of 0.10 dB and 0.0013, respectively, thereby achieving state-of-the art performance.

Residual trio feature network for efficient super-resolution

Deep learning-based approaches have demonstrated impressive performance in single-image super-resolution (SISR). Efficient super-resolution compromises the reconstructed image’s quality to have fewer parameters and Flops. Ensured efficiency in image reconstruction and improved reconstruction quality of the model are significant challenges. This paper proposes a trio branch module (TBM) based on structural reparameterization. TBM achieves equivalence transformation through structural reparameterization operations, which use a complex network structure in the training phase and convert it to a more lightweight structure in the inference, achieving efficient inference while maintaining accuracy. Based on the TBM, we further design a lightweight version of the enhanced spatial attention mini (ESA-mini) and the residual trio feature block (RTFB). Moreover, the multiple RTFBs are combined to construct the residual trio network (RTFN). Finally, we introduce a localized contrast loss for better applicability to the super-resolution task, which enhances the reconstruction quality of the super-resolution model. Experiments show that the RTFN framework proposed in this paper outperforms other state-of-the-art efficient super-resolution methods in terms of inference speed and reconstruction quality.

Untrained Network for Super-resolution for Non-contrast-enhanced Wholeheart MRI Acquired using Cardiac-triggered REACT (SRNN-REACT).

Three-dimensional (3D) whole-heart magnetic resonance imaging (MRI) is an excellent tool to check the heart anatomy of patients with congenital and acquired heart disease. However, most 3D whole-heart MRI acquisitions take a long time to perform, and the sequence used is susceptible to banding artifacts. To validate an unsupervised neural network that can reduce acquisition time and improve image quality for 3D whole-heart MRI by superresolving low-resolution images. The results of the super-resolution neural network (SRNN) were compared with bilinear interpolation, a state-of-the-art method known as AdapSR, and the ground truth high-resolution images qualitatively and quantitatively. Thirty pediatric patients with varying congenital and acquired heart diseases were included in this study. Results from the SRNN without a ground truth image were compared qualitatively with the contrast-enhanced whole-heart images. Signal-to-noise ratio (SNR) was used to quantitatively compare each of the methods and the high-resolution ground truth. As confirmed by both the quantitative and qualitative results, the SRNN improves image quality. Furthermore, because it only requires a lowresolution acquisition, the use of the SRNN reduces acquisition time. The SRNN lessens noise and eliminates artifacts while maintaining correct anatomical structure in the images.

Enhanced local multi-windows attention network for lightweight image super-resolution

Since the global self-attention mechanism can capture long-distance dependencies well, Transformer-based methods have achieved remarkable performance in many vision tasks, including single-image super-resolution (SISR). However, there are strong local self-similarities in images, if the global self-attention mechanism is still used for image processing, it may lead to excessive use of computing resources on parts of the image with weak correlation. Especially in the high-resolution large-size image, the global self-attention will lead to a large number of redundant calculations. To solve this problem, we propose the Enhanced Local Multi-windows Attention Network (ELMA), which contains two main designs. First, different from the traditional self-attention based on square window partition, we propose a Multi-windows Self-Attention (M-WSA) which uses a new window partitioning mechanism to obtain different types of local long-distance dependencies. Compared with original self-attention mechanisms commonly used in other SR networks, M-WSA reduces computational complexity and achieves superior performance through analysis and experiments. Secondly, we propose a Spatial Gated Network (SGN) as a feed-forward network, which can effectively overcome the problem of intermediate channel redundancy in traditional MLP, thereby improving the parameter utilization and computational efficiency of the network. Meanwhile, SGN introduces spatial information into the feed-forward network that traditional MLP cannot obtain. It can better understand and use the spatial structure information in the image, and enhances the network performance and generalization ability. Extensive experiments show that ELMA achieves competitive performance compared to state-of-the-art methods while maintaining fewer parameters and computational costs.

FLFusionSR: a fast and lightweight fusion and super-resolution network for infrared and visible images on edge devices

FLFusionSR: a fast and lightweight fusion and super-resolution network for infrared and visible images on edge devices

GAN-Based Dual Image Super Resolution for Satellite Imagery Decreasing Radiometric Uncertainty

Abstract. Super resolution for satellite imagery possesses specific challenges due to its unique feature geometry compared to classic computer vision. Specifically, satellite images contain a high amount of relatively small, distributed high-frequency features that are hard to preserve when sampling up to a higher resolution. General adversarial networks (GANs) are suitable for super resolution tasks but need special attention concerning the geometric and radiometric accuracy of the results. We propose a network for versatile satellite imagery super resolution (VSISR) that focuses on high-frequency detail preservation and radiometric consistency. As a novelty, it is able to utilize a reference high resolution image during inference to lower hallucinations without altering the source image’s radiometric characteristics.A GAN that is already handling well high frequencies in standard computer vision cases is adapted for satellite imagery and supplemented with a mixed pixel approach for data augmentation. Training on a diverse RGB dataset from four satellite missions results in a versatile super resolution model that is optimized to preserve radiometric features and minimize hallucinations. Compared to other neural networks for satellite image super resolution in their respective datasets, VSISR performs in the middle regarding mean PSNR (25.30 dB) and SSIM (0.8098). Nevertheless, it is the first time that, using mixed pixel training on a comparably small dataset, this versatility concerning the satellite data source is achieved while maintaining their unique radiometric traits.

Disentangled feature fusion network for lightweight image super-resolution

Recently, the quality of generated images in image super-resolution (SR) has significantly improved due to the widespread application of convolutional neural networks. Existing super-resolution methods often overlook the importance of network width and instead prioritize designing deeper network structures to improve performance. Deeper networks, however, demand greater computational resources and are more challenging to train, making them less suitable for devices with limited performance. To address this issue, we propose a novel method called the disentangled feature fusion network (DFFN). Specifically, we construct dual feature extraction streams (DFES) using the strategies of feature disentanglement and parameter sharing, which increase the network's width without increasing the number of parameters. To facilitate the interaction of information between the dual feature extraction streams, we design a feature interaction module (FIM) that separately enhances high and low-frequency features. Furthermore, a feature fusion module (FFM) is presented to efficiently fuse different levels of high and low-frequency feature information. The proposed DFFN integrates DFES, FIM, and FFM, which not only widens the network without increasing parameters but also minimizes the loss of crucial feature information. Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms other state-of-the-art lightweight super-resolution methods, achieving a superior balance between parameter quantity, reconstruction performance, and model complexity. The code is available at https://github.com/zhoujy-aust/DFFN.

Spectral network based on lattice convolution and adversarial training for noise-robust speech super-resolution.

Speech super-resolution aims to predict a high-resolution speech signal from its low-resolution counterpart. The previous models usually perform this task at a fixed sampling rate, reconstructing only high-frequency spectrogram components and merging them with low-frequency ones in noise-free cases. These methods achieve high accuracy, but they are less effective in real-world settings, where ambient noise and flexible sampling rates are presented. To develop a robust model that fits practical applications, in this work, we introduce Super Denoise Net (SDNet), a neural network for noise-robust super-resolution with flexible input sampling rates. To this end, SDNet's design includes gated and lattice convolution blocks for enhanced repair and temporal-spectral information capture. The frequency transform blocks are employed to model long frequency dependencies, and a multi-scale discriminator is proposed to facilitate the multi-adversarial loss training. The experiments show that SDNet outperforms current state-of-the-art noise-robust speech super-resolution models on multiple test sets, indicating its robustness and effectiveness in real-world scenarios.

Super resolution reconstruction of fluorescence microscopy images by a convolutional network with physical priors.

Super-solution fluorescence microscopy, such as single-molecule localization microscopy (SMLM), is effective in observing subcellular structures and achieving excellent enhancement in spatial resolution in contrast to traditional fluorescence microscopy. Recently, deep learning has demonstrated excellent performance in SMLM in solving the trade-offs between spatiotemporal resolution, phototoxicity, and signal intensity. However, most of these researches rely on sufficient and high-quality datasets. Here, we propose a physical priors-based convolutional super-resolution network (PCSR), which incorporates a physical-based loss term and an initial optimization process based on the Wiener filter to create excellent super-resolution images directly using low-resolution images. The experimental results demonstrate that PCSR enables the achievement of a fast reconstruction time of 100 ms and a high spatial resolution of 10 nm by training on a limited dataset, allowing subcellular research with high spatiotemporal resolution, low cell phototoxic illumination, and high accessibility. In addition, the generalizability of PCSR to different live cell structures makes it a practical instrument for diverse cell research.

Dual-path aggregation transformer network for super-resolution with images occlusions and variability

While Transformer-based approaches have recently achieved notable success in super-resolution, their extensive computational requirements impede widespread practical adoption. High-resolution meteorological satellite cloud imagery is essential for weather analysis and forecasting. Enhancing image resolution through super-resolution techniques facilitates the accurate identification and localization of geographic features by meteorological systems. However, current super-resolution methods fail to restore the intricacies of cloud formations and complex regions fully. This research introduces a novel dual-path aggregation Transformer network (DPAT) tailored to enhance the super-resolution of meteorological satellite cloud images. The DPAT network adeptly captures cloud imagery's subtle details and textures, effectively addressing occlusions and the variability inherent in satellite imagery. It bolsters the model's ability to manage the complex attributes of cloud images through the introduction of the Dual-path Aggregation Self-Attention (DASA) mechanism and the Multi-scale Feature Aggregation Block (MFAB), thereby enhancing performance in processing intricate cloud features. The DASA mechanism synthesizes features across spatial, depth, and channel dimensions via a dual-path approach, thoroughly exploiting feature correlations. The MFAB, designed to supplant the multilayer perceptron, incorporates shift convolution and a multi-scale interaction block to augment feature information, compensating for the deficiency in local information absorption due to fixed receptive fields. Experimental outcomes indicate that DPAT delivers superior super-resolution outcomes. With a parameter count of only 32% of the Enhanced Deep Residual Network (EDSR) or 77% of the Image Restoration using Shift Window Transformer (SwinIR), DPAT matches SwinIR's performance on the satellite cloud dataset. Moreover, DPAT balances accuracy and parameter economy across various datasets. This technology is expected to improve image super-resolution capabilities in multiple fields such as human action recognition and industrial recognition, and indirectly improve the accuracy of image perception tasks.

Digging into Depth and Color Spaces: A Mapping Constraint Network for Depth Super-Resolution

Scene depth super-resolution (DSR) poses an inherently ill-posed problem due to the extremely large space of one-to-many mapping functions from a given low-resolution (LR) depth map, which possesses limited depth information, to multiple plausible high-resolution (HR) depth maps. This characteristic renders the task highly challenging, as identifying an optimal solution becomes significantly intricate amidst this multitude of potential mappings. While simplistic constraints have been proposed to address the DSR task, the relationship between LR and HR depth maps and the color image has not been thoroughly investigated. In this paper, we introduce a novel mapping constraint network (MCNet) that incorporates additional constraints derived from both LR depth maps and color images. This integration aims to optimize the space of mapping functions and enhance the performance of DSR. Specifically, alongside the primary DSR network (DSRNet) dedicated to learning LR-to-HR mapping, we have developed an auxiliary degradation network (ADNet) that operates in reverse, generating the LR depth map from the reconstructed HR depth map to obtain depth features in LR space. To enhance the learning process of DSRNet in LR-to-HR mapping, we introduce two mapping constraints in LR space: (1) the cycle-consistent constraint, which offers additional supervision by establishing a closed loop between LR-to-HR and HR-to-LR mappings, and (2) the region-level contrastive constraint, aimed at reinforcing region-specific HR representations by explicitly modeling the consistency between LR and HR spaces. To leverage the color image effectively, we introduce a feature screening module to adaptively fuse color features at different layers, which can simultaneously maintain strong structural context and suppress texture distraction through subspace generation and image projection. Comprehensive experimental results across synthetic and real-world benchmark datasets unequivocally demonstrate the superiority of our proposed method over state-of-the-art DSR methods. Our MCNet achieves an average MAD reduction of 3.7% and 7.5% over state-of-the-art DSR method for ×8 and ×16 cases on Milddleburry dataset, respectively, without incurring additional costs during inference.

A Feature-Driven Inception Dilated Network for Infrared Image Super-Resolution Reconstruction

Image super-resolution (SR) algorithms based on deep learning yield good visual performances on visible images. Due to the blurred edges and low contrast of infrared (IR) images, methods transferred directly from visible images to IR images have a poor performance and ignore the demands of downstream detection tasks. Therefore, an Inception Dilated Super-Resolution (IDSR) network with multiple branches is proposed. A dilated convolutional branch captures high-frequency information to reconstruct edge details, while a non-local operation branch captures long-range dependencies between any two positions to maintain the global structure. Furthermore, deformable convolution is utilized to fuse features extracted from different branches, enabling adaptation to targets of various shapes. To enhance the detection performance of low-resolution (LR) images, we crop the images into patches based on target labels before feeding them to the network. This allows the network to focus on learning the reconstruction of the target areas only, reducing the interference of background areas in the target areas’ reconstruction. Additionally, a feature-driven module is cascaded at the end of the IDSR network to guide the high-resolution (HR) image reconstruction with feature prior information from a detection backbone. This method has been tested on the FLIR Thermal Dataset and the M3FD Dataset and compared with five mainstream SR algorithms. The final results demonstrate that our method effectively maintains image texture details. More importantly, our method achieves 80.55% mAP, outperforming other methods on FLIR Dataset detection accuracy, and with 74.7% mAP outperforms other methods on M3FD Dataset detection accuracy.

Report on the AAPM grand challenge on deep generative modeling for learning medical image statistics.

The findings of the 2023 AAPM Grand Challenge on Deep Generative Modeling for Learning Medical Image Statistics are reported in this SpecialReport. The goal of this challenge was to promote the development of deep generative models for medical imaging and to emphasize the need for their domain-relevant assessments via the analysis of relevant imagestatistics. As part of this Grand Challenge, a common training dataset and an evaluation procedure was developed for benchmarking deep generative models for medical image synthesis. To create the training dataset, an established 3D virtual breast phantom was adapted. The resulting dataset comprised about 108000 images of size 512 512. For the evaluation of submissions to the Challenge, an ensemble of 10000 DGM-generated images from each submission was employed. The evaluation procedure consisted of two stages. In the first stage, a preliminary check for memorization and image quality (via the Fréchet Inception Distance [FID]) was performed. Submissions that passed the first stage were then evaluated for the reproducibility of image statistics corresponding to several feature families including texture, morphology, image moments, fractal statistics, and skeleton statistics. A summary measure in this feature space was employed to rank the submissions. Additional analyses of submissions was performed to assess DGM performance specific to individual feature families, the four classes in the training data, and also to identify various artifacts. Fifty-eight submissions from 12 unique users were received for this Challenge. Out of these 12 submissions, 9 submissions passed the first stage of evaluation and were eligible for ranking. The top-ranked submission employed a conditional latent diffusion model, whereas the joint runners-up employed a generative adversarial network, followed by another network for image superresolution. In general, we observed that the overall ranking of the top 9 submissions according to our evaluation method (i) did not match the FID-based ranking, and (ii) differed with respect to individual feature families. Another important finding from our additional analyses was that different DGMs demonstrated similar kinds ofartifacts. This Grand Challenge highlighted the need for domain-specific evaluation to further DGM design as well as deployment. It also demonstrated that the specification of a DGM may differ depending on its intendeduse.

Partial convolutional reparameterization network for lightweight image super-resolution

Partial convolutional reparameterization network for lightweight image super-resolution