Super-resolution Network Research Articles

Deep learning enables the use of ultra-high-density array in DNBSEQ.

DNBSEQ employs a patterned array to facilitate massively parallel sequencing of DNA nanoballs (DNBs), leading to a considerable boost in throughput. By employing the ultra-high-density (UHD) array with an increased density of DNB binding sites, the throughput of DNBSEQ can be further expanded. However, the typical imaging system of the DNBSEQ sequencer is unable to resolve adjacent DNBs spaced smaller than the resolution limit, resulting in poor base-calling performance of the UHD array and hindering its practical application. In this study, we propose a deep-learning-based DNB image super-resolution network named DNBSRN to address this problem. DNBSRN has a specifically designed structure for DNB images and employs a histogram-matching-based preprocessing approach. For the eight DNB image datasets generated from the DNBSEQ sequencer using UHD arrays with 360nm pitch, the base-calling performances are significantly improved after super-resolution reconstruction by DNBSRN and reached a comparable level to those of the regular density array. In terms of reconstruction speed, DNBSRN takes only 7.61ms for an input image with 500 × 500 pixels, which minimizes its influence on throughput. Furthermore, compared with state-of-the-art super-resolution networks, DNBSRN demonstrates superior performance in terms of both the quality and speed of DNB image reconstruction. DNBSRN successfully addresses the DNB image super-resolution task. Integrating DNBSRN into the image analysis workflow of DNBSEQ will allow for the application of UHD array, hence enabling a considerable improvement in throughput as well as tremendous savings in unit reagent cost.

Asymmetric Large Kernel Distillation Network for efficient single image super-resolution

Recently, significant advancements have been made in the field of efficient single-image super-resolution, primarily driven by the innovative concept of information distillation. This method adeptly leverages multi-level features to facilitate high-resolution image reconstruction, allowing for enhanced detail and clarity. However, many existing approaches predominantly emphasize the enhancement of distilled features, often overlooking the critical aspect of improving the feature extraction capabilities of the distillation module itself. In this paper, we address this limitation by introducing an asymmetric large-kernel convolution design. By increasing the size of the convolution kernel, we expand the receptive field, which enables the model to more effectively capture long-range dependencies among image pixels. This enhancement significantly improves the model's perceptual ability, leading to more accurate reconstructions. To maintain a manageable level of model complexity, we adopt a lightweight architecture that employs asymmetric convolution techniques. Building on this foundation, we propose the Lightweight Asymmetric Large Kernel Distillation Network (ALKDNet). Comprehensive experiments conducted on five widely recognized benchmark datasets-Set5, Set14, BSD100, Urban100, and Manga109-indicate that ALKDNet not only preserves efficiency but also demonstrates performance enhancements relative to existing super-resolution methods. The average PSNR and SSIM values show improvements of 0.10 dB and 0.0013, respectively, thereby achieving state-of-the art performance.

Residual trio feature network for efficient super-resolution

Deep learning-based approaches have demonstrated impressive performance in single-image super-resolution (SISR). Efficient super-resolution compromises the reconstructed image’s quality to have fewer parameters and Flops. Ensured efficiency in image reconstruction and improved reconstruction quality of the model are significant challenges. This paper proposes a trio branch module (TBM) based on structural reparameterization. TBM achieves equivalence transformation through structural reparameterization operations, which use a complex network structure in the training phase and convert it to a more lightweight structure in the inference, achieving efficient inference while maintaining accuracy. Based on the TBM, we further design a lightweight version of the enhanced spatial attention mini (ESA-mini) and the residual trio feature block (RTFB). Moreover, the multiple RTFBs are combined to construct the residual trio network (RTFN). Finally, we introduce a localized contrast loss for better applicability to the super-resolution task, which enhances the reconstruction quality of the super-resolution model. Experiments show that the RTFN framework proposed in this paper outperforms other state-of-the-art efficient super-resolution methods in terms of inference speed and reconstruction quality.

FLFusionSR: a fast and lightweight fusion and super-resolution network for infrared and visible images on edge devices

FLFusionSR: a fast and lightweight fusion and super-resolution network for infrared and visible images on edge devices

GAN-Based Dual Image Super Resolution for Satellite Imagery Decreasing Radiometric Uncertainty

Abstract. Super resolution for satellite imagery possesses specific challenges due to its unique feature geometry compared to classic computer vision. Specifically, satellite images contain a high amount of relatively small, distributed high-frequency features that are hard to preserve when sampling up to a higher resolution. General adversarial networks (GANs) are suitable for super resolution tasks but need special attention concerning the geometric and radiometric accuracy of the results. We propose a network for versatile satellite imagery super resolution (VSISR) that focuses on high-frequency detail preservation and radiometric consistency. As a novelty, it is able to utilize a reference high resolution image during inference to lower hallucinations without altering the source image’s radiometric characteristics.A GAN that is already handling well high frequencies in standard computer vision cases is adapted for satellite imagery and supplemented with a mixed pixel approach for data augmentation. Training on a diverse RGB dataset from four satellite missions results in a versatile super resolution model that is optimized to preserve radiometric features and minimize hallucinations. Compared to other neural networks for satellite image super resolution in their respective datasets, VSISR performs in the middle regarding mean PSNR (25.30 dB) and SSIM (0.8098). Nevertheless, it is the first time that, using mixed pixel training on a comparably small dataset, this versatility concerning the satellite data source is achieved while maintaining their unique radiometric traits.

Enhanced local multi-windows attention network for lightweight image super-resolution

Since the global self-attention mechanism can capture long-distance dependencies well, Transformer-based methods have achieved remarkable performance in many vision tasks, including single-image super-resolution (SISR). However, there are strong local self-similarities in images, if the global self-attention mechanism is still used for image processing, it may lead to excessive use of computing resources on parts of the image with weak correlation. Especially in the high-resolution large-size image, the global self-attention will lead to a large number of redundant calculations. To solve this problem, we propose the Enhanced Local Multi-windows Attention Network (ELMA), which contains two main designs. First, different from the traditional self-attention based on square window partition, we propose a Multi-windows Self-Attention (M-WSA) which uses a new window partitioning mechanism to obtain different types of local long-distance dependencies. Compared with original self-attention mechanisms commonly used in other SR networks, M-WSA reduces computational complexity and achieves superior performance through analysis and experiments. Secondly, we propose a Spatial Gated Network (SGN) as a feed-forward network, which can effectively overcome the problem of intermediate channel redundancy in traditional MLP, thereby improving the parameter utilization and computational efficiency of the network. Meanwhile, SGN introduces spatial information into the feed-forward network that traditional MLP cannot obtain. It can better understand and use the spatial structure information in the image, and enhances the network performance and generalization ability. Extensive experiments show that ELMA achieves competitive performance compared to state-of-the-art methods while maintaining fewer parameters and computational costs.

Disentangled feature fusion network for lightweight image super-resolution

Recently, the quality of generated images in image super-resolution (SR) has significantly improved due to the widespread application of convolutional neural networks. Existing super-resolution methods often overlook the importance of network width and instead prioritize designing deeper network structures to improve performance. Deeper networks, however, demand greater computational resources and are more challenging to train, making them less suitable for devices with limited performance. To address this issue, we propose a novel method called the disentangled feature fusion network (DFFN). Specifically, we construct dual feature extraction streams (DFES) using the strategies of feature disentanglement and parameter sharing, which increase the network's width without increasing the number of parameters. To facilitate the interaction of information between the dual feature extraction streams, we design a feature interaction module (FIM) that separately enhances high and low-frequency features. Furthermore, a feature fusion module (FFM) is presented to efficiently fuse different levels of high and low-frequency feature information. The proposed DFFN integrates DFES, FIM, and FFM, which not only widens the network without increasing parameters but also minimizes the loss of crucial feature information. Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms other state-of-the-art lightweight super-resolution methods, achieving a superior balance between parameter quantity, reconstruction performance, and model complexity. The code is available at https://github.com/zhoujy-aust/DFFN.

Spectral network based on lattice convolution and adversarial training for noise-robust speech super-resolution.

Speech super-resolution aims to predict a high-resolution speech signal from its low-resolution counterpart. The previous models usually perform this task at a fixed sampling rate, reconstructing only high-frequency spectrogram components and merging them with low-frequency ones in noise-free cases. These methods achieve high accuracy, but they are less effective in real-world settings, where ambient noise and flexible sampling rates are presented. To develop a robust model that fits practical applications, in this work, we introduce Super Denoise Net (SDNet), a neural network for noise-robust super-resolution with flexible input sampling rates. To this end, SDNet's design includes gated and lattice convolution blocks for enhanced repair and temporal-spectral information capture. The frequency transform blocks are employed to model long frequency dependencies, and a multi-scale discriminator is proposed to facilitate the multi-adversarial loss training. The experiments show that SDNet outperforms current state-of-the-art noise-robust speech super-resolution models on multiple test sets, indicating its robustness and effectiveness in real-world scenarios.

Super resolution reconstruction of fluorescence microscopy images by a convolutional network with physical priors

Super-solution fluorescence microscopy, such as single-molecule localization microscopy (SMLM), is effective in observing subcellular structures and achieving excellent enhancement in spatial resolution in contrast to traditional fluorescence microscopy. Recently, deep learning has demonstrated excellent performance in SMLM in solving the trade-offs between spatiotemporal resolution, phototoxicity, and signal intensity. However, most of these researches rely on sufficient and high-quality datasets. Here, we propose a physical priors-based convolutional super-resolution network (PCSR), which incorporates a physical-based loss term and an initial optimization process based on the Wiener filter to create excellent super-resolution images directly using low-resolution images. The experimental results demonstrate that PCSR enables the achievement of a fast reconstruction time of 100 ms and a high spatial resolution of 10 nm by training on a limited dataset, allowing subcellular research with high spatiotemporal resolution, low cell phototoxic illumination, and high accessibility. In addition, the generalizability of PCSR to different live cell structures makes it a practical instrument for diverse cell research.

Dual-path aggregation transformer network for super-resolution with images occlusions and variability

While Transformer-based approaches have recently achieved notable success in super-resolution, their extensive computational requirements impede widespread practical adoption. High-resolution meteorological satellite cloud imagery is essential for weather analysis and forecasting. Enhancing image resolution through super-resolution techniques facilitates the accurate identification and localization of geographic features by meteorological systems. However, current super-resolution methods fail to restore the intricacies of cloud formations and complex regions fully. This research introduces a novel dual-path aggregation Transformer network (DPAT) tailored to enhance the super-resolution of meteorological satellite cloud images. The DPAT network adeptly captures cloud imagery's subtle details and textures, effectively addressing occlusions and the variability inherent in satellite imagery. It bolsters the model's ability to manage the complex attributes of cloud images through the introduction of the Dual-path Aggregation Self-Attention (DASA) mechanism and the Multi-scale Feature Aggregation Block (MFAB), thereby enhancing performance in processing intricate cloud features. The DASA mechanism synthesizes features across spatial, depth, and channel dimensions via a dual-path approach, thoroughly exploiting feature correlations. The MFAB, designed to supplant the multilayer perceptron, incorporates shift convolution and a multi-scale interaction block to augment feature information, compensating for the deficiency in local information absorption due to fixed receptive fields. Experimental outcomes indicate that DPAT delivers superior super-resolution outcomes. With a parameter count of only 32% of the Enhanced Deep Residual Network (EDSR) or 77% of the Image Restoration using Shift Window Transformer (SwinIR), DPAT matches SwinIR's performance on the satellite cloud dataset. Moreover, DPAT balances accuracy and parameter economy across various datasets. This technology is expected to improve image super-resolution capabilities in multiple fields such as human action recognition and industrial recognition, and indirectly improve the accuracy of image perception tasks.

A Feature-Driven Inception Dilated Network for Infrared Image Super-Resolution Reconstruction

Image super-resolution (SR) algorithms based on deep learning yield good visual performances on visible images. Due to the blurred edges and low contrast of infrared (IR) images, methods transferred directly from visible images to IR images have a poor performance and ignore the demands of downstream detection tasks. Therefore, an Inception Dilated Super-Resolution (IDSR) network with multiple branches is proposed. A dilated convolutional branch captures high-frequency information to reconstruct edge details, while a non-local operation branch captures long-range dependencies between any two positions to maintain the global structure. Furthermore, deformable convolution is utilized to fuse features extracted from different branches, enabling adaptation to targets of various shapes. To enhance the detection performance of low-resolution (LR) images, we crop the images into patches based on target labels before feeding them to the network. This allows the network to focus on learning the reconstruction of the target areas only, reducing the interference of background areas in the target areas’ reconstruction. Additionally, a feature-driven module is cascaded at the end of the IDSR network to guide the high-resolution (HR) image reconstruction with feature prior information from a detection backbone. This method has been tested on the FLIR Thermal Dataset and the M3FD Dataset and compared with five mainstream SR algorithms. The final results demonstrate that our method effectively maintains image texture details. More importantly, our method achieves 80.55% mAP, outperforming other methods on FLIR Dataset detection accuracy, and with 74.7% mAP outperforms other methods on M3FD Dataset detection accuracy.

Report on the AAPM grand challenge on deep generative modeling for learning medical image statistics.

The findings of the 2023 AAPM Grand Challenge on Deep Generative Modeling for Learning Medical Image Statistics are reported in this SpecialReport. The goal of this challenge was to promote the development of deep generative models for medical imaging and to emphasize the need for their domain-relevant assessments via the analysis of relevant imagestatistics. As part of this Grand Challenge, a common training dataset and an evaluation procedure was developed for benchmarking deep generative models for medical image synthesis. To create the training dataset, an established 3D virtual breast phantom was adapted. The resulting dataset comprised about 108000 images of size 512 512. For the evaluation of submissions to the Challenge, an ensemble of 10000 DGM-generated images from each submission was employed. The evaluation procedure consisted of two stages. In the first stage, a preliminary check for memorization and image quality (via the Fréchet Inception Distance [FID]) was performed. Submissions that passed the first stage were then evaluated for the reproducibility of image statistics corresponding to several feature families including texture, morphology, image moments, fractal statistics, and skeleton statistics. A summary measure in this feature space was employed to rank the submissions. Additional analyses of submissions was performed to assess DGM performance specific to individual feature families, the four classes in the training data, and also to identify various artifacts. Fifty-eight submissions from 12 unique users were received for this Challenge. Out of these 12 submissions, 9 submissions passed the first stage of evaluation and were eligible for ranking. The top-ranked submission employed a conditional latent diffusion model, whereas the joint runners-up employed a generative adversarial network, followed by another network for image superresolution. In general, we observed that the overall ranking of the top 9 submissions according to our evaluation method (i) did not match the FID-based ranking, and (ii) differed with respect to individual feature families. Another important finding from our additional analyses was that different DGMs demonstrated similar kinds ofartifacts. This Grand Challenge highlighted the need for domain-specific evaluation to further DGM design as well as deployment. It also demonstrated that the specification of a DGM may differ depending on its intendeduse.

Partial convolutional reparameterization network for lightweight image super-resolution

Partial convolutional reparameterization network for lightweight image super-resolution

Estimating the Spectral Response of Eight-Band MSFA One-Shot Cameras Using Deep Learning

Eight-band one-shot MSFA (multispectral filter array) cameras are innovative technologies used to capture multispectral images by capturing multiple spectral bands simultaneously. They thus make it possible to collect detailed information on the spectral properties of the observed scenes economically. These cameras are widely used for object detection, material analysis, and agronomy. The evolution of one-shot MSFA cameras from 8 to 32 bands makes obtaining much more detailed spectral data possible, which is crucial for applications requiring delicate and precise analysis of the spectral properties of the observed scenes. Our study aims to develop models based on deep learning to estimate the spectral response of this type of camera and provide images close to the spectral properties of objects. First, we prepare our experiment data by projecting them to reflect the characteristics of our camera. Next, we harness the power of deep super-resolution neural networks, such as very deep super-resolution (VDSR), Laplacian pyramid super-resolution networks (LapSRN), and deeply recursive convolutional networks (DRCN), which we adapt to approximate the spectral response. These models learn the complex relationship between 8-band multispectral data from the camera and 31-band multispectral data from the multi-object database, enabling accurate and efficient conversion. Finally, we evaluate the images’ quality using metrics such as loss function, PSNR, and SSIM. The model evaluation revealed that DRCN outperforms others in crucial performance. DRCN achieved the lowest loss with 0.0047 and stood out in image quality metrics, with a PSNR of 25.5059, SSIM of 0.8355, and SAM of 0.13215, indicating better preservation of details and textures. Additionally, DRCN showed the lowest RMSE 0.05849 and MAE 0.0415 values, confirming its ability to minimize reconstruction errors more effectively than VDSR and LapSRN.

Enhanced Feature Refinement Network Based on Depthwise Separable Convolution for Lightweight Image Super-Resolution

Image super-resolution (SR) techniques aim to enhance the clarity and realism of images. Recently, a wide range of excellent SR algorithms with powerful characterization capabilities have emerged and are widely used. However, there are still challenges and room for improvement in designing a lighter and more edge-friendly SR networks for hardware devices. In this paper, we propose a lightweight enhanced feature refinement network (EFRN) based on depthwise separatable convolution for SR reconstruction. The core network components consist of multiple enhanced feature refinement blocks (EFRB), which fully fuse channel features to extract more accurate low-frequency information based on the attention of different channels. In addition, a lightweight residual block (LRB) and a lightweight dual attention block (LDAB) are designed to enhance network information extraction with minimal parameter cost. We improve the feature refinement by using 1 × 1 convolution instead of a channel selection operation to reduce the dimensionality of the features and extract the refined features more efficiently. Finally, to achieve better reconstruction performance, the depth and number of channels of the network are expanded while keeping the total number of parameters at a low level. Extensive experiments have been conducted to demonstrate the superiority of our EFRN over other mainstream SR algorithms in terms of reconstruction results and the number of parameters.

Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach.

Recent progress in single-image super-resolution (SISR) has achieved remarkable performance, yet the computational costs of these methods remain a challenge for deployment on resource-constrained devices. In particular, transformer-based methods, which leverage self-attention mechanisms, have led to significant breakthroughs but also introduce substantial computational costs. To tackle this issue, we introduce the Convolutional Transformer layer (ConvFormer) and propose a ConvFormer-based Super-Resolution network (CFSR), offering an effective and efficient solution for lightweight image super-resolution. The proposed method inherits the advantages of both convolution-based and transformer-based approaches. Specifically, CFSR utilizes large kernel convolutions as a feature mixer to replace the self-attention module, efficiently modeling long-range dependencies and extensive receptive fields with minimal computational overhead. Furthermore, we propose an edge-preserving feed-forward network (EFN) designed to achieve local feature aggregation while effectively preserving high-frequency information. Extensive experiments demonstrate that CFSR strikes an optimal balance between computational cost and performance compared to existing lightweight SR methods. When benchmarked against state-of-the-art methods such as ShuffleMixer, the proposed CFSR achieves a gain of 0.39 dB on the Urban100 dataset for the x2 super-resolution task while requiring 26% and 31% fewer parameters and FLOPs, respectively. The code and pre-trained models are available at https://github.com/Aitical/CFSR.

A multi-level text feature enhancement super-resolution network via group convolution and wavelet transform

A multi-level text feature enhancement super-resolution network via group convolution and wavelet transform

Infrared image super-resolution reconstruction based on visible light image guidance and recursive fusion

Due to the limitations of hardware equipment, infrared images generally have problems such as low resolution, blurred details and poor visual quality during acquisition. Using visible light images to guide the super-resolution reconstruction of infrared images is an effective way to improve the resolution of infrared images. However, the imaging principles of visible light images and infrared images are different, resulting in differences in detail information between the two images, so problems such as blur and ghosting may occur during reconstruction. This paper proposes an infrared image super-resolution network based on visible light image guidance and recursive fusion. In this network, a flow Fourier residual module is designed, and modules of different depths are used to extract information of different frequencies in visible light images and infrared images, so that each module focuses on the appropriate frequency information. At the same time, a hybrid attention module is used to obtain detail information in multimodal images from channel and spatial perspectives, and fuse them in a complementary way, which helps to eliminate the generation of artifacts. On this basis, a global recursive fusion branch is designed to consider the correlation between multi-layer features, adaptively fuse multi-layer features, and generate clearer high-resolution infrared images. Experimental results show that compared with the comparative methods, this method performs better in objective evaluation indicators; in terms of subjective visual comparison, the images reconstructed by this method have clearer textures and fewer artifacts, and better object distinction in complex environments.

Detection of tea leaf blight in UAV remote sensing images by integrating super-resolution and detection networks.

Tea leaf blight (TLB) is a common disease of tea plants and is widely distributed in tea gardens. Although the use of unmanned aerial vehicle (UAV) remote sensing can help to achieve a wider scale for TLB detection, the blurring of UAV images, overlapping of tea leaves, and small size of TLB spots pose significant challenges to the task of detection. This study proposes a method of detecting TLB in UAV remote sensing images by integrating super-resolution (SR) and detection networks. We use an SR network called SERB-Swin2sr to reconstruct the detailed features of UAV images and solve the problem of detail loss caused by the blurring in UAV images. In SERB-Swin2sr, a squeeze-and-excitation ResNet block (SERB) is introduced to enhance the models' ability to extract the target details in the images, and the convolution stem replaces the convolution block in order to increase the convergence rate and stability of the network. A detection network called SDDA-YOLO is applied to achieve precise detection of TLB in UAV remote sensing images. In SDDA-YOLO, a shuffle dual-dimensional attention (SDDA) module is introduced to enhance the feature fusion capability of the network, and an Xsmall-scale detection layer is used to enhance the detection ability of small lesions. Experimental results show that the proposed method is superior to current detection methods. Compared with a baseline YOLOv8 model, the precision, mAP@0.5, and mAP@0.5:0.95 of the proposed method are improved by 4.2%, 1.6%, and 1.8%, and the size of our model is only 4.6MB.

Image super-resolution based on improved ESRGAN and its application in camera calibration

Camera calibration is a key technique in the field of computer vision. The Zhang’s method is currently the most commonly used camera calibration method, but its accuracy is mainly dependent on the image quality, thusly constrained by camera hardware conditions. Considering that high-performance cameras are often expensive, a cost-effective method to improve the quality of calibration images is of great practical value. Without changing the existing hardware conditions, image super-resolution can be considered to enhance the quality of the calibration image. Image super-resolution (SR) is the process of reconstructing an image from low resolution (LR) to high resolution (HR). It can enhance the clarity and detail of the calibration images, which will be beneficial in improving the accuracy of camera calibration. However, there is very little research on the application of image super-resolution to camera calibration. Therefore, this study innovatively proposed a method and process for applying image super-resolution in camera calibration. Firstly, the ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) was optimized and improved, followed by the proposal of the MMRSRGAN (Multi-scale Multi-adaptive-weights Residual Super-Resolution Generative Adversarial Networks). Two methods for local image super-resolution were also proposed for efficient training and testing of datasets, and two evaluation metrics based on reprojection error were proposed evaluate the effectiveness of the proposed model. Training and testing experiments of image super-resolution for camera calibration were conducted on both virtual and real image datasets. The proposed MMRSRGAN was compared with several advanced image super-resolution networks developed in recent years across multiple dimensions and indicators. The positive results have demonstrated the feasibility of applying image super-resolution to camera calibration and showcased the good performance of the MMRSRGAN.