Super-resolution Method Research Articles

An efficient blur kernel estimation method for blind image Super-Resolution

Many existing space-variant blind image Super-Resolution (SR) methods require the generation of blur kernels for all input pixels. However, the existing methods overlook the possibility of similar blur kernel patterns among adjacent pixels, leading to spatial incoherence and wasteful computational resources. In this study, we introduce an efficient two-stage method for estimating blur kernels. Instead of generating pixel-wise kernels, a limited set of kernels at pixels with distinct information are generated first and the remaining kernels are reconstructed based on this initial set. This novel method, referred to as Anchor Detection and Kernel Reconstruction, comprises two main components: an Anchor Detection Module (ADM) and a Kernel Reconstruction Module (KRM). The objective of ADM is to identify pixels in a given Low-Resolution image that contain rich information, which are called anchors. The corresponding blur kernels, denoted as anchor kernels, are then generated for these identified pixels using a complete backbone network. The remaining blur kernels are reconstructed using KRM with a lightweight interpolation method based on the anchor kernels to enhance spatial consistency among the reconstructed pixels. Extensive experiments demonstrate that the proposed ADKR method maintains comparable performance while estimating only 50% of blur kernels with a full backbone network, reaching approximately 20% reduction in FLOPs. The code has been made available at https://github.com/xuyimin0926/ADKR.

Ultrastructural 3D Microscopy for Biomedicine: Principles, Applications, and Perspectives.

Modern biomedical research often requires a three-dimensional microscopic analysis of the ultrastructure of biological objects and materials. Conceptual technical and methodological solutions for three-dimensional structure reconstruction are needed to improve the conventional optical, electron, and probe microscopy methods, which to begin with allow one to obtain two-dimensional images and data. This review discusses the principles and potential applications of such techniques as serial section transmission electron microscopy; techniques based on scanning electron microscopy (SEM) (array tomography, focused ion beam SEM, and serial block-face SEM). 3D analysis techniques based on modern super-resolution optical microscopy methods are described (stochastic optical reconstruction microscopy and stimulated emission depletion microscopy), as well as ultrastructural 3D microscopy methods based on scanning probe microscopy and the feasibility of combining them with optical techniques. A comparative analysis of the advantages and shortcomings of the discussed approaches is performed.

Enhanced Solar Coronal Imaging: A GAN Approach with Fused Attention and Perceptual Quality Enhancement

The activity of the solar corona has a significant impact on all aspects of human life. People typically use images obtained from astronomical telescopes to observe coronal activities, among which the Atmospheric Imaging Assembly (AIA) of the Solar Dynamics Observatory (SDO) is particularly widely used. However, due to resolution limitations, we have begun to study the application of generative adversarial network super-resolution techniques to enhance the image data quality for a clearer observation of the fine structures and dynamic processes in the solar atmosphere, which improves the prediction accuracy of solar activities. We aligned SDO/AIA images with images from the High-Resolution Coronal Imager (Hi-C) to create a dataset. This research proposes a new super-resolution method named SAFCSRGAN, which includes a spatial attention module that incorporates channel information, allowing the network model to better capture the corona’s features. A Charbonnier loss function was introduced to enhance the perceptual quality of the super-resolution images. Compared to the original method using ESRGAN, our method achieved an 11.9% increase in Peak Signal-to-Noise Ratio (PSNR) and a 4.8% increase in Structural Similarity (SSIM). Additionally, we introduced two perceptual image quality assessment metrics, the Natural Image Quality Evaluator (NIQE) and Learned Perceptual Image Patch Similarity (LPIPS), which improved perceptual quality by 10.8% and 1.3%, respectively. Finally, our experiments demonstrated that our improved model surpasses other models in restoring the details of coronal images.

Bio-inspired foveal super-resolution method for multi-focal-length images based on local gradient constraints

Most existing super-resolution (SR) imaging systems, inspired by the bionic compound eye, utilize image registration and reconstruction algorithms to overcome the angular resolution limitations of individual imaging systems. This article introduces a multi-aperture multi-focal-length imaging system and a multi-focal-length image super-resolution algorithm, mimicking the foveal imaging of the human eye. Experimental results demonstrate that with the proposed imaging system and an SR imaging algorithm inspired by the human visual system, the proposed method can enhance the spatial resolution of the foveal region by up to 4 × compared to the original acquired image. These findings validate the effectiveness of the proposed imaging system and computational imaging algorithm in enhancing image texture and spatial resolution.

Deep multi-scale feature mixture model for image super-resolution with multiple-focal-length degradation

Single image super-resolution is a classic problem in computer vision. In recent years, deep learning-based models have achieved unprecedented success with this problem. However, most existing deep super-resolution models unavoidably produce degraded results when applied to real-world images captured by cameras with different focal lengths. The degradation in these images is called multiple-focal-length degradation, which is spatially variant and more complicated than the bicubic downsampling degradation. To address such a challenging issue, we propose a multi-scale feature mixture model in this paper. The proposed model can intensively exploit local patterns from different scales for image super-resolution. To improve the performance, we further propose a novel loss function based on the Laplacian pyramid, which guides the model to recover the information separately of different frequency subbands. Comprehensive experiments show that our proposed model has a better ability to preserve the structure of objects and generate high-quality images, leading to the best performance compared with other state-of-the-art deep single image super-resolution methods.

Learning content-aware feature fusion for guided depth map super-resolution

RGB-D data including paired RGB color images and depth maps is widely used in downstream computer vision tasks. However, compared with the acquisition of high-resolution color images, the depth maps captured by consumer-level sensors are always in low resolution. Within decades of research, the most state-of-the-art (SOTA) methods of depth map super-resolution cannot adaptively tune the guidance fusion for all feature positions by channel-wise feature concatenation with spatially sharing convolutional kernels. This paper proposes JTFNet to resolve this issue, which simulates the traditional Joint Trilateral Filter (JTF). Specifically, a novel JTF block is introduced to adaptively tune the fusion pattern between the color features and the depth features for all feature positions. Moreover, based on the variant of JTF block whose target features and guidance features are in the cross-scale shape, the fusion for depth features is performed in a bi-directional way. Therefore, the error accumulation along scales can be effectively mitigated by iteratively HR feature guidance. Compared with the SOTA methods, the sufficient experiment is conducted on the mainstream synthetic datasets and real datasets, i.e., Middlebury, NYU and ToF-Mark, which shows remarkable improvement of our JTFNet.

Enhancing unsupervised rock CT image super-resolution with non-local attention

Traditional low-resolution rock core images fail to capture the details and features of rocks, limiting the accurate analysis of rock properties. Therefore, researchers have proposed supervised and unsupervised digital rock super-resolution reconstruction methods. Supervised methods require paired data and manual annotation, while unsupervised methods generate high-resolution images through self-learning that are suitable for real-world scenarios with a lack of paired data. However, the current unsupervised methods still face challenges in terms of consistency and feature capturing. To address this problem, a novel unsupervised single-image super-resolution model called NL-CycleGAN is proposed. This model employs a non-local attention-guided CycleGAN framework to effectively capture low-level pixel variations between unpaired source and target images while preserving the overall tone. To evaluate the performance of NL-CycleGAN, we conduct both quantitative and qualitative tests using the DeepRock-SR dataset. In terms of quantitative evaluation, our method achieved the lowest perceptual loss (LPIPS) metric compared to existing methods. Additionally, our model obtained the highest performance in peak signal-to-noise ratio (PSNR), and multi-scale structural similarity index measure (MS-SSIM). In qualitative testing, the comparison images clearly show that our model accurately captures the intricate details of pores in carbonate and sandstone, as well as the complex crack patterns in coal. In conclusion, our model contributes to improving rock resolution and provides more accurate image processing techniques for rock property analysis and geological research.

Collaborative brightening and amplification of low-light imagery via bi-level adversarial learning

Poor light conditions constrain the high pursuit of clarity and visible quality of photography especially smartphone devices. Admittedly, existing specific image processing methods, whether super-resolution methods or low-light enhancement methods, can hardly simultaneously enhance the resolution and brightness of low-light images at the same time. This paper dedicates a specialized enhancer with a dual-path modulated-interactive structure to recover high-quality sharp images in conditions of near absence of light, dubbed CollaBA, which learns the direct mapping from low-resolution dark-light images to their high-resolution normal sharp version. Specifically, we construct the generative modulation prior, serving as illuminance attention information, to regulate the exposure level of the neighborhood range. In addition, we construct an interactive degradation removal branch that progressively embeds the generated intrinsic prior to recover high-frequency detail and contrast at the feature level. We also introduce a multi-substrate up-scaler to integrate multi-scale sampling features, effectively addressing artifact-related problems. Rather than adopting the naive time-consuming learning strategy, we design a novel bi-level implicit adversarial learning mechanism as our fast training strategy. Extensive experiments on benchmark datasets — demonstrate our model’s wide-ranging applicability in various ultra-low-light scenarios, across 8 key performance metrics with significant improvements, notably achieving a 35.8% improvement in LPIPS and a 23.1% increase in RMSE. The code will be available at https://github.com/moriyaya/CollaBA.

A High-Performance Accelerator for Real-Time Super-Resolution on Edge FPGAs

In the digital era, the prevalence of low-quality images contrasts with the widespread use of high-definition displays, primarily due to low-resolution cameras and compression technologies. Image super-resolution (SR) techniques, particularly those leveraging deep learning, aim to enhance these images for high-definition presentation. However, real-time execution of deep neural network (DNN)-based SR methods at the edge poses challenges due to their high computational and storage requirements. To address this, field-programmable gate arrays (FPGAs) have emerged as a promising platform, offering flexibility, programmability, and adaptability to evolving models. Previous FPGA-based SR solutions have focused on reducing computational and memory costs through aggressive simplification techniques, often sacrificing the quality of the reconstructed images. This paper introduces a novel SR network specifically designed for edge applications, which maintains reconstruction performance while managing computation costs effectively. Additionally, we propose an architectural design that enables the real-time and end-to-end inference of the proposed SR network on embedded FPGAs. Our key contributions include a tailored SR algorithm optimized for embedded FPGAs, a DSP-enhanced design that achieves a significant four-fold speedup, a novel scalable cache strategy for handling large feature maps, optimization of DSP cascade consumption, and a constraint optimization approach for resource allocation. Experimental results demonstrate that our FPGA-specific accelerator surpasses existing solutions, delivering superior throughput, energy efficiency, and image quality.

Super-Resolution Imaging of Clickable Lipids With Lipid Expansion Microscopy (LExM).

Fluorescent imaging of cellular membranes is challenged by the size of lipid bilayers, which are smaller than the diffraction limit of light. Recently, expansion microscopy (ExM) has emerged as an approachable super-resolution method that requires only widely accessible confocal microscopes. In this method, biomolecules of interest are anchored to hydrogel-based, polymeric networks that are expanded through osmosis to physically separate and resolve features smaller than the diffraction limit of light. Whereas ExM has been employed for super-resolution imaging of proteins, DNA, RNA, and glycans, the application of this method to the study of lipids is challenged by the requirement of permeabilization procedures that remove lipids and compromise the integrity of the membrane. Here, we describe our recently developed protocols for lipid expansion microscopy (LExM), a method that enables ExM of membranes without permeabilization. These detailed protocols and accompanying commentary sections aim to make LExM accessible to any experimentalist interested in imaging membranes with super-resolution. © 2024 Wiley Periodicals LLC. Basic Protocol 1: LExM of alkyne-choline lipids Basic Protocol 2: LExM of IMPACT-labeled lipids Basic Protocol 3: LExM of clickable cholesterol Basic Protocol 4: Determining the expansion factor.

FSRDiff: A fast diffusion-based super-resolution method using GAN

Single image super-resolution with diffusion probabilistic models (SRDiff) is a successful diffusion model for image super-resolution that produces high-quality images and is stable during training. However, due to the long sampling time, it is slower in the testing phase than other deep learning-based algorithms. Reducing the total number of diffusion steps can accelerate sampling, but it also causes the inverse diffusion process to deviate from the Gaussian distribution and exhibit a multimodal distribution, which violates the diffusion assumption and degrades the results. To overcome this limitation, we propose a fast SRDiff (FSRDiff) algorithm that integrates a generative adversarial network (GAN) with a diffusion model to speed up SRDiff. FSRDiff employs conditional GAN to approximate the multimodal distribution in the inverse diffusion process of the diffusion model, thus enhancing its sampling efficiency when reducing the total number of diffusion steps. The experimental results show that FSRDiff is nearly 20 times faster than SRDiff in reconstruction while maintaining comparable performance on the DIV2K test set.

Image Super-resolution Networks based on Structure Reparameterized Convolution

The traditional image super resolution method based on convolutional neural network generally brings two problems: First, the model has only a single scale of receptive field, which cannot use the image information in a smaller range; Second, the size of the feature map will become smaller and smaller in the process of continuous convolution, and it is necessary to continuously carry out edge zeroing operation to maintain the original size, which leads to the loss of part of the edge information. To solve the above two problems, we propose a network structure based on structure re-parameterized convolution, which sets a small convolution kernel in parallel beside a large convolution kernel in the same layer, trains the two cores simultaneously, and finally merges the two cores. The experimental results show that in this way, we make the large convolution kernel could capture smaller information, which not only improves the high-frequency details of the reconstructed image but also avoids frequent edge filling to reduce the information density, and effectively speeds up the reasoning speed after the model is re-parameterized. Compared with the advanced methods, our method achieves good results.

An Omnidirectional Image Super-Resolution Method Based on Enhanced SwinIR

For the significant distortion problem caused by the special projection method of equi-rectangular projection (ERP) images, this paper proposes an omnidirectional image super-resolution algorithm model based on position information transformation, taking SwinIR as the base. By introducing a space position transformation module that supports deformable convolution, the image preprocessing process is optimized to reduce the distortion effects in the polar regions of the ERP image. Meanwhile, by introducing deformable convolution in the deep feature extraction process, the model’s adaptability to local deformations of images is enhanced. Experimental results on publicly available datasets have shown that our method outperforms SwinIR, with an average improvement of over 0.2 dB in WS-PSNR and over 0.030 in WS-SSIM for ×4 pixel upscaling.

Research on Methods of Micro-target Detection in Various Application Scenarios

This paper provides a comprehensive introduction and comparison of the research on small target detection algorithms in different application scenarios and systematically explores the application of deep learning algorithms for small target detection in remote sensing imagery, visible imagery, and infrared imagery. In remote sensing images, the Deconvolution R-CNN algorithm is used to improve the accuracy of small target detection by adding a deconvolution layer. In the field of visible images, a Gaussian Mixture Model (GMM)-based approach is introduced to achieve fast detection of small targets with SIFT features and a GMM target detector. In infrared images, a comprehensive algorithm adapted to low signal-to-noise ratio and complex background is proposed to automatically detect tiny targets through background suppression and threshold selection optimization. In this paper, we compare and analyze the performance of these methods in different scenarios, and propose directions for future research, including expanding the training dataset, applying super-resolution methods to improve the image resolution, and enhancing the universality of the algorithms. This paper provides a comprehensive overview of research in the field of small target detection, as well as an outlook on future research directions, which provides a useful reference for the further development of the field.

A novel spectral super-resolution network with dominant information between spatial and spectral domains

Existing spectral super-resolution (SSR) methods have achieved satisfactory performance by designing complicated deep convolution neural networks (DCNNs) to extract spectral and spatial features. However, these methods ignore the fact that the significance of spatial and spectral information in each hyperspectral image (HSI) is different, and most of them directly fuse two kinds of information with concatenation and convolution operation, which resulting in generating redundant information and have negative effects on reconstruction. To address such inadequacies, this paper proposes a novel adaptive spatial–spectral modulation network (ASSM-Net). Specifically, we propose a new adaptive feature fusion module (AFFM) to replace traditional convolutional fusion schemes. Through explicitly measuring the weights of spatial information and spectral features of HSI, AFFM can select dominant features for each pixel to modulate spatial and spectral information. Additionally, we develop a pixel-weighted aware attention (PAA) mechanism to enhance the feature interdependences for recovering finer structure information. Finally, a large number of quantitative and qualitative experiments reveal that the proposed network achieves competitive reconstruction results on three benchmark datasets (NTIRE 2022, CAVE and Harvard).

Single-Image Super-Resolution Method for Rotating Synthetic Aperture System Using Masking Mechanism

The emerging technology of rotating synthetic aperture (RSA) presents a promising solution for the development of lightweight, large-aperture, and high-resolution optical remote sensing systems in geostationary orbit. However, the rectangular shape of the primary mirror and the distinctive imaging mechanism involving the continuous rotation of the mirror lead to a pronounced decline in image resolution along the shorter side of the rectangle compared to the longer side. The resolution also exhibits periodic time-varying characteristics. To address these limitations and enhance image quality, we begin by analyzing the imaging mechanism of the RSA system. Subsequently, we propose a single-image super-resolution method that utilizes a rotated varied-size window attention mechanism instead of full attention, based on the Vision Transformer architecture. We employ a two-stage training methodology for the network, where we pre-train it on images masked with stripe-shaped masks along the shorter side of the rectangular pupil. Following that, we fine-tune the network using unmasked images. Through the strip-wise mask sampling strategy, this two-stage training approach effectively circumvents the interference of lower confidence (clarity) information and outperforms training the network from scratch using the unmasked degraded images. Our digital simulation and semi-physical imaging experiments demonstrate that the proposed method achieves satisfactory performance. This work establishes a valuable reference for future space applications of the RSA system.

SwinDPSR: Dual-Path Face Super-Resolution Network Integrating Swin Transformer

Whether to use face priors in the face super-resolution (FSR) methods is a symmetry problem.Various face priors are used to describe the overall and local face features, making the generation of super-resolution face images expensive and laborious. FSR methods that do not require any prior information tend to focus too much on the local features of the face, ignoring the modeling of global information. To solve this problem, we propose a dual-path facial image super-resolution network (SwinDPSR) fused with Swin Transformer. The network does not require additional face priors, and it learns global face shape and local face components through two independent branches. In addition, the channel attention ECA module is used to aggregate the global and local face information in the above dual-path sub-networks, which can generate corresponding high-quality face images. The results of face super-resolution reconstruction experiments on public face datasets and a real-scene face dataset show that SwinDPSR is superior to previous advanced methods both in terms of visual effects and objective indicators. The reconstruction results are evaluated with four evaluation metrics: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), and mean perceptual score (MPS).

A boosted degradation representation learning for blind image super-resolution

The significant gap between the assumed and actual degradation model will undoubtedly lead to a serious performance decrease for deep learning based image super-resolution (SR) methods. To solve this shortcoming, this paper presents an enhanced blind image super-resolution (EBSR) network based on unsupervised degradation representation learning. The proposed EBSR mainly is made up of two branching subnetworks. The first subnetwork is a residual-based degradation encoder, which is responsible to learn a high-dimensional abstract degradation representation vector from the input low-resolution (LR) image patches using contrast learning. Another branch is the degradation-aware fusion SR (DAFSR) network that completes the image SR task with the aid of the degradation representation vector learned by the encoder. To obtain an improved SR performance under various degradation settings, the degradation-aware fusion block (DAFB), the core block of DAFSR, has been elaborated, which embeds the degradation model information learned by the encoder into every layer of the network. Additionally, a shallow feature extraction block (SFEB) is constructed for DAFSR to extract global information from the LR image. The proposed EBSR can flexibly adapt to various degradation models. Extensive experimental results on synthetic datasets and real images show that the proposed EBSR is able to achieve a leading reconstruction performance over other blind SR algorithms.

Accurate blood glucose level monitoring using microwave imaging

Painless and non-invasive detection techniques are needed to replace finger-prick blood collection for people with diabetes. A first-of-its-kind, noninvasive, and continuous blood glucose level (BGL) detection method based on microwave imaging is introduced in this paper. This method avoids the complex task of frequency choice for the design of electromagnetic sensors. A radar-based microwave imaging technology combined with an improved very-deep super-resolution (VDSR-BL) method is presented to obtain high-resolution (HR) microwave images. After image super-resolution reconstruction by VDSR-BL, the peak signal-to-noise ratio and structural similarity index of HR images reach 35.4461 dB and 0.9761, respectively. Then, an ensemble learning strategy based on support vector regression and random forest algorithms is proposed to identify HR microwave images for BGL estimation. The developed detection system has been verified on the medium under tests with different glucose solutions. The final detection results obtain a root mean squared error of 0.1394 mg ml−1 and a mean absolute relative difference of 8.02%, which show good accuracy with clinical acceptance. Meanwhile, we also conducted human trials. A high correlation coefficient (R) of 0.9254 was achieved between the results of microwave imaging and invasive BGL. Together, these results show that microwave imaging offers a promising new approach for noninvasive BGL monitoring.

Learning-based and quality preserving super-resolution of noisy images

AbstractPurpose: Several applications require the super-resolution of noisy images and the preservation of geometrical and texture features. State-of-the-art super-resolution methods do not account for noise and generally enhance the output image’s artefacts (e.g., aliasing, blurring). Methods: We propose a learning-based method that accounts for the presence of noise and preserves the properties of the input image, as measured by quantitative metrics, e.g., normalised crossed correlation, normalised mean squared error, peak-signal-to-noise-ration, structural similarity feature-based similarity, universal image quality. We train our network to up-sample a low-resolution noisy image while preserving its properties. We perform our tests on the Cineca Marconi100 cluster, at the 26th position in the “top500” list. Results: The experimental results show that our method outperforms learning-based methods, has comparable results with standard methods, preserves the properties of the input image as contours, brightness, and textures, and reduces the artefacts. As average quantitative metrics, our approach has a PSNR value of 23.81 on the super-resolution of Gaussian noise images with a 2X up-sampling factor. In contrast, previous work has a PSNR value of 23.09 (standard method) and 21.78 (learning-based method). Conclusion: Our learning-based and quality-preserving super-resolution improves the high-resolution prediction of noisy images with respect to state-of-the-art methods with different noise types and up-sampling factors.