Image Super-resolution Method Research Articles

Reconstruction methods for super-resolution imaging with PSF modulation

Recently, a super-resolution imaging method called the PSF (point spread function) modulation method was developed (Lu, IEEE TUFFC 2024). In this method, the amplitude, phase, or both of the PSF of a linear shift-invariant (LSI) imaging system is modulated so that the modulated PSF has a higher spatial frequency than that of the original PSF to reconstruct super-resolution images. The modulator can be produced and manipulated remotely by methods such as radiation force or it can be a physical particle such as micro- or nano-particle manipulated by an external force such as electrical and electromagnetic force. In principle, the super-resolution imaging method can be applied to any LSI imaging system, such as ultrasound, optical, photoacoustic, electromagnetic, underwater, nondestructive evaluation (NDE), and magnetic resonance imaging (MRI) system. These include pulse-echo ultrasound imaging, transmission imaging, wave source/field imaging, acoustical camera, and optical bright-field microscope. To optimize the quality of the images, methods for the reconstruction of pulse-echo and wave source/field super-resolution images are studied and the results will be presented. These methods include the uses of an analytic envelope of radio-frequency (RF) signals with and without windowing, “I” (in-phase) and “Q” (quadrature) signals, and the DC (direct current) component removal.

Super-resolution ultrasound imaging with monodisperse microbubbles in a chicken embryo model

ULM is a super-resolution imaging method that has transformed ultrasound imaging by beating the diffraction limit, enabling the visualization of blood vessel down to the capillary size. The development of innovative ultrasound responsive agents may allow to further improve the performance of this technology. Bracco is engaged in the formulation and the evaluation of various ultrasound responsive agents for ULM including monodisperse microbubbles. Our recent studies have shown that monodisperse microbubbles increase imaging sensitivity by an order of magnitude in comparison to polydisperse microbubbles. The benefit of this for ultrasound localization microscopy (ULM) has been tested in vitro in flow phantom and in a chicken embryo chorio-allantoic membrane (CAM) model. The performances of the monodisperse versus polydisperse formulations were evaluated through a collection of numeric metrics extracted from the super-resolved reconstructed CAM images, including not only quantitative criteria, such as the number of detected microbubbles or the rate of successful tracking, but also geometrical considerations such as vessel density or number of segments of the reconstructed vasculature. The study showed that monodisperse formulations outperformed the other polydisperse formulations, particularly using low to moderate microbubble concentrations.

Cartoon single-image super-resolution approach based on generative adversarial network

In recent years, the study of a single image super-resolution (SISR) is crucial to improving image resolution and using hardware technology to improve image resolution. SISR is widely used in satellite remote sensing, video surveillance, and medical image processing because it mainly relies on deep learning algorithms to realize the conversion from low-resolution (LR) images to high-resolution images. It has the advantages of low cost, simple operation, and high efficiency. This paper proposes an image super-resolution method based on a generative adversarial network named text localization generative adversarial nets (TLGAN) model. The method is improved based on super-resolution generative adversarial networks (SRGAN), and the batch normalization layer is removed, which significantly reduces the computational burden of the model. In TLGAN model, we used the transfer learning method to pre-trained the model on the large dataset ImageNet, and then apply the pre-trained model to the cartoon image data set animes to achieve image super-resolution. Experimental results report that the proposed method has the advantages of fast running speed and excellent visual perception of super-resolution images compared with bicubic interpolation and SRGAN method.

Super-sectioning with multi-sheet reversible saturable optical fluorescence transitions (RESOLFT) microscopy

Light-sheet fluorescence microscopy is an invaluable tool for four-dimensional biological imaging of multicellular systems due to the rapid volumetric imaging and minimal illumination dosage. However, it is challenging to retrieve fine subcellular information, especially in living cells, due to the width of the sheet of light (>1 μm). Here, using reversibly switchable fluorescent proteins (RSFPs) and a periodic light pattern for photoswitching, we demonstrate a super-resolution imaging method for rapid volumetric imaging of subcellular structures called multi-sheet RESOLFT. Multiple emission-sheets with a width that is far below the diffraction limit are created in parallel increasing recording speed (1–2 Hz) to provide super-sectioning ability (<100 nm). Our technology is compatible with various RSFPs due to its minimal requirement in the number of switching cycles and can be used to study a plethora of cellular structures. We track cellular processes such as cell division, actin motion and the dynamics of virus-like particles in three dimensions.

RoughSet-DDPM: An Image Super-Resolution Method Based on Rough set Denoising Diffusion Probability Model

RoughSet-DDPM: An Image Super-Resolution Method Based on Rough set Denoising Diffusion Probability Model

Multi-level feature extraction and reconstruction for 3D MRI image super-resolution

Magnetic resonance imaging (MRI) is an essential radiology technique in clinical diagnosis, but its spatial resolution may not suffice to meet the growing need for precise diagnosis due to hardware limitations and thicker slice thickness. Therefore, it is crucial to explore suitable methods to increase the resolution of MRI images. Recently, deep learning has yielded many impressive results in MRI image super-resolution (SR) reconstruction. However, current SR networks mainly use convolutions to extract relatively single image features, which may not be optimal for further enhancing the quality of image reconstruction. In this work, we propose a multi-level feature extraction and reconstruction (MFER) method to restore the degraded high-resolution details of MRI images. Specifically, to comprehensively extract different types of features, we design the triple-mixed convolution by leveraging the strengths and uniqueness of different filter operations. For the features of each level, we then apply deconvolutions to upsample them separately at the tail of the network, followed by the feature calibration of spatial and channel attention. Besides, we also use a soft cross-scale residual operation to improve the effectiveness of parameter optimization. Experiments on lesion-free and glioma datasets indicate that our method obtains superior quantitative performance and visual effects when compared with state-of-the-art MRI image SR methods.

A spectral and spatial transformer for hyperspectral remote sensing image super-resolution

ABSTRACT Due to the generally low spatial resolution of hyperspectral images (HSIs), early multispectral images lacked corresponding panchromatic bands, and as a result, fusion methods could not be used to enhance resolution. Many researchers have proposed various image super-resolution methods to address these limitations. However, these methods still suffered from issues, such as inadequate feature representation, lack of spectral feature representation, and high computational cost and inefficiency. To address these challenges, a spectral and spatial transformer (SST) algorithm for hyperspectral remote sensing image super-resolution is introduced. This algorithm uses a spatial transformer structure to extract the spatial features between the image pixels and a spectral transformer structure to extract the spectral features within the image pixels. The integration of these two components is applied to HSI super-resolution. After comparative experiments with currently advanced methods on three publicly available hyperspectral datasets, the results consistently show that our algorithm has better performance in both spectral fidelity and spatial restoration. Furthermore, our proposed algorithm was applied to real-world super-resolution experiments in the region of China's Ruoergai National Park, and subsequently, pixel-based classification was conducted on the super-resolution images, the results indicate that our algorithm could also be applied to future remote sensing interpretation tasks.

One-Shot Dual-Detection-Based Single-Molecule Super-Resolution Imaging Method for Real-Time Observation of Spatiotemporal Catalytic Activity Variations on the Plasmonic Gold Nanoparticle Surface.

Understanding the relationship between the surface properties of a single plasmonic nanoparticle and its catalytic performance is critical for developing highly efficient nanocatalysts. In this study, a one-shot dual-detection-based single-molecule super-resolution imaging method in the evanescent field was developed to observe real-time spatiotemporal catalytic activity on a single plasmonic gold nanoparticle (AuNP) surface. The scattering intensity of AuNPs and the fluorescence of resorufin molecules produced on the AuNP surface were obtained simultaneously to investigate the relationship between nanoparticles and catalytic reactions at a single-molecule level. Chemisorbed adsorbates (i.e., catalytic product and resorufin) changed the electron density of individual AuNPs throughout the catalytic cycle, resulting in the fluctuation of the scattering intensity of individual AuNPs, which was attributed to the electron transfer between reactant resazurin molecules and AuNPs. The increase in the electron density of individual AuNPs affected the catalytic reaction rate. Furthermore, sequential mapping of individual catalytic events at the subdiffraction limit resolution was completed for real-time surface dynamics and spatiotemporal activity variations on the single AuNP surface. The developed method can aid in understanding surface-property-dependent catalytic kinetics and facilitate the development of nanoparticle-based heterogeneous catalysts at subdiffraction limit resolution.

Multi-scale dense spatially-adaptive residual distillation network for lightweight underwater image super-resolution

Underwater images are typically of poor quality, lacking texture and edge information, and are blurry and full of artifacts, which restricts the performance of subsequent tasks such as underwater object detection, and path planning for underwater unmanned submersible vehicles (UUVs). Additionally, the limitation of underwater equipment, most existing image enhancement and super-resolution methods cannot be implemented directly. Hence, developing a weightless technique for improving the resolution of submerged images while balancing performance and parameters is vital. In this paper, a multi-scale dense spatially-adaptive residual distillation network (MDSRDN) is proposed aiming at obtaining high-quality (HR) underwater images with odd parameters and fast running time. In particular, a multi-scale dense spatially-adaptive residual distillation module (MDSRD) is developed to facilitate the multi-scale global-to-local feature extraction like a multi-head transformer and enriching spatial attention maps. By introducing a spatial feature transformer layer (SFT layer) and residual spatial-adaptive feature attention (RSFA), an enhancing attention map for spatially-adaptive feature modulation is generated. Furthermore, to maintain the network lightweight enough, blue separable convolution (BS-Conv) and distillation module are applied. Extensive experimental results illustrate the superiority of MDSDRN in underwater image super-resolution reconstruction, which can achieve a great balance between parameters (only 0.32M), multi-adds (only 13G), and performance (26.38 dB on PSNR in USR-248) with the scale of ×4.

Tensor Ring Based Image Enhancement.

Image enhancement, including image de-noising, super-resolution, registration, reconstruction, in-painting, and so on, is an important issue in different research areas. Different methods which have been exploited for image analysis were mostly based on matrix or low order analysis. However, recent researches show the superior power of tensor-based methods for image enhancement. In this article, a new method for image super-resolution using Tensor Ring decomposition has been proposed. The proposed image super-resolution technique has been derived for the super-resolution of low resolution and noisy images. The new approach is based on a modification and extension of previous tensor-based approaches used for super-resolution of datasets. In this method, a weighted combination of the original and the resulting image of the previous stage has been computed and used to provide a new input to the algorithm. This enables the method to do the super-resolution and de-noising simultaneously. Simulation results show the effectiveness of the proposed approach, especially in highly noisy situations.

Real-World Light Field Image Super-Resolution Via Degradation Modulation.

Recent years have witnessed the great advances of deep neural networks (DNNs) in light field (LF) image super-resolution (SR). However, existing DNN-based LF image SR methods are developed on a single fixed degradation (e.g., bicubic downsampling), and thus cannot be applied to super-resolve real LF images with diverse degradation. In this article, we propose a simple yet effective method for real-world LF image SR. In our method, a practical LF degradation model is developed to formulate the degradation process of real LF images. Then, a convolutional neural network is designed to incorporate the degradation prior into the SR process. By training on LF images using our formulated degradation, our network can learn to modulate different degradation while incorporating both spatial and angular information in LF images. Extensive experiments on both synthetically degraded and real-world LF images demonstrate the effectiveness of our method. Compared with existing state-of-the-art single and LF image SR methods, our method achieves superior SR performance under a wide range of degradation, and generalizes better to real LF images. Codes and models are available at https://yingqianwang.github.io/LF-DMnet/.

Bridging the Domain Gap: A Simple Domain Matching Method for Reference-Based Image Super-Resolution in Remote Sensing

Bridging the Domain Gap: A Simple Domain Matching Method for Reference-Based Image Super-Resolution in Remote Sensing

Anycost Network Quantization for Image Super-Resolution.

In this paper, we propose an anycost network quantization method for efficient image super-resolution with variable resource budgets. Conventional quantization approaches acquire discrete network parameters for deployment with fixed complexity constraints, while image super-resolution networks are usually applied on mobile devices with frequently modified resource budgets due to the change of battery levels or computing chips. Hence, exhaustively optimizing quantized networks with each complexity constraint results in unacceptable training costs. On the contrary, we construct a hyper-network whose parameters can efficiently adapt to different resource budgets with negligible finetuning cost, so that the image super-resolution networks can be feasibly deployed in diversified devices with variable resource budgets. Specifically, we dynamically search the optimal bitwidth for each patch in convolution according to feature maps and complexity constraints, which aims to achieve the best efficiency-accuracy trade-off in image super-resolution given the resource budget. To acquire the hyper-network that can be efficiently adapted to different bitwidth settings, we actively sample the patch-wise bitwidth during training and adaptively ensemble gradients from hyper-network in different precision for faster convergence and higher generalization ability. Compared with existing quantization methods, experimental results demonstrate that our method significantly reduces the cost of adapting models in new resource budgets with comparable efficiency-accuracy trade-offs.

Fast Sparse Bayesian Learning Method for Scanning Radar Super-Resolution Imaging

Fast Sparse Bayesian Learning Method for Scanning Radar Super-Resolution Imaging

SBHSR: single-band super-resolution method for hyperspectral images based on blind degradation and fusion of auxiliary band

Despite natural image super-resolution (SR) methods have achieved great success, super-resolution methods for hyperspectral image (HSI) with rich spectral features are still a very challenging task. Furthermore, due to the diversity of HSI captured by different cameras, their degradation conditions vary. Currently available HSI SR methods are mainly based on fixed degradation models, and their performance is severely affected when the actual degradation does not match the assumed degradation model. To address this issue, this paper proposes a single-band hyperspectral image super-resolution (SBHSR) method for HSI. This method assumes that the degradation Gaussian blur kernel parameters of the HSI are unknown, and adapts to various degradation conditions by conducting blind super-resolution on the image. It also combines another band with better spatial structure as an auxiliary band for feature fusion. To address the spectral differences between the super-resolution band and the auxiliary band, we use the Spatial Adaptation Module (SAM) to map the feature distribution, ensuring consistent spectral brightness between the auxiliary band and the super-resolution band. Due to the use of blind degradation super-resolution method, our method is more robust compared to previous HSI SR methods. Additionally, as it is specifically designed for single-band hyperspectral image super-resolution, it has the advantage of being faster and more efficient. Experimental validation on two hyperspectral datasets demonstrates the superiority of our method, as it improves the spatial resolution of the image while preserving the spectral features, and performs better than existing blind super-resolution methods.

Wavelet Pyramid Recurrent Structure-Preserving Attention Network for Single Image Super-Resolution.

Many single image super-resolution (SISR) methods that use convolutional neural networks (CNNs) learn the relationship between low-and high-resolution images directly, without considering the context structure and detail fidelity. This can limit the potential of CNNs and result in unrealistic, distorted edges and textures in the reconstructed images. A more effective approach is to incorporate prior knowledge about the image into the model to aid in image reconstruction. In this study, we propose a novel recurrent structure-preserving mechanism that innovatively uses the multiscale wavelet transform (WT) as an image prior, namely, wavelet pyramid recurrent structure-preserving attention network (WRSANet), to process both low-and high-frequency subnetworks at each level separately and recursively. We propose a novel structure scale preservation (SSP) architecture that differs from traditional WTs. This architecture allows us to incorporate and learn structure preservation subnetworks at each level. By using our proposed structure scale fusion (SSF) combined with inverse WT, we can recursively restore and preserve rich low-frequency image structure through the combination of SSP at various levels. Furthermore, we also propose novel low-to-high-frequency information transmission (L2HIT) and detail enhancement (DE) mechanisms to address the issue of detail distortion in high-frequency images by transferring information from structure preservation subnetworks. This allows us to preserve the low-frequency structure while reconstructing high-frequency details, improving detail fidelity and avoiding structural distortion. Finally, a joint loss function is also used to balance the fusion of low-and high-frequency information at different degrees, with hyperparameters being adjusted during training. The experimental results demonstrate that the proposed WRSANet achieves better performance and visual presentation than the state-of-the-art (SOTA) on synthetic and real datasets, especially in terms of context structure and texture details.

Super-resolution fluorescence imaging of PEDOT:PSS films.

Poly(3,4-ethylenedioxythiophene):poly(styrenesulfonate) (PEDOT:PSS) is a popular conduction polymer and widely used in organic electronics, bioelectronics and printed electronics. It is believed that PEDOT:PSS has a core-shell colloidal structure dispersed in the formulation. However, the size and surface functional groups of the PEDOT:PSS dispersion and films remain to be visualized. Here, we have introduced the concept of aggregation-induced emission (AIE) to super-resolution imaging for designing the cationic probe TPE-4N+ and accomplished the nanoscale optical visualization of PEDOT:PSS films through reversible electrostatic interactions. Information on the PEDOT:PSS size and surface charge has been successfully collected via super-resolution imaging. The full-width at half-maximum (FWHM) of PSS nanoparticles was observed to be approximately 30-40 nm. The super-resolution fluorescence imaging method can also be used to monitor the PEDOT:PSS film after acid treatment. It was observed that PSS chains were washed away when exposed to concentrated sulfuric acid, which explains why concentrated sulfuric acid treatment greatly improves the conductivity of the PEDOT:PSS film. Super-resolution imaging is promising as an effective method for characterizing PEDOT:PSS films.

Single hyperspectral image super-resolution using a progressive upsampling deep prior network

&lt;p&gt;Hyperspectral image super-resolution (SR) aims to enhance the spectral and spatial resolution of remote sensing images, enabling more accurate and detailed analysis of ground objects. However, hyperspectral images have high dimensional characteristics and complex spectral patterns. As a result, it is critical to effectively leverage the spatial non-local self-similarity and spectral correlation within hyperspectral images. To address this, we have proposed a novel single hyperspectral image SR method based on a progressive upsampling deep prior network. Specifically, we introduced the spatial-spectral attention fusion unit (S&lt;sup&gt;2&lt;/sup&gt;AF) based on residual connections, in order to extract spatial and spectral information from hyperspectral images. Then we developed the group convolutional upsampling (GCU) to efficiently utilize the spatial and spectral prior information inherent in hyperspectral images. To address the challenges posed by the high dimensionality of hyperspectral images and limited training dataset, we implemented a parameter-sharing grouped convolutional upsampling framework within the GCU to ensure model stability and enhance performance. The experimental results on three benchmark datasets demonstrated that the proposed single hyperspectral image SR using a progressive upsampling deep prior network (PUDPN) method effectively improves the reconstruction quality of hyperspectral images and achieves promising performance.&lt;/p&gt;

Exploring the Spectral Prior for Hyperspectral Image Super-Resolution.

In recent years, many single hyperspectral image super-resolution methods have emerged to enhance the spatial resolution of hyperspectral images without hardware modification. However, existing methods typically face two significant challenges. First, they struggle to handle the high-dimensional nature of hyperspectral data, which often results in high computational complexity and inefficient information utilization. Second, they have not fully leveraged the abundant spectral information in hyperspectral images. To address these challenges, we propose a novel hyperspectral super-resolution network named SNLSR, which transfers the super-resolution problem into the abundance domain. Our SNLSR leverages a spatial preserve decomposition network to estimate the abundance representations of the input hyperspectral image. Notably, the network acknowledges and utilizes the commonly overlooked spatial correlations of hyperspectral images, leading to better reconstruction performance. Then, the estimated low-resolution abundance is super-resolved through a spatial spectral attention network, where the informative features from both spatial and spectral domains are fully exploited. Considering that the hyperspectral image is highly spectrally correlated, we customize a spectral-wise non-local attention module to mine similar pixels along spectral dimension for high-frequency detail recovery. Extensive experiments demonstrate the superiority of our method over other state-of-the-art methods both visually and metrically. Our code is publicly available at https://github.com/HuQ1an/SNLSR.

Gradient residual attention network for infrared image super-resolution

Image super-resolution techniques overcome the limitations of physical limits on infrared imaging systems and allow higher-resolution images of targets to be obtained on top of existing systems. In this paper, we proved that infrared images are more reconstructed than visible images based on data compression. We propose a CNN-based Gradient Residual Attention Network (GRAN) for infrared image super-resolution. Specifically, the residual dense module (RDB) is used to acquire depth features and the gradient operator (GO) gains fine-grained detail features. Meanwhile, the 3D attention block (3DAB) learns features' channel and spatial correlation to selectively capture more useful informative features. The experimental results show that the proposed method performs well compared to popular image super-resolution methods in terms of quantitative metrics and visual quality.