Super-resolution Method Research Articles

SBHSR: single-band super-resolution method for hyperspectral images based on blind degradation and fusion of auxiliary band

Despite natural image super-resolution (SR) methods have achieved great success, super-resolution methods for hyperspectral image (HSI) with rich spectral features are still a very challenging task. Furthermore, due to the diversity of HSI captured by different cameras, their degradation conditions vary. Currently available HSI SR methods are mainly based on fixed degradation models, and their performance is severely affected when the actual degradation does not match the assumed degradation model. To address this issue, this paper proposes a single-band hyperspectral image super-resolution (SBHSR) method for HSI. This method assumes that the degradation Gaussian blur kernel parameters of the HSI are unknown, and adapts to various degradation conditions by conducting blind super-resolution on the image. It also combines another band with better spatial structure as an auxiliary band for feature fusion. To address the spectral differences between the super-resolution band and the auxiliary band, we use the Spatial Adaptation Module (SAM) to map the feature distribution, ensuring consistent spectral brightness between the auxiliary band and the super-resolution band. Due to the use of blind degradation super-resolution method, our method is more robust compared to previous HSI SR methods. Additionally, as it is specifically designed for single-band hyperspectral image super-resolution, it has the advantage of being faster and more efficient. Experimental validation on two hyperspectral datasets demonstrates the superiority of our method, as it improves the spatial resolution of the image while preserving the spectral features, and performs better than existing blind super-resolution methods.

COMET: Cross-space Optimization-based Mutual learning network for super-resolution of CEST-MRI.

Chemical Exchange Saturation Transfer Magn-etic Resonance Imaging (CEST-MRI) is a promising approach for detecting tissue metabolic changes. However, due to the constraints of scan time and contrast-noise-ratio, CEST-MRI always exhibits low spatial resolution, hindering the clinical applications especially for detection of small lesions. Many super-resolution (SR) methods have shown good performance in medical images. However, when applied to CEST-MRI, these methods have two shortcomings that may limit their performance. Firstly, CEST-MRI has an additional frequency dimension, but the information along this dimension is not fully utilized. The second is that these SR methods mainly focus on improving the quality of the CEST-weighted images, while the accuracy of the quantitative maps is the most concerned aspect for CEST-MRI. To address these shortcomings, we propose a Cross-space Optimization-based Mutual learning nETwork (COMET) for SR of CEST-MRI. COMET incorporates novel spatio-frequency extraction modules and a mutual learning module to leverage and combine information from both spatial and frequency spaces, thereby enhancing the SR performance. Furthermore, we propose a novel CEST-based normalization loss to address the normalization-induced distribution problem and preserve the sharpness of quantitative maps, enabling more accurate CEST-MRI quantification. COMET is evaluated on an ischemia rat brain dataset and a human brain dataset. The results demonstrate COMET achieves 8-fold SR, providing accurate quantitative maps. Moreover, COMET outperforms all other state-of-the-art SR methods. Additionally, COMET exhibits its potential in prospective study.

MFCEN: A lightweight multi-scale feature cooperative enhancement network for single-image super-resolution

<p>In recent years, significant progress has been made in single-image super-resolution with the advancements of deep convolutional neural networks (CNNs) and transformer-based architectures. These two techniques have led the way in the field of super-resolution technology research. However, performance improvements often come at the cost of a substantial increase in the number of parameters, thereby limiting the practical applications of super-resolution methods. Existing lightweight super-resolution methods, which primarily focus on single-scale feature extraction, lead to the issue of missing multi-scale features. This results in incomplete feature acquisition and poor reconstruction of the image. In response to these challenges, this paper proposed a lightweight multi-scale feature cooperative enhancement network (MFCEN). The network consists of three parts: shallow feature extraction, deep feature extraction, and image reconstruction. In the deep feature extraction part, a novel integrated multi-level feature module was introduced. Compared to existing CNN and transformer hybrid super-resolution networks, MFCEN significantly reduced the number of parameters while maintaining performance. This improvement was particularly evident at a scale factor of 3. The network introduced a novel comprehensive integrated multi-level feature module, leveraging the strong local perceptual capabilities of CNNs and the superior global information processing of transformers. It was designed with depthwise separable convolutions for extracting local information and a block-scale and global feature extraction module based on vision transformers (ViTs). While extracting the three scales of features, a satisfiability attention mechanism with a feed-forward network that can control the information was used to keep the network lightweight. Experiments demonstrated that the proposed model surpasses the reconstruction performance of the 498K-parameter SPAN model with a mere 488K parameters. Extensive experiments on commonly used image super-resolution datasets further validated the effectiveness of the network.</p>

Single-image super-resolution reconstruction based on phase-aware visual multi-layer perceptron (MLP).

Many advanced super-resolution reconstruction methods have been proposed recently, but they often require high computational and memory resources, making them incompatible with low-power devices in reality. To address this problem, we propose a simple yet efficient super-resolution reconstruction method using waveform representation and multi-layer perceptron (MLP) for image processing. Firstly, we partition the original image and its down-sampled version into multiple patches and introduce WaveBlock to process these patches. WaveBlock represents patches as waveform functions with amplitude and phase and extracts representative feature representations by dynamically adjusting phase terms between tokens and fixed weights. Next, we fuse the extracted features through a feature fusion block and finally reconstruct the image using sub-pixel convolution. Extensive experimental results demonstrate that SRWave-MLP performs excellently in both quantitative evaluation metrics and visual quality while having significantly fewer parameters than state-of-the-art efficient super-resolution methods.

MDRN: Multi-distillation residual network for efficient MR image super-resolution

<p>Super-resolution (SR) of magnetic resonance imaging (MRI) is gaining increasing attention for being able to provide detailed anatomical information. However, current SR methods often use the complex convolutional network for feature extraction, which is difficult to train and not suitable for limited computation resources in the medical scenario. To tackle these bottlenecks, we propose a multi-distillation residual network (MDRN) for more differential feature refinement, which has a superior trade-off between reconstruction accuracy and computation cost. Specifically, a novel feature multi-distillation residual block with a contrast-aware channel attention module was designed to make the residual features more focused on low-vision information, which maximizes the power of MDRN. Comprehensive experiments demonstrate the superiority of our MDRN over state-of-the-art methods in reconstruction quality and efficiency. Our method outperforms other existing methods in peak signal-noise ratio by up to 0.44–1.82 dB in 4× scale when GPU memory and runtime are lower than in other SR methods. The source code will be available at <ext-link ext-link-type="uri" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://github.com/Jennieyy/MDRN">https://github.com/Jennieyy/MDRN</ext-link>.</p>

Tensor Ring Based Image Enhancement.

Image enhancement, including image de-noising, super-resolution, registration, reconstruction, in-painting, and so on, is an important issue in different research areas. Different methods which have been exploited for image analysis were mostly based on matrix or low order analysis. However, recent researches show the superior power of tensor-based methods for image enhancement. In this article, a new method for image super-resolution using Tensor Ring decomposition has been proposed. The proposed image super-resolution technique has been derived for the super-resolution of low resolution and noisy images. The new approach is based on a modification and extension of previous tensor-based approaches used for super-resolution of datasets. In this method, a weighted combination of the original and the resulting image of the previous stage has been computed and used to provide a new input to the algorithm. This enables the method to do the super-resolution and de-noising simultaneously. Simulation results show the effectiveness of the proposed approach, especially in highly noisy situations.

Scale-Frequency Dual Modulation Method for Remote Sensing Image Continuous Super-Resolution

Scale-Frequency Dual Modulation Method for Remote Sensing Image Continuous Super-Resolution

Bridging the Domain Gap: A Simple Domain Matching Method for Reference-Based Image Super-Resolution in Remote Sensing

Bridging the Domain Gap: A Simple Domain Matching Method for Reference-Based Image Super-Resolution in Remote Sensing

A Federated Deep Unrolling Method for Lidar Super-Resolution: Benefits in SLAM

A Federated Deep Unrolling Method for Lidar Super-Resolution: Benefits in SLAM

Simpler Gradient Methods for Blind Super-Resolution with Lower Iteration Complexity

Simpler Gradient Methods for Blind Super-Resolution with Lower Iteration Complexity

Anycost Network Quantization for Image Super-Resolution.

In this paper, we propose an anycost network quantization method for efficient image super-resolution with variable resource budgets. Conventional quantization approaches acquire discrete network parameters for deployment with fixed complexity constraints, while image super-resolution networks are usually applied on mobile devices with frequently modified resource budgets due to the change of battery levels or computing chips. Hence, exhaustively optimizing quantized networks with each complexity constraint results in unacceptable training costs. On the contrary, we construct a hyper-network whose parameters can efficiently adapt to different resource budgets with negligible finetuning cost, so that the image super-resolution networks can be feasibly deployed in diversified devices with variable resource budgets. Specifically, we dynamically search the optimal bitwidth for each patch in convolution according to feature maps and complexity constraints, which aims to achieve the best efficiency-accuracy trade-off in image super-resolution given the resource budget. To acquire the hyper-network that can be efficiently adapted to different bitwidth settings, we actively sample the patch-wise bitwidth during training and adaptively ensemble gradients from hyper-network in different precision for faster convergence and higher generalization ability. Compared with existing quantization methods, experimental results demonstrate that our method significantly reduces the cost of adapting models in new resource budgets with comparable efficiency-accuracy trade-offs.

Real-World Light Field Image Super-Resolution Via Degradation Modulation.

Recent years have witnessed the great advances of deep neural networks (DNNs) in light field (LF) image super-resolution (SR). However, existing DNN-based LF image SR methods are developed on a single fixed degradation (e.g., bicubic downsampling), and thus cannot be applied to super-resolve real LF images with diverse degradation. In this article, we propose a simple yet effective method for real-world LF image SR. In our method, a practical LF degradation model is developed to formulate the degradation process of real LF images. Then, a convolutional neural network is designed to incorporate the degradation prior into the SR process. By training on LF images using our formulated degradation, our network can learn to modulate different degradation while incorporating both spatial and angular information in LF images. Extensive experiments on both synthetically degraded and real-world LF images demonstrate the effectiveness of our method. Compared with existing state-of-the-art single and LF image SR methods, our method achieves superior SR performance under a wide range of degradation, and generalizes better to real LF images. Codes and models are available at https://yingqianwang.github.io/LF-DMnet/.

Super-resolution fluorescence imaging of PEDOT:PSS films.

Poly(3,4-ethylenedioxythiophene):poly(styrenesulfonate) (PEDOT:PSS) is a popular conduction polymer and widely used in organic electronics, bioelectronics and printed electronics. It is believed that PEDOT:PSS has a core-shell colloidal structure dispersed in the formulation. However, the size and surface functional groups of the PEDOT:PSS dispersion and films remain to be visualized. Here, we have introduced the concept of aggregation-induced emission (AIE) to super-resolution imaging for designing the cationic probe TPE-4N+ and accomplished the nanoscale optical visualization of PEDOT:PSS films through reversible electrostatic interactions. Information on the PEDOT:PSS size and surface charge has been successfully collected via super-resolution imaging. The full-width at half-maximum (FWHM) of PSS nanoparticles was observed to be approximately 30-40 nm. The super-resolution fluorescence imaging method can also be used to monitor the PEDOT:PSS film after acid treatment. It was observed that PSS chains were washed away when exposed to concentrated sulfuric acid, which explains why concentrated sulfuric acid treatment greatly improves the conductivity of the PEDOT:PSS film. Super-resolution imaging is promising as an effective method for characterizing PEDOT:PSS films.

Super-resolution microscopy methods to study membrane pores in situ.

Super-resolution microscopy methods to study membrane pores in situ.

Wavelet Pyramid Recurrent Structure-Preserving Attention Network for Single Image Super-Resolution.

Many single image super-resolution (SISR) methods that use convolutional neural networks (CNNs) learn the relationship between low-and high-resolution images directly, without considering the context structure and detail fidelity. This can limit the potential of CNNs and result in unrealistic, distorted edges and textures in the reconstructed images. A more effective approach is to incorporate prior knowledge about the image into the model to aid in image reconstruction. In this study, we propose a novel recurrent structure-preserving mechanism that innovatively uses the multiscale wavelet transform (WT) as an image prior, namely, wavelet pyramid recurrent structure-preserving attention network (WRSANet), to process both low-and high-frequency subnetworks at each level separately and recursively. We propose a novel structure scale preservation (SSP) architecture that differs from traditional WTs. This architecture allows us to incorporate and learn structure preservation subnetworks at each level. By using our proposed structure scale fusion (SSF) combined with inverse WT, we can recursively restore and preserve rich low-frequency image structure through the combination of SSP at various levels. Furthermore, we also propose novel low-to-high-frequency information transmission (L2HIT) and detail enhancement (DE) mechanisms to address the issue of detail distortion in high-frequency images by transferring information from structure preservation subnetworks. This allows us to preserve the low-frequency structure while reconstructing high-frequency details, improving detail fidelity and avoiding structural distortion. Finally, a joint loss function is also used to balance the fusion of low-and high-frequency information at different degrees, with hyperparameters being adjusted during training. The experimental results demonstrate that the proposed WRSANet achieves better performance and visual presentation than the state-of-the-art (SOTA) on synthetic and real datasets, especially in terms of context structure and texture details.

Difficulty-Guided Variant Degradation Learning for Blind Image Super-Resolution.

Recent blind super-resolution (BSR) methods are explored to handle unknown degradations and achieve impressive performance. However, the prevailing assumption in most BSR methods is the spatial invariance of degradation kernels across the entire image, which leads to significant performance declines when faced with spatially variant degradations caused by object motion or defocusing. Additionally, these methods do not account for the human visual system's tendency to focus differently on areas of varying perceptual difficulty, as they uniformly process each pixel during reconstruction. To cope with these issues, we propose a difficulty-guided variant degradation learning network for BSR, named difficulty-guided degradation learning (DDL)-BSR, which explores the relationship between reconstruction difficulty and degradation estimation. Accordingly, the proposed DDL-BSR consists of three customized networks: reconstruction difficulty prediction (RDP), space-variant degradation estimation (SDE), and degradation and difficulty-informed reconstruction (DDR). Specifically, RDP learns the reconstruction difficulty with the proposed reconstruction-distance supervision. Then, SDE is designed to estimate space-variant degradation kernels according to the difficulty map. Finally, both degradation kernels and reconstruction difficulty are fed into DDR, which takes into account such two prior knowledge information to guide super-resolution (SR). Experimental analysis on various synthetic datasets demonstrates that DDL-BSR invariably surpasses state-of-the-art (SOTA) methods, producing SR images with enhanced realism and texture quality. Code is available at https://github.com/JiaWang0704/DDL-BSR.

Overview of Essential Components in deep learning reference-based super resolution methods

Reference-based super resolution (RefSR) aims to recover the lost details in a low-resolution image and generate a high-resolution result, guided by a high-resolution reference image with similar contents or textures. In contrast to the traditional single-image super-resolution, which focuses on the intrinsic properties of the single low-resolution image, the challenge of RefSR lies in matching and aggregating highly-related but misaligned reference images with low-resolution images. Several effective but complex designs have been proposed to address this challenge, which poses difficulties in implementing RefSR in real-world applications. In order to better understand the working mechanism of RefSR and design a more efficient and lightweight architecture, we provide a review about the essential components of the existing deep learning-based RefSR. We decompose and classify the common pipeline into four submodules according to the functionalities. Then, we summarize and describe the implementation details of the commonly-adopted approaches in each submodule. Finally, we discuss the challenges and promising research directions.

Hierarchical Prior-Based Super Resolution for Point Cloud Geometry Compression.

The Geometry-based Point Cloud Compression (G-PCC) has been developed by the Moving Picture Experts Group to compress point clouds efficiently. Nevertheless, in its lossy mode, the reconstructed point cloud by G-PCC often suffers from noticeable distortions due to naïve geometry quantization (i.e., grid downsampling). This paper proposes a hierarchical prior-based super resolution method for point cloud geometry compression. The content-dependent hierarchical prior is constructed at the encoder side, which enables coarse-to-fine super resolution of the point cloud geometry at the decoder side. A more accurate prior generally yields improved reconstruction performance, albeit at the cost of increased bits required to encode this piece of side information. Our experiments on the MPEG Cat1A dataset demonstrate substantial Bjøntegaard-delta bitrate savings, surpassing the performance of the octree-based and trisoup-based G-PCC v14. We provide our implementations for reproducible research at https://github.com/lidq92/mpeg-pcc-tmc13.

A Virtual-Sensor Construction Network Based on Physical Imaging for Image Super-Resolution.

Image imaging in the real world is based on physical imaging mechanisms. Existing super-resolution methods mainly focus on designing complex network structures to extract and fuse image features more effectively, but ignore the guiding role of physical imaging mechanisms for model design, and can not mine features from a physical perspective. Inspired by the mechanism of physical imaging, we propose a novel network architecture called Virtual-Sensor Construction network (VSCNet) to simulate the sensor array inside the camera. Specifically, VSCNet first generates different splitting directions to distribute photons to construct virtual sensors, and then performs a multi-stage adaptive fine-tuning operation to fine-tune the number of photons on the virtual sensors to increase the photosensitive area and eliminate photon cross-talk, and finally converts the obtained photon distributions into RGB images. These operations can naturally be regarded as the virtual expansion of the camera's sensor array in the feature space, which makes our VSCNet bridge the physical space and feature space, and uses their complementarity to mine more effective features to improve performance. Extensive experiments on various datasets show that the proposed VSCNet achieves state-of-the-art performance with fewer parameters. Moreover, we perform experiments to validate the connection between the proposed VSCNet and the physical imaging mechanism. The implementation code is available at https://github.com/GZ-T/VSCNet.

Toward quantitative super-resolution methods for cryo-CLEM.

Cryogenic ultrastructural imaging techniques such as cryo-electron tomography have produced a revolution in how the structure of biological systems is investigated by enabling the determination of structures of protein complexes immersed in a complex biological matrix within vitrified cell and model organisms. However, so far, the portfolio of successes has been mostly limited to highly abundant complexes or to structures that are relatively unambiguous and easy to identify through electron microscopy. In order to realize the full potential of this revolution, researchers would have to be able to pinpoint lower abundance species and obtain functional annotations on the state of objects of interest which would then be correlated to ultrastructural information to build a complete picture of the structure-function relationships underpinning biological processes. Fluorescence imaging at cryogenic conditions has the potential to be able to meet these demands. However, wide-field images acquired at low numeric aperture (NA) using air immersion objective have a low resolving power and cannot provide accurate enough three-dimensional (3D) localization to enable the assignment of functional annotations to individual objects of interest or target sample debulking to ensure the preservation of the structures of interest. It is therefore necessary to develop super-resolved cryo-fluorescence workflows capable of fulfilling this role and enabling new biological discoveries. In this chapter, we present the current state of development of two super-resolution cryogenic fluorescence techniques, superSIL-STORM and astigmatism-based 3D STORM, show their application to a variety of biological systems and discuss their advantages and limitations. We further discuss the future applicability to cryo-CLEM workflows though examples of practical application to the study of membrane protein complexes both in mammalian cells and in Escherichia coli.