Image Super-resolution Method Research Articles

FedSR: Federated Learning for Image Super-Resolution via detail-assisted contrastive learning

Federated learning (FL) is a distributed learning paradigm that collaboratively learns a global model from multiple decentralized clients without sharing raw data, providing an effective solution for privacy preservation. However, current FL works primarily focus on high-level visual tasks (e.g., classification), the exploration for image super-resolution (SR), a fundamental low-level visual task, is rarely addressed. In this study, we propose Federated Image Super-Resolution (FedSR), a novel FL approach for image SR task, aiming to generate high-quality images while preserving data privacy. Specifically, we introduce a detail-assisted contrastive loss at the decentralized client, aligning the shallow representations by rectifying the relationship of low-level features between local clients and global server. In addition, we develop a hierarchical aggregation policy at the central server to better integrate dispersed client models into the global model, with the aggregation weight determined based on the layer-wise similarity between the updated local models and the historical global model. Extensive experiments conducted on general SR benchmarks and facial image datasets demonstrate the superiority of our FedSR in terms of image quality and model performance. Notably, FedSR can be seamlessly integrated with various prevalent image SR methods, including CNN-based and Transformer-based architectures.

ITSRN++: Stronger and better implicit transformer network for screen content image continuous super-resolution

Nowadays, online screen sharing and remote cooperation are becoming ubiquitous. However, the screen content may be downsampled and compressed during transmission, while it may be displayed on large screens or the users would zoom in for detail observation at the receiver side. Therefore, developing a strong and effective screen content image (SCI) super-resolution (SR) method is demanded. We observe that the weight-sharing upsampler (such as deconvolution or pixel shuffle) could be harmful to sharp and thin edges in SCIs, and the fixed scale upsampler makes it inflexible to fit screens with various sizes. To solve this problem, we propose an implicit transformer network for SCI continuous SR (termed as ITSRN++). Specifically, we propose a modulation based transformer as the upsampler, which modulates the pixel features in discrete space via a periodic nonlinear function to generate features in continuous space. To better restore the high-frequency details in screen content images, we further propose dual branch block (DBB) as the feature extraction backbone, where convolution and attention branches are utilized parallelly on the same linear transformed value. Besides, we construct a large-scale SCI2K dataset to facilitate the research on SCI SR. Experimental results on nine datasets demonstrate that the proposed method achieves state-of-the-art performance for SCI SR and also works well for natural image SR.

Correlative light and soft X-ray tomography of in situ mesoscale heterochromatin structure in intact cells

Heterochromatin organization is critical to many genome-related programs including transcriptional silencing and DNA repair. While super-resolution imaging, electron microscopy, and multiomics methods have provided indirect insights into the heterochromatin organization, a direct measurement of mesoscale heterochromatin ultrastructure is still missing. We use a combination of correlative light microscopy and cryo-soft X-ray tomography (CLXT) to analyze heterochromatin organization in the intact hydrated state of human mammary fibroblast cells. Our analysis reveals that the heterochromatin ultra-structure has a typical mean domain size of approximately 80 nm and a mean separation of approximately 120 nm between domains. Functional perturbations yield further insights into the molecular density and alterations in the mesoscale organization of the heterochromatin regions. Furthermore, our polymer simulations provide a mechanistic basis for the experimentally observed size and separation distributions of the mesoscale chromatin domains. Collectively, our results provide direct, label-free observation of heterochromatin organization in the intact hydrated state of cells.

Wavelet Pyramid Recurrent Structure-Preserving Attention Network for Single Image Super-Resolution.

Many single image super-resolution (SISR) methods that use convolutional neural networks (CNNs) learn the relationship between low-and high-resolution images directly, without considering the context structure and detail fidelity. This can limit the potential of CNNs and result in unrealistic, distorted edges and textures in the reconstructed images. A more effective approach is to incorporate prior knowledge about the image into the model to aid in image reconstruction. In this study, we propose a novel recurrent structure-preserving mechanism that innovatively uses the multiscale wavelet transform (WT) as an image prior, namely, wavelet pyramid recurrent structure-preserving attention network (WRSANet), to process both low-and high-frequency subnetworks at each level separately and recursively. We propose a novel structure scale preservation (SSP) architecture that differs from traditional WTs. This architecture allows us to incorporate and learn structure preservation subnetworks at each level. By using our proposed structure scale fusion (SSF) combined with inverse WT, we can recursively restore and preserve rich low-frequency image structure through the combination of SSP at various levels. Furthermore, we also propose novel low-to-high-frequency information transmission (L2HIT) and detail enhancement (DE) mechanisms to address the issue of detail distortion in high-frequency images by transferring information from structure preservation subnetworks. This allows us to preserve the low-frequency structure while reconstructing high-frequency details, improving detail fidelity and avoiding structural distortion. Finally, a joint loss function is also used to balance the fusion of low-and high-frequency information at different degrees, with hyperparameters being adjusted during training. The experimental results demonstrate that the proposed WRSANet achieves better performance and visual presentation than the state-of-the-art (SOTA) on synthetic and real datasets, especially in terms of context structure and texture details.

Representative Image Outpainting and Image Super-Resolution Methods Based on Deep Learning

The image generation based on deep learning is a technology that can generate new images or outpaint old images or improve visual effect of old images through learning from input data, according to deep learning structure. The representative technologies of image generation are image outpainting and image super-resolution. Deep learning is widely utilized in the field of computer vision. Facing different needs, it is essential to choose proper ways. This essay reviews several representative and new methods of image outpainting and image super-resolution. Compared with the results generated by old methods, the results generated by new image outpainting methods introduced in this essay have greater precision and clarity, the methods of image super-resolution that are suitable for all fields and professional fields can learn from each others dataset manually to train and develop. There is still unimaginable prospect of deep learning in the field of image outpainting and image super-resolution. Therefore, it is worthy of attention.

Representative Image Outpainting and Image Super-Resolution Methods Based on Deep Learning

Representative Image Outpainting and Image Super-Resolution Methods Based on Deep Learning

Image super-resolution based on improved ESRGAN and its application in camera calibration

Camera calibration is a key technique in the field of computer vision. The Zhang’s method is currently the most commonly used camera calibration method, but its accuracy is mainly dependent on the image quality, thusly constrained by camera hardware conditions. Considering that high-performance cameras are often expensive, a cost-effective method to improve the quality of calibration images is of great practical value. Without changing the existing hardware conditions, image super-resolution can be considered to enhance the quality of the calibration image. Image super-resolution (SR) is the process of reconstructing an image from low resolution (LR) to high resolution (HR). It can enhance the clarity and detail of the calibration images, which will be beneficial in improving the accuracy of camera calibration. However, there is very little research on the application of image super-resolution to camera calibration. Therefore, this study innovatively proposed a method and process for applying image super-resolution in camera calibration. Firstly, the ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) was optimized and improved, followed by the proposal of the MMRSRGAN (Multi-scale Multi-adaptive-weights Residual Super-Resolution Generative Adversarial Networks). Two methods for local image super-resolution were also proposed for efficient training and testing of datasets, and two evaluation metrics based on reprojection error were proposed evaluate the effectiveness of the proposed model. Training and testing experiments of image super-resolution for camera calibration were conducted on both virtual and real image datasets. The proposed MMRSRGAN was compared with several advanced image super-resolution networks developed in recent years across multiple dimensions and indicators. The positive results have demonstrated the feasibility of applying image super-resolution to camera calibration and showcased the good performance of the MMRSRGAN.

EdgeStereoSR: A multi-task network with transformers for stereo image super-resolution considering edge prior

Recently, stereo image super-resolution methods focusing on exploring cross-view information have been widely studied and achieved good performance. However, it is still challenging for them to reconstruct high-quality high-frequency details. In addition, they mainly focus on improving quantitative metrics, neglecting the perceptual quality of reconstructed images. In this paper, to improve the accuracy of high-frequency reconstruction, we propose a multi-task network with Transformers considering edge prior, named EdgeStereoSR, which achieves better stereo image SR under the guidance of edge detection. Basically, edge priors have two contributions. First, we propose a cross-view Transformer (CVT), which utilizes edge priors to guide the correspondence search, thus more accurate cross-view information can be captured. Second, we propose a cross-task Transformer (CTT), which exploits edge priors to guide the high-frequency reconstruction, thus images with more details and sharper edges can be reconstructed. To further improve the visual quality, we propose EdgeStereoSR-G, integrating the generative adversarial network into EdgeStereoSR. Specially, a spatial-view discriminator is designed to learn the stereo image distribution so as to make the reconstructed stereo image more photo-realistic and avoid parallax inconsistency. Extensive experiments show that the proposed methods are superior to other state-of-the-art methods in terms of both quantitative metrics and visual quality.

Residual SwinV2 transformer coordinate attention network for image super resolution

Swin Transformers have been designed and used in various image super-resolution (SR) applications. One of the recent image restoration methods is RSTCANet, which combines Swin Transformer with Channel Attention. However, for some channels of images that may carry less useful information or noise, Channel Attention cannot automatically learn the insignificance of these channels. Instead, it tries to enhance their expression capability by adjusting the weights. It may lead to excessive focus on noise information while neglecting more essential features. In this paper, we propose a new image SR method, RSVTCANet, based on an extension of Swin2SR. Specifically, to effectively gather global information for the channel of images, we modify the Residual SwinV2 Transformer blocks in Swin2SR by introducing the coordinate attention for each two successive SwinV2 Transformer Layers (S2TL) and replacing Multi-head Self-Attention (MSA) with Efficient Multi-head Self-Attention version 2 (EMSAv2) to employ the resulting residual SwinV2 Transformer coordinate attention blocks (RSVTCABs) for feature extraction. Additionally, to improve the generalization of RSVTCANet during training, we apply an optimized RandAugment for data augmentation on the training dataset. Extensive experimental results show that RSVTCANet outperforms the recent image SR method regarding visual quality and measures such as PSNR and SSIM.

Fast Multiphoton Microscopic Imaging Joint Image Super-Resolution for Automated Gleason Grading of Prostate Cancers.

Gleason grading system is dependable for quantifying prostate cancer. This paper introduces a fast multiphoton microscopic imaging method via deep learning for automatic Gleason grading. Due to the contradiction between multiphoton microscopy (MPM) imaging speed and quality, a deep learning architecture (SwinIR) is used for image super-resolution to address this issue. The quality of low-resolution image is improved, which increased the acquisition speed from 7.55 s per frame to 0.24 s per frame. A classification network (Swin Transformer) was introduced for automated Gleason grading. The classification accuracy and Macro-F1 achieved by training on high-resolution images are respectively 90.9% and 90.9%. For training on super-resolution images, the classification accuracy and Macro-F1 are respectively 89.9% and 89.9%. It shows that super-resolution image can provide a comparable performance to high-resolution image. Our results suggested that MPM joint image super-resolution and automatic classification methods hold the potential to be a real-time clinical diagnostic tool for prostate cancer diagnosis.

Self-supervised memory learning for scene text image super-resolution

Computerised recognition of low-resolution scene text images has been a persistent challenge. To improve the recognition performance, image quality enhancement via image super-resolution technology provides an intuitive solution. Typical deep learning-based scene text image super-resolution methods assume that the image quality degradation from high-resolution images to their corresponding low-resolution counterparts can be represented by mapping well-distributed samples, which limits their reconstruction performance in a practical text recognition system. For real-world scenarios this assumption typically does not hold since image degradations arise from multiple sources during image capture and processing. In this paper, to alleviate this problem, we propose a novel self-supervised end-to-end memory network model for scene text image super-resolution. In particular, after extracting enriched and finer representations from low-resolution text images via a spatial refinement block, we introduce a memory-based network to yield an improved super-resolution model that can handle complex degradation sources. Furthermore, to boost the effectiveness of our method, we design a multi-term loss to exploit textual structure information, where, in addition to the traditional reconstruction loss, we embed a character perceptual loss and a boundary enhancement loss. Extensive experiments on different datasets demonstrate that our proposed MNTSR method effectively improves the recognition accuracy for several scene text image recognition models and achieves state-of-the-art results. The source code is made available at https://github.com/xyzhu1/MNTSR.

A Single-Frame and Multi-Frame Cascaded Image Super-Resolution Method.

The objective of image super-resolution is to reconstruct a high-resolution (HR) image with the prior knowledge from one or several low-resolution (LR) images. However, in the real world, due to the limited complementary information, the performance of both single-frame and multi-frame super-resolution reconstruction degrades rapidly as the magnification increases. In this paper, we propose a novel two-step image super resolution method concatenating multi-frame super-resolution (MFSR) with single-frame super-resolution (SFSR), to progressively upsample images to the desired resolution. The proposed method consisting of an L0-norm constrained reconstruction scheme and an enhanced residual back-projection network, integrating the flexibility of the variational model-based method and the feature learning capacity of the deep learning-based method. To verify the effectiveness of the proposed algorithm, extensive experiments with both simulated and real world sequences were implemented. The experimental results show that the proposed method yields superior performance in both objective and perceptual quality measurements. The average PSNRs of the cascade model in set5 and set14 are 33.413 dB and 29.658 dB respectively, which are 0.76 dB and 0.621 dB more than the baseline method. In addition, the experiment indicates that this cascade model can be robustly applied to different SFSR and MFSR methods.

Deep blind super-resolution for hyperspectral images

Current deep learning methods for single hyperspectral image super-resolution are non-blind ones, which adopt the simplistic bicubic degradation model. These models have poor generalization performance and cannot handle unknown degradation. Additionally, blind super-resolution methods for RGB images neglect rich spectral information in hyperspectral images, which causes spectral distortion in results. To address these issues, we consider the degeneration estimation and propose a single hyperspectral image blind super-resolution algorithm. Specifically, we first use a blur kernel estimation network and a deblurring network to obtain the image without blur. We change the receptive field by exchanging spatial channel information when estimating the kernel, thereby obtaining blurry information at different scales. In addition, to reduce the errors introduced by kernel mismatch and the relevant information among bands ignored by the group convolution, we fuse the spatial information of the blurred image to compensate for the errors and then utilize the spatial and spectral information of other bands to guide the feature extraction of the current band. Finally, we integrate the feature information from different branches in a global fusion network to improve spatial and spectral information fidelity further. Extensive experiments on synthetic and real-world hyperspectral datasets unequivocally demonstrate that our approach outperforms other state-of-the-art approaches in terms of both quantitative and qualitative evaluations. Our code is publicly available at https://github.com/YoungP2001/DBSR.

Synergizing frequency domain and texture-aware implicit module for MRI arbitrary-scale super-resolution

Arbitrary-scale MR image super-resolution (SR) methods have emerged as a prominent research topic in medical imaging due to their ability to address the demand for continuous resolution in medical image analysis and diagnostics. Although several methods currently employ Implicit Neural Representations (INR) to achieve arbitrary-scale SR for MR images, they still face the following issues that constrain further enhancements in SR performance. (1) They solely perform feature extraction on MR images in the image domain, disregarding the frequency domain information in MR images, which results in aliasing artifacts and blurriness in the reconstructed SR images. (2) They utilize a standard multi-layer perceptron in INR, which tend to learn global structures (i.e., low-frequency components) and fail to capture and restore detailed texture information (i.e., high-frequency components) in MR images adequately. To address these issues, we propose a brand-new method called FTNet, which introduces the frequency domain information and employs the texture-aware implicit module for MR image arbitrary-scale SR. Specifically, we design an additional branch to extract the feature of MR images in the frequency domain to maximize removing aliasing artifacts. To further improve the quality of SR images, we introduce a texture extraction branch and a texture-aware implicit module to capture and restore detailed texture information within the MR images. Extensive experiments demonstrate the superiority of our network over state-of-the-art methods and its great potential for clinical practice.

Simultaneous Mapping of the Nanoscale Organization and Redox State of Extracellular Space in Living Brain Tissue.

The spatial organization characteristics and redox status of the extracellular space (ECS) are crucial in the development of brain diseases. However, it remains a challenge to simultaneously capture dynamic changes in microstructural features and redox states at the submicron level within the ECS. Here, we developed a reversible glutathione (GSH)-responsive nanoprobe (RGN) for mapping the spatial organization features and redox status of the ECS in brain tissues with nanoscale resolution. The RGN is composed of polymer nanoparticles modified with GSH-responsive molecules and amino-functionalized methoxypoly(ethylene glycol), which exhibit exceptional single-particle brightness and excellent free diffusion capability in the ECS of brain tissues. Tracking single RGNs in acute brain slices allowed us to dynamically map spatial organizational features and redox levels within the ECS of brain tissues in disease models. This provides a powerful super-resolution imaging method that offers a potential opportunity to study the dynamic changes in the ECS microenvironment and to understand the physiological and pathological roles of the ECS in vivo.

Terahertz super-resolution imaging with large depth of field enabled by the terajet of bilayer dielectric spheres.

Terahertz imaging has found extensive applications in non-destructive testing, security inspection, and other various fields. Intensive research on terahertz imaging systems has been executed to pursue high performance on imaging resolution and depth of field (DOF). However, the terahertz imaging systems with both high imaging resolution and large DOF have rarely been reported. In this paper, a mesoscopic-sized dielectric bilayer sphere-assisted super-resolution imaging method was proposed to simultaneously achieve enormously improved imaging resolution and extended DOF. Simulation analyses revealed that the ultrathin and long terajets were generated by the well-designed bilayer sphere. The THz super-resolution image for the samples can be captured by the point-by-point terajet scanning. The experimental results demonstrated that the best resolution reached up to 0.4λ, the DOF with super-resolution was up to 2λ, and the DOF with sub-wavelength resolution was up to 4λ. This method holds great potential for widespread application in terahertz imaging and detection, especially for curved or complex sample structures.

Unsupervised deep learning method for single image super-resolution of the thick pinhole imaging system using deep image prior

Thick pinhole imaging system is widely used for diagnosing intense pulsed radiation sources. However, owing to the trade-off among spatial resolution, field of view (FOV) and signal-to-noise ratio (SNR), the imaging system normally falls short in achieving high-precision spatial diagnosis. In this paper, we propose an unsupervised deep learning method for single image super-resolution (SISR) of the thick pinhole imaging system. The point spread function (PSF) of the imaging system is obtained by analytical calculation and Monte Carlo simulation methods, and the mathematical model of the imaging system is established using a linear equation. To solve the ill-posed inverse problem, we adopt randomly initialized deep convolutional neural networks (DCNNs) as an image prior without pre-training, which is named deep image prior (DIP). The results demonstrate that, by utilizing the SISR technique to increase the number of pixels in reconstructed images, the proposed DIP algorithm can mitigate the spatial resolution degradation caused by an insufficient spatial sampling frequency of the camera. Compared with various classical algorithms, the proposed DIP algorithm exhibits superior capabilities in recovering high-frequency signals and suppressing ringing artifacts. Furthermore, the convergence and robustness of the proposed DIP algorithm under different random seeds and SNR conditions are also verified.

MambaSR: Arbitrary-Scale Super-Resolution Integrating Mamba with Fast Fourier Convolution Blocks

Traditional single image super-resolution (SISR) methods, which focus on integer scale super-resolution, often require separate training for each scale factor, leading to increased computational resource consumption. In this paper, we propose MambaSR, a novel arbitrary-scale super-resolution approach integrating Mamba with Fast Fourier Convolution Blocks. MambaSR leverages the strengths of the Mamba state-space model to extract long-range dependencies. In addition, Fast Fourier Convolution Blocks are proposed to capture the global information in the frequency domain. The experimental results demonstrate that MambaSR achieves superior performance compared to different methods across various benchmark datasets. Specifically, on the Urban100 dataset, MambaSR outperforms MetaSR by 0.93 dB in PSNR and 0.0203 dB in SSIM, and on the Manga109 dataset, it achieves an average PSNR improvement of 1.00 dB and an SSIM improvement of 0.0093 dB. These results highlight the efficacy of MambaSR in enhancing image quality for arbitrary-scale super-resolution.

A Novel Electrical Equipment Status Diagnosis Method Based on Super-Resolution Reconstruction and Logical Reasoning.

The accurate detection of electrical equipment states and faults is crucial for the reliable operation of such equipment and for maintaining the health of the overall power system. The state of power equipment can be effectively monitored through deep learning-based visual inspection methods, which provide essential information for diagnosing and predicting equipment failures. However, there are significant challenges: on the one hand, electrical equipment typically operates in complex environments, thus resulting in captured images that contain environmental noise, which significantly reduces the accuracy of state recognition based on visual perception. This, in turn, affects the comprehensiveness of the power system's situational awareness. On the other hand, visual perception is limited to obtaining the appearance characteristics of the equipment. The lack of logical reasoning makes it difficult for purely visual analysis to conduct a deeper analysis and diagnosis of the complex equipment state. Therefore, to address these two issues, we first designed an image super-resolution reconstruction method based on the Generative Adversarial Network (GAN) to filter environmental noise. Then, the pixel information is analyzed using a deep learning-based method to obtain the spatial feature of the equipment. Finally, by constructing the logic diagram for electrical equipment clusters, we propose an interpretable fault diagnosis method that integrates the spatial features and temporal states of the electrical equipment. To verify the effectiveness of the proposed algorithm, extensive experiments are conducted on six datasets. The results demonstrate that the proposed method can achieve high accuracy in diagnosing electrical equipment faults.

Deep Residual-in-Residual Model-Based PET Image Super-Resolution with Motion Blur

Positron emission tomography (PET) is a non-invasive molecular imaging technique. The limited spatial resolution of PET images, due to technological and physical imaging constraints, directly affects the precise localization and interpretation of small lesions and biological processes. The super-resolution (SR) technique aims to enhance image quality by improving spatial resolution, thereby aiding clinicians in achieving more accurate diagnoses. However, most conventional SR methods rely on idealized degradation models and fail to effectively capture both low- and high-frequency information present in medical images. For the challenging SR reconstruction of PET images exhibiting motion-induced artefacts, a degradation model that better aligns with practical scanning scenarios was designed by us. Furthermore, we proposed a PET image SR method based on the deep residual-in-residual network (DRRN), focusing on the recovery of both low- and high-frequency information. By incorporating multi-level residual connections, our approach facilitates direct feature propagation across different network levels. This design effectively mitigates the lack of feature correlation between adjacent convolutional layers in deep networks. Our proposed method surpasses benchmark methods in both full-reference and no-reference metrics and subjective visual effects across small animal PET (SAPET), phantoms, and Alzheimer’s Disease Neuroimaging Initiative (ADNI) datasets. The experimental findings confirm the remarkable efficacy of DRRN in enhancing spatial resolution and mitigating blurring in PET images. In comparison to conventional SR techniques, this method demonstrates superior proficiency in restoring low-frequency structural texture information while simultaneously maintaining high-frequency details, thus showcasing exceptional multi-frequency information fusion capabilities.