Super-resolution Field Research Articles

Frequency Domain-Based Super Resolution Using Two-Dimensional Structure Consistency for Ultra-High-Resolution Display

Recent advancements in the field of super resolution (SR) have seen the adoption of generative adversarial networks (GANs) for realistic images. In this case, when performing with low-resolution (LR) images, several challenges arise due to the loss of high-frequency details from high-resolution (HR) images, potentially leading to unwanted distortions in the generated SR images. Our paper presents a novel solution by using two-dimensional structure consistency (TSC) for image analysis. The TSC serves as a mask, enabling adaptive analysis based on the unique frequency characteristics of different image regions. Furthermore, a mutual loss mechanism, which dynamically adjusts the training process based on the results filtered by the TSC-based mask, is introduced. Additionally, the TSC loss is proposed to enhance our model capacity to generate precise TSC in high-frequency regions. As a result, our method effectively reduces distortions in high-frequency areas while preserving clarity in regions containing low-frequency components. Our method outperforms other SR techniques, demonstrating superior results in both qualitative and quantitative evaluations. Quantitative measurements, including PSNR, SSIM, and the perceptual metric LPIPS, show comparable PSNR and SSIM values, while the perceptual SR quality is notably improved according to the LPIPS metric.

Spatial–angular–epipolar transformer for light field spatial and angular super-resolution

Transformer-based light field (LF) super-resolution (SR) methods have recently achieved significant performance improvements due to global feature modeling by self-attention mechanisms. However, as a method designed for natural language processing, 4D LFs are reshaped into 1D sequences with an immense set of tokens, which results in a quadratic computational complexity cost. In this paper, a spatial–angular–epipolar swin transformer (SAEST) is proposed for spatial and angular SR (SASR), which sufficiently extracts SR information in the spatial, angular, and epipolar domains using local self-attention with shifted windows. Specifically, in SAEST, a spatial swin transformer and an angular standard transformer are firstly cascaded to extract spatial and angular SR features, separately. Then, the extracted SR feature is reshaped into the epipolar plane image pattern and fed into an epipolar swin transformer to extract the spatial–angular correlation information. Finally, several SAEST blocks are cascaded in a Unet framework to extract multi-scale SR features for SASR. Experiment results indicate that SAEST is a fast transformer-based SASR method with less running time and GPU consumption and has outstanding performance on simulated and real-world public datasets.

Local Means Binary Networks for Image Super-Resolution.

The success of modern single image super-resolution (SISR) algorithms is inspired by the development of deep convolutional neural networks (CNNs). However, these CNN-based methods require considerable computation and complexity, making it impossible for these methods to perform real-time calculations in edge devices. Thus, lightweight model design has become a development trend in the super-resolution field, including pruning, quantization, and other methods. The 1-bit quantization is an extreme lightweight method which can reduce the calculation amount of the model in an extreme manner and is friendly to hardware such as edge devices. Most existing binary quantization approaches lead to a large information loss during forward propagation, especially in detailed color information (e.g., edge, texture, and contrast). The loss of color information makes modern binary methods unsuitable for SISR tasks. We think the loss occurs because these methods typically utilize a uniform threshold to quantize the weights and activations. Thus, in this article, we thoroughly analyze the difference between normal classification tasks and SISR tasks, and present a binarization scheme based on local means. The proposed method can maintain more detailed information in feature maps using dynamic thresholds during quantization. Specifically, each value in the full precision activations has a corresponding threshold during the quantization process, and those thresholds are determined by the full precision values of the surroundings. In addition, a gradient approximator is introduced to adaptively optimize the gradient for updating binary weights. We then verify the effectiveness of our method for training binary networks on several SISR benchmarks including VDSR and SRResNet. Experimental results show that the proposed method can outperform the state-of-the-art algorithms to obtain binary networks for image super-resolution with better peak signal-to-noise ratio (PSNR) values and visual quality.

Light Field Super-Resolution Using Decoupled Selective Matching

Light Field Super-Resolution Using Decoupled Selective Matching

Large-scale flow field super-resolution via local-global fusion convolutional neural networks

Particle image velocimetry (PIV) techniques have a limited field of view of the flow field and can only capture high-resolution flow fields in localized areas. To obtain a larger measurement range, multiple cameras must be used to capture the flow field simultaneously and then stitch the parts together. However, this method can be expensive. We propose the local-global fusion convolutional neural network (LGF-CNN) for reconstructing large-field flow fields with high spatial resolution based only on two flow data types: local small-field high spatial resolution wake velocity fields and global large-field low spatial resolution velocity fields. The core of the model consists of convolutional neural network (CNN) architecture to learn the mapping relationship between the small field of view with high spatial resolution and the large field of view with low spatial resolution. Using the effectively trained LGF-CNN model, we demonstrate its ability to reconstruct high-resolution velocity fields around the circular cylinder. The LGF-CNN is rigorously validated on a number of representative datasets, including simulated data for Reynolds numbers of 200 and 500, as well as experimental data for a Reynolds number of 3.3 × 104 with the steady jet at the rear stagnation point of the cylinder. The results demonstrate the ability of LGF-CNN to generate accurate velocity fields with high spatial resolution, including reliable detection of high-frequency components. The proposed method could reduce the number of cameras required for large-field, high spatial resolution PIV measurements, thereby reducing experimental costs.

Full field-of-view hexagonal lattice structured illumination microscopy based on the phase shift of electro-optic modulators.

High throughput has become an important research direction in the field of super-resolution (SR) microscopy, especially in improving the capability of dynamic observations. In this study, we present a hexagonal lattice structured illumination microscopy (hexSIM) system characterized by a large field of view (FOV), rapid imaging speed, and high power efficiency. Our approach employs spatial light interference to generate a two-dimensional hexagonal SIM pattern, and utilizes electro-optical modulators for high-speed phase shifting. This design enables the achievement of a 210-µm diameter SIM illumination FOV when using a 100×/1.49 objective lens, capturing 2048 × 2048 pixel images at an impressive 98 frames per second (fps) single frame rate. Notably, this method attains a near 100% full field-of-view and power efficiency, with the speed limited only by the camera's capabilities. Our hexSIM demonstrates a substantial 1.73-fold improvement in spatial resolution and necessitates only seven phase-shift images, thus enhancing the imaging speed compared to conventional 2D-SIM.

A Flow Field Super-resolution Strategy for Direct Numerical Simulation Based on Physics-informed Convolutional Neural Networks

In the computational fluid dynamics method, the discretization of the solution domain has an important impact on the calculation results. The higher resolution grid improves the solution accuracy and is accompanied by a significant increase in the calculation time. How to improve efficiency under the premise of ensuring accuracy is of great significance in engineering. To this end, we propose a super-resolution strategy for direct numerical simulation (DNS): take the numerical simulation results at low-resolution grid as the initial solution, construct a model for super-resolution utilizing the convolutional neural networks, and embed the flow governing equations in the model to modify the initial solution. The proposed method is verified in the engineering case of pipeline transportation of non-Newtonian fluids. The results show that this strategy can improve the solution accuracy and shorten the simulation time. The deviation between the high-resolution results reconstructed by the model and the high-resolution flow field simulated by DNS is 63.18% lower than that of the low-resolution one simulated by DNS, and the calculation time is saved by 84.65%.

A COMPARISON OF DEEP LEARNING-BASED SUPER-RESOLUTION FRAMEWORKS FOR SENTINEL-2 IMAGERY IN URBAN AREAS

Abstract. The high-resolution images are in demand for many applications in the monitoring of urban areas. The advent of remote sensing satellites such as Sentinel-2 has made data more accessible as it provides free multispectral imagery. However, the spatial resolution of these images is not sufficient for many of the tasks. With the advent of deep learning techniques, significant progress has been made in the field of super-resolution, which has shown promising results in the improvement of the spatial resolution of satellite images. In this study, we compare four the most common deep learning-based models for the super-resolution of Sentinel-2 imagery in dense urban areas using aerial images. These methods are including enhanced deep super-resolution network (EDSR), super-resolution generative adversarial networks (ESRGAN), residual feature distillation network (RFDN), and Super-Resolution Convolutional Neural Network (SRCNN). To determine the effectiveness of the models in improving image resolution, they were evaluated using visual quality and quantitative metrics. The super-resolution results show that deep learning-based models have high potential for the generation of the high-resolution dataset from Sentinel-2 imagery in urban areas. The RFDN outperformed other deep learning-based models that achieved the peak signal-to-noise ratio (PSNR) more than 17.8.

Adaptive pixel aggregation for joint spatial and angular super-resolution of light field images

Recent advancements in Light Field (LF) super-resolution (SR) have significantly enhanced the spatial size and angular sampling rate in LF imagery. Typically, most methods treat spatial and angular SR as separate tasks. However, the two tasks are closely related and can be jointly optimized by leveraging the intrinsic high-dimensional property of LF images. Existing methods for joint spatial and angular SR of LF images also suffer from the limitations of disparity-based warping operations. To address these challenges, we propose a novel one-stage approach for directly synthesizing densely-sampled high-resolution LF images from sparsely-sampled low-resolution observations. Specifically, we propose to adaptively aggregate advantageous pixels from different sub-aperture images of input LF for each novel view interpolation. This adaptive strategy enables our method to effectively incorporate the spatial and angular correlations of input views. Additionally, to enhance the parallax structure of reconstructed LF images, we propose a feature separation and interaction module to refine the intermediate features. Experimental results on public datasets demonstrate that our proposed method can achieve state-of-the-art performance both numerically and visually compared with previous methods. Our codes are available at https://github.com/GaoshengLiu/LFSR.

Cross-View Recurrence-Based Self-Supervised Super-Resolution of Light Field

Compared with external-supervised learning-based (ESLB) methods, self-supervised learning-based (SSLB) methods can overcome the domain gap problem caused by different light field (LF) acquisition conditions, which results in the performance degradation of light field super-resolution on unseen test datasets. Current SSLB methods exploit the cross-scale recurrence feature in the single view image for super-resolution, ignoring the correlation information among views. Different from previous works, we propose a cross-view recurrence-based self-supervised mapping framework to correlate complementary information among views in the down-scaled input LF. Specifically, the cross-view recurrence information consists of geometry structure features and similar structure features. The former is to provide sub-pixel information according to disparity correlations among adjacent views, and the latter is to acquire similar color and contour information among arbitrary views, which can compensate for error disparity guidance of geometry structure features in sharp variance areas. Moreover, instead of the widely used “All-to-All” strategy, we propose a “Part-to-Part” mapping strategy, which is better competent for SSLB approaches with limited training examples solely extracted from input LF. Finally, considering that self-supervised methods need to retrain from the beginning toward each test image, based on the proposed “part-to-part” strategy, an efficient end-to-end network is designed to extract these cross-view features for superior SASR performance with less training time. Experiment results demonstrate that our method outperforms other state-of-the-art ESLB methods on both large and small domain gap cases. Compared with the only SSLB method (LFZSSR [1]), our approach achieves better performance with 524 times less training time.

KGSR: A kernel guided network for real-world blind super-resolution

In recent years, deep learning-based methods have emerged as dominant players in the field of super-resolution (SR), owing to their exceptional reconstruction performance. The primary driver of their effectiveness lies in their utilization of extensive sets of paired low-resolution and high-resolution images for training deep learning models. This training enables the models to effectively replicate the intricate mapping relationship between low-resolution and high-resolution images. Nevertheless, at present, acquiring a sufficient quantity of such image pairs that satisfy the requirements remains a formidable obstacle. Therefore, in order to break the restriction of limited training sets, self-supervised learning has been introduced to train a model for each low-quality image, without requiring pairwise ground-truths. However, they generally presuppose the generation of low-resolution (LR) images from their high-resolution (HR) counterparts using a pre-defined kernel, such as Bicubic downscaling. Such an assumption is seldom valid for real-world LR images, where degradation processes in practical applications are diverse, intricate, and often undisclosed. Therefore, when the presumed downscaling kernel does not match the actual one, the outcomes of state-of-the-art approaches degrade substantially. In this paper, we introduce KGSR, a kernel-guided network for addressing real-world blind SR, effectively avoiding requiring large training image pairs and transforming the blind image super-resolution problem into a supervised learning and non-blind scenario. Specifically, KGSR trains two networks, namely Upscaling and Downscaling, utilizing only patches extracted from the input test image. On one hand, owing to the cross-scale recurrence property of the SR kernel within a single image, the Downscaling network acquires knowledge of the image-specific degradation process through a generative adversarial network. Consequently, the Downscaling network is capable of generating a downsampled version of the LR test image even when the acquisition process is unknown or less than ideal. Additionally, we employ a dedicated discriminator to compel the Downscaling network to prioritize the characterization of kernel orientations. Conversely, a precise blur kernel has the potential to yield superior performance. Guided by the accurate image-specific SR kernel acquired from the Downscaling network and the downsampled LR input, the Upscaling network is capable of producing a high-quality HR image from the LR input. Within the Upscaling network, we additionally introduce an effective module for harnessing the acquired image-specific SR kernel. KGSR operates as a fully unsupervised approach, yet it can concurrently produce both the image-specific SR kernel and high-quality HR images. Comprehensive experiments conducted on standard benchmarks validate the efficacy of the proposed approach compared to state-of-the-art methodologies. Moreover, the suggested method can deliver visually appealing SR outcomes while exhibiting shorter processing times when applied to real-world LR images.

AIF-LFNet: All-in-Focus Light Field Super-Resolution Method Considering the Depth-Varying Defocus

As an aperture-divided computational imaging system, microlens array (MLA) -based light field (LF) imaging is playing an increasingly important role in computer vision. As the trade-off between the spatial and angular resolutions, deep learning (DL) -based image super-resolution (SR) methods have been applied to enhance the spatial resolution. However, in existing DL-based methods, the depth-varying defocus is not considered both in dataset development and algorithm design, which restricts many applications such as depth estimation and object recognition. To overcome this shortcoming, a super-resolution task that reconstructs all-in-focus high-resolution (HR) LF images from low-resolution (LR) LF images is proposed by designing a large dataset and proposing a convolutional neural network (CNN) -based SR method. The dataset is constructed by using Blender software, consisting of 150 light field images used as training data, and 15 light field images used as validation and testing data. The proposed network is designed by proposing the dilated deformable convolutional network (DCN) -based feature extraction block and the LF subaperture image (SAI) Deblur-SR block. The experimental results demonstrate that the proposed method achieves more appealing results both quantitatively and qualitatively.

Light field super-resolution using complementary-view feature attention

Light field (LF) cameras record multiple perspectives by a sparse sampling of real scenes, and these perspectives provide complementary information. This information is beneficial to LF super-resolution (LFSR). Compared with traditional single-image super-resolution, LF can exploit parallax structure and perspective correlation among different LF views. Furthermore, the performance of existing methods are limited as they fail to deeply explore the complementary information across LF views. In this paper, we propose a novel network, called the light field complementary-view feature attention network (LF-CFANet), to improve LFSR by dynamically learning the complementary information in LF views. Specifically, we design a residual complementary-view spatial and channel attention module (RCSCAM) to effectively interact with complementary information between complementary views. Moreover, RCSCAM captures the relationships between different channels, and it is able to generate informative features for reconstructing LF images while ignoring redundant information. Then, a maximum-difference information supplementary branch (MDISB) is used to supplement information from the maximum-difference angular positions based on the geometric structure of LF images. This branch also can guide the process of reconstruction. Experimental results on both synthetic and real-world datasets demonstrate the superiority of our method. The proposed LF-CFANet has a more advanced reconstruction performance that displays faithful details with higher SR accuracy than state-of-the-art methods.

MFSRNet: spatial-angular correlation retaining for light field super-resolution

MFSRNet: spatial-angular correlation retaining for light field super-resolution

SOGO-SOFI, light-modulated super-resolution optical fluctuation imaging using only 20 raw frames for high-fidelity reconstruction

Taking advantage of the stochastic photoswitching of genetically encodable reversibly photoswitchable fluorescent proteins (RSFPs), super-resolution optical fluctuation imaging (SOFI) and its variant photochromic stochastic optical fluctuation imaging (pcSOFI) are valuable tools for wide field super-resolution (SR) imaging. Live-cell (pc)SOFI, which requires a small number of original frames to reconstruct an SR image, is prone to structural discontinuity artifacts and low spatial resolution. Herein, we developed a repeated synchronized on- and gradually off-switching SOFI (SOGO-SOFI) that maximized the photoswitching frequency of RSFPs by light modulation and required only 20 frames for high-quality reconstruction. Live-cell SOGO-SOFI imaging of the endoplasmic reticulum (ER) exhibited 10 times higher temporal resolution (100 fps) and fewer artifacts than pcSOFI. Moreover, a combination of SOGO-SOFI with Airyscan further increased the image contrast and the resolution of Airyscan by a factor of 1.5 from 140 nm to 91 nm. The capabilities of SOGO-SOFI were further demonstrated by dual-color imaging of nucleolar proteins in mammalian cells and deep imaging of ER structures in thick brain slices (20.6 µm).

MFSR: Light Field Images Spatial Super Resolution Model Integrated with Multiple Features

Light Field (LF) cameras can capture angular and spatial information simultaneously, making them suitable for a wide range of applications such as refocusing, disparity estimation, and virtual reality. However, the limited spatial resolution of the LF images hinders their applicability. In order to address this issue, we propose an end-to-end learning-based light field super-resolution (LFSR) model called MFSR, which integrates multiple features, including spatial, angular, epipolar plane images (EPI), and global features. These features are extracted separately from the LF image and then fused together to obtain a comprehensive feature using the Feature Extract Block (FE Block) iteratively. Gradient loss is added into the loss function to ensure that the MFSR has good performance for LF images with rich texture. Experimental results on synthetic and real-world datasets demonstrate that the proposed method outperforms other state-of-the-art methods, with a peak signal-to-noise ratio (PSNR) improvement of 0.208 dB and 0.274 dB on average for the 2× and 4× super-resolution tasks, and structural similarity (SSIM) of both improvements of 0.01 on average.

Fourier Ptychographic Reconstruction Method of Self-Training Physical Model

Fourier ptychographic microscopy is a new microscopic computational imaging technology. A series of low-resolution intensity images are collected by a Fourier ptychographic microscopy system, and high-resolution intensity and phase images are reconstructed from the collected low-resolution images by a reconstruction algorithm. It is a kind of microscopy that can achieve both a large field of view and high resolution. Here in this article, a Fourier ptychographic reconstruction method applied to a self-training physical model is proposed. The SwinIR network in the field of super-resolution is introduced into the reconstruction method for the first time. The input of the SwinIR physical model is modified to a two-channel input, and a data set is established to train the network. Finally, the results of high-quality Fourier stack microscopic reconstruction are realized. The SwinIR network is used as the physical model, and the network hyperparameters and processes such as the loss function and optimizer of the custom network are reconstructed. The experimental results show that by using multiple different types of data sets, the two evaluation index values of the proposed method perform best, and the image reconstruction quality is the best after model training. Two different evaluation indexes are used to quantitatively analyze the reconstruction results through numerical results. The reconstruction results of the fine-tuning data set with some real captured images are qualitatively analyzed from the visual effect. The results show that the proposed method is effective, the network model is stable and feasible, the image reconstruction is realized in a short time, and the reconstruction effect is good.

Speckle structured illumination endoscopy with enhanced resolution at wide field of view and depth of field

Structured illumination microscopy (SIM) is one of the most widely applied wide field super resolution imaging techniques with high temporal resolution and low phototoxicity. The spatial resolution of SIM is typically limited to two times of the diffraction limit and the depth of field is small. In this work, we propose and experimentally demonstrate a low cost, easy to implement, novel technique called speckle structured illumination endoscopy (SSIE) to enhance the resolution of a wide field endoscope with large depth of field. Here, speckle patterns are used to excite objects on the sample which is then followed by a blind-SIM algorithm for super resolution image reconstruction. Our approach is insensitive to the 3D morphology of the specimen, or the deformation of illuminations used. It greatly simplifies the experimental setup as there are no calibration protocols and no stringent control of illumination patterns nor focusing optics. We demonstrate that the SSIE can enhance the resolution 2–4.5 times that of a standard white light endoscopic (WLE) system. The SSIE presents a unique route to super resolution in endoscopic imaging at wide field of view and depth of field, which might be beneficial to the practice of clinical endoscopy.

Neural Radiance Field-Based Light Field Super-Resolution in Angular Domain

Neural Radiance Field-Based Light Field Super-Resolution in Angular Domain

Progressive Multi-Scale Fusion Network for Light Field Super-Resolution

Light field (LF) cameras can record multi-view images from a single scene, and these images can provide spatial and angular information to improve the performance of image super-resolution (SR). However, it is a challenge to incorporate distinctive information from different LF views. At the same time, due to the inherent resolution of the image sensor, high spatial and angular resolution are trade-off problems. In this paper, we propose a progressive multi-scale fusion network (PMFN) to improve the LFSR performance. Specifically, a progressive feature fusion block (PFFB) based on an encoder-and-decoder structure is designed to implicitly align disparities and integrate complementary information between complementary views. The core module of the PFFB is a dual-branch multi-scale fusion module (DMFM), which can integrate the information from a reference view and auxiliary views to produce a fusion feature. Each DMFM consists of two parallel branches, which have different receptive fields to fuse hierarchical features from complementary views. Three DMFMs with a dense connection are used in the PFFB, which can fully exploit multi-level features to improve the SR performance. Experimental results on both synthetic and real-world datasets demonstrate that the proposed model achieves state-of-the-art performance among existing methods. Moreover, quantitative results show that our method can also generate faithful details.