Super-resolution Research Articles

Dark-field super-oscillatory microscopy with high-pass super-oscillatory filters

Super-resolution imaging of transparent objects is particularly challenging due to the interference caused by background light. This interference can significantly degrade image quality by reducing contrast and obscuring fine details. In this work, high-pass super-oscillatory masks are designed to address this issue by filtering out the background light on the Fourier plane. The resolution of the corresponding super-oscillatory point-spread function is 0.66 times the diffraction limit, enabling the resolution of transparent objects with 67% transparency at sub-wavelength scales. Additionally, the high-pass super-oscillatory masks are designed for a 1080 × 1920 liquid-crystal spatial light modulator, making future experimental implementation straightforward. This facilitates easy switching between bright-field and dark-field super-oscillatory microscopy, broadening the range of application scenarios for the system.

Novel intercellular spread mode of respiratory syncytial virus contributes to neutralization escape

Developing widely used respiratory syncytial virus (RSV) vaccines remains a significant challenge, despite the recent authorization of two pre-F vaccines for elderly adults. Previous reports have suggested that even when vaccine-induced immunity generates high titers of potent neutralizing antibodies targeting the pre-F protein, it may not fully inhibit breakthrough of RSV infections. This incomplete inhibition of RSV breakthrough infections can lead to an increased risk of enhanced respiratory disease (ERD) in vaccinated individuals. The reasons why potent neutralizing antibodies cannot fully prevent RSV breakthrough infections are not yet clear. In an attempt to explain this phenomenon, we investigated the effect of potent neutralizing antibodies on the intercellular spread of RSV. Our findings indicated that a specific titer of potent neutralizing antibodies, such as 5C4, could block certain modes of intercellular spread, such as the diffusion of cell-free virions and the delivery of virions through filopodia. However, these antibodies did not fully inhibit the entire process of intercellular spread. Through the use of super-resolution imaging techniques, we observed a novel and efficient spread mode called the transition of viral materials through intercellular nanotubes (TVMIN), independent of virions and insensitive to the presence of antibodies. TVMIN allowed RSV-infected cells to directly transfer viral materials to neighboring cells via intercellular nanotubes that are rich in microfilaments. TVMIN began as early as 5 h post-infection (h.p.i.) and rapidly initiated infection in recipient cells. Our data provided new insights into the intercellular spread of RSV and might help explain the occurrence of breakthrough infections.

Learning accurate and enriched features for stereo image super-resolution

Learning accurate and enriched features for stereo image super-resolution

Enhanced local multi-windows attention network for lightweight image super-resolution

Since the global self-attention mechanism can capture long-distance dependencies well, Transformer-based methods have achieved remarkable performance in many vision tasks, including single-image super-resolution (SISR). However, there are strong local self-similarities in images, if the global self-attention mechanism is still used for image processing, it may lead to excessive use of computing resources on parts of the image with weak correlation. Especially in the high-resolution large-size image, the global self-attention will lead to a large number of redundant calculations. To solve this problem, we propose the Enhanced Local Multi-windows Attention Network (ELMA), which contains two main designs. First, different from the traditional self-attention based on square window partition, we propose a Multi-windows Self-Attention (M-WSA) which uses a new window partitioning mechanism to obtain different types of local long-distance dependencies. Compared with original self-attention mechanisms commonly used in other SR networks, M-WSA reduces computational complexity and achieves superior performance through analysis and experiments. Secondly, we propose a Spatial Gated Network (SGN) as a feed-forward network, which can effectively overcome the problem of intermediate channel redundancy in traditional MLP, thereby improving the parameter utilization and computational efficiency of the network. Meanwhile, SGN introduces spatial information into the feed-forward network that traditional MLP cannot obtain. It can better understand and use the spatial structure information in the image, and enhances the network performance and generalization ability. Extensive experiments show that ELMA achieves competitive performance compared to state-of-the-art methods while maintaining fewer parameters and computational costs.

Corrigendum to “CVGSR: Stereo image super-resolution with cross-view guidance” [Displays 83 (2024) 102736

Corrigendum to “CVGSR: Stereo image super-resolution with cross-view guidance” [Displays 83 (2024) 102736

Confidential Image Super-Resolution with Privacy Protection

Confidential Image Super-Resolution with Privacy Protection

A First-Order Image Super-Resolution Model Promoting Sharp Boundaries and Suppressing the Staircase Effect

A First-Order Image Super-Resolution Model Promoting Sharp Boundaries and Suppressing the Staircase Effect

Representative Image Outpainting and Image Super-Resolution Methods Based on Deep Learning

The image generation based on deep learning is a technology that can generate new images or outpaint old images or improve visual effect of old images through learning from input data, according to deep learning structure. The representative technologies of image generation are image outpainting and image super-resolution. Deep learning is widely utilized in the field of computer vision. Facing different needs, it is essential to choose proper ways. This essay reviews several representative and new methods of image outpainting and image super-resolution. Compared with the results generated by old methods, the results generated by new image outpainting methods introduced in this essay have greater precision and clarity, the methods of image super-resolution that are suitable for all fields and professional fields can learn from each others dataset manually to train and develop. There is still unimaginable prospect of deep learning in the field of image outpainting and image super-resolution. Therefore, it is worthy of attention.

Simultaneous single image super‐resolution and blind Gaussian denoising via slim ghost full‐frequency residual blocks

AbstractGiven that super‐resolution (SR) aims to recover lost information, and low‐resolution (LR) images in real‐world conditions might be corrupted with multiple degradations, considering basic bicubic down‐sampling as the sole degradation significantly limits the performance of most existing SR models. This paper presents a model for simultaneous super‐resolution and blind additive white Gaussian noise (AWGN) denoising with two components (netdeg and netSR) that is based on a generative adversarial network (GAN) to achieve detailed results. netdeg, featuring residual and innovative cost‐effective ghost residual blocks with a frequency separation module for obtaining long‐range information, blindly restores a clean version of the LR image. netSR leverages slim ghost full‐frequency residual blocks to process low‐frequency (LF) and high‐frequency (HF) information via static large convolutions and pixel‐wise highlighted input‐adaptive dynamic convolutions, respectively. To address the susceptibility of dynamic layers to noise and preserve feature diversity while reducing model’s costs, static and dynamic layer features are combined and highlighted. Diverse and non‐redundant features are then processed using ghost‐style blocks. The proposed model achieves comparable SR results in bicubic down‐sampling scenarios, outperform existing SR methods in the complex task of concurrent SR and AWGN denoising, and demonstrate robustness in handling images corrupted with varying levels of AWGN.

Self-learning based joint multi image super-resolution and sub-pixel registration

Multi Image Super-resolution (MISR) refers to the task of enhancing the spatial resolution of a stack of low-resolution (LR) images representing the same scene. Although many deep learning-based single image super-resolution (SISR) technologies have recently been developed, deep learning has not been widely exploited for MISR, even though it can achieve higher reconstruction accuracy because more information can be extracted from the stack of LR images. One of the primary obstacles encountered by deep networks when addressing the MISR problem is the variability in the number of LR images that act as input to the network. This impedes the feasibility of adopting an end-to-end learning approach, because the varying number of input images makes it difficult to construct a training dataset for the network. Another challenge arises from the requirement to align the LR input images to generate high-resolution (HR) image of high quality, which requires complex and sophisticated methods.In this paper, we propose a self-learning based method that can simultaneously perform super-resolution and sub-pixel registration of multiple LR images. The proposed method trains a neural network with only the LR images as input and without any true target HR images; i.e., the proposed method requires no extra training dataset. Therefore, it is easy to use the proposed method to deal with different numbers of input images. To our knowledge this is the first time that a neural network is trained using only LR images to perform a joint MISR and sub-pixel registration. Experimental results confirmed that the HR images generated by the proposed method achieved better results in both quantitative and qualitative evaluations than those generated by other deep learning-based methods.

Dual-path aggregation transformer network for super-resolution with images occlusions and variability

While Transformer-based approaches have recently achieved notable success in super-resolution, their extensive computational requirements impede widespread practical adoption. High-resolution meteorological satellite cloud imagery is essential for weather analysis and forecasting. Enhancing image resolution through super-resolution techniques facilitates the accurate identification and localization of geographic features by meteorological systems. However, current super-resolution methods fail to restore the intricacies of cloud formations and complex regions fully. This research introduces a novel dual-path aggregation Transformer network (DPAT) tailored to enhance the super-resolution of meteorological satellite cloud images. The DPAT network adeptly captures cloud imagery's subtle details and textures, effectively addressing occlusions and the variability inherent in satellite imagery. It bolsters the model's ability to manage the complex attributes of cloud images through the introduction of the Dual-path Aggregation Self-Attention (DASA) mechanism and the Multi-scale Feature Aggregation Block (MFAB), thereby enhancing performance in processing intricate cloud features. The DASA mechanism synthesizes features across spatial, depth, and channel dimensions via a dual-path approach, thoroughly exploiting feature correlations. The MFAB, designed to supplant the multilayer perceptron, incorporates shift convolution and a multi-scale interaction block to augment feature information, compensating for the deficiency in local information absorption due to fixed receptive fields. Experimental outcomes indicate that DPAT delivers superior super-resolution outcomes. With a parameter count of only 32% of the Enhanced Deep Residual Network (EDSR) or 77% of the Image Restoration using Shift Window Transformer (SwinIR), DPAT matches SwinIR's performance on the satellite cloud dataset. Moreover, DPAT balances accuracy and parameter economy across various datasets. This technology is expected to improve image super-resolution capabilities in multiple fields such as human action recognition and industrial recognition, and indirectly improve the accuracy of image perception tasks.

Efficient Super-resolution of Phase Images Encoded with Random Phase Mask by machine learning techniques

Efficient Super-resolution of Phase Images Encoded with Random Phase Mask by machine learning techniques

Super-resolution imaging quality enhancement method for distributed array infrared camera

Abstract To address issues related to low resolution and high noise in infrared cameras, a distributed array infrared camera imaging system utilizing four cameras is proposed. The four cameras are arranged in an unconstrained array, and the combination algorithm of Projections onto Convex Sets (POCS) and Real-Enhanced Super-Resolution Generative Adversarial Networks (Real-ESRGAN) is applied to achieve high-quality super-resolution infrared imaging. The wavelet fusion algorithm is used to preprocess four low-resolution infrared images to reduce noise. Then, the POCS algorithm is used to reconstruct the preprocessed image. Finally, the Real-ESRGAN is employed for image reconstruction, resulting in an ultra-high-resolution infrared image. The results show that compared to single infrared camera imaging, the resolution of images reconstructed using the distributed infrared camera array is increased by 0.58 times, with the Modulation Transfer Function (MTF) increased by 1.2 times. Additionally, the entropy is increased by 18.87%, the standard deviation is increased by 13.51%, and the Naturalness Image Quality Evaluator (NIQE) is reduced by 18.87%. This demonstrates a significant enhancement in the super-resolution imaging quality of the distributed infrared camera array.

High-Fidelity and High-Efficiency Talking Portrait Synthesis With Detail-Aware Neural Radiance Fields.

In this paper, we propose a novel rendering framework based on neural radiance fields (NeRF) named HH-NeRF that can generate high-resolution audio-driven talking portrait videos with high fidelity and fast rendering. Specifically, our framework includes a detail-aware NeRF module and an efficient conditional super-resolution module. Firstly, a detail-aware NeRF is proposed to efficiently generate a high-fidelity low-resolution talking head, by using the encoded volume density estimation and audio-eye-aware color calculation. This module can capture natural eye blinks and high-frequency details, and maintain a similar rendering time as previous fast methods. Secondly, we present an efficient conditional super-resolution module on the dynamic scene to directly generate the high-resolution portrait with our low-resolution head. Incorporated with the prior information, such as depth map and audio features, our new proposed efficient conditional super resolution module can adopt a lightweight network to efficiently generate realistic and distinct high-resolution videos. Extensive experiments demonstrate that our method can generate more distinct and fidelity talking portraits on high resolution (900 × 900) videos compared to state-of-the-art methods. Our code is available at https://github.com/muyuWang/HHNeRF.

Comparative analysis of accelerated gradient algorithms for convex optimization: high and super resolution ODE approach

We investigate convex differentiable optimization and explore the temporal discretization of damped inertial dynamics driven by the gradient of the objective function. This leads to three accelerated gradient algorithms: Nesterov Accelerated Gradient (NAG), Ravine Accelerated Gradient (RAG), and (IGAHD). The latter was introduced by discretizing inertial dynamics with Hessian-driven damping to attenuate inherent oscillations in inertial methods. By analysing the high-resolution ODEs of order p = 0,1,2 for these algorithms, we gain insights into their similarities and differences. All three algorithms share the same low-resolution ODE of order 0, which is the dynamic proposed as a continuous surrogate for (NAG). To differentiate Nesterov from Ravine, we refine the comparison and demonstrate distinct high-resolution ODEs of order 2 in h (termed super-resolution). The corresponding Taylor expansions in h reveal matching terms of order 1 but differing terms of order 2. To the best of our knowledge, this result is completely new and emphasizes the need to avoid confusion between the Ravine and Nesterov methods in the literature. We present numerical experiments to illustrate our theoretical results. Performance profiles, measuring the number of iterations, indicate that (IGAHD) outperforms both (NAG) and (RAG) methods. (RAG) exhibits a slight advantage over (NAG) in terms of the average number of iterations. When considering CPU-time, both (RAG) and (NAG) outperform (IGAHD). All three algorithms exhibit similar behaviour when evaluating based on gradient norms.

A Feature-Driven Inception Dilated Network for Infrared Image Super-Resolution Reconstruction

Image super-resolution (SR) algorithms based on deep learning yield good visual performances on visible images. Due to the blurred edges and low contrast of infrared (IR) images, methods transferred directly from visible images to IR images have a poor performance and ignore the demands of downstream detection tasks. Therefore, an Inception Dilated Super-Resolution (IDSR) network with multiple branches is proposed. A dilated convolutional branch captures high-frequency information to reconstruct edge details, while a non-local operation branch captures long-range dependencies between any two positions to maintain the global structure. Furthermore, deformable convolution is utilized to fuse features extracted from different branches, enabling adaptation to targets of various shapes. To enhance the detection performance of low-resolution (LR) images, we crop the images into patches based on target labels before feeding them to the network. This allows the network to focus on learning the reconstruction of the target areas only, reducing the interference of background areas in the target areas’ reconstruction. Additionally, a feature-driven module is cascaded at the end of the IDSR network to guide the high-resolution (HR) image reconstruction with feature prior information from a detection backbone. This method has been tested on the FLIR Thermal Dataset and the M3FD Dataset and compared with five mainstream SR algorithms. The final results demonstrate that our method effectively maintains image texture details. More importantly, our method achieves 80.55% mAP, outperforming other methods on FLIR Dataset detection accuracy, and with 74.7% mAP outperforms other methods on M3FD Dataset detection accuracy.

Clog P-Guided Development of Multi-Colored Buffering Fluorescent Probes for Super-Resolution Imaging of Lipid Droplet Dynamics.

Super-resolution fluorescence imaging of live cells increasingly demands fluorescent probes capable of multi-color and long-term dynamic imaging. Understanding the mechanisms of probe-target recognition is essential for the engineered development of such probes. In this study, it is discovered that the molecular lipid solubility parameter, Clog P, determines the staining performance of fluorescent dyes on lipid droplets (LDs). Fluorescent dyes with Clog P values between 2.5 and 4 can form buffering pools outside LDs, replacing photobleached dyes within LDs to maintain constant fluorescence intensity in LDs, thereby enabling dynamic super-resolution imaging of LDs. Guided by Clog P, four different colored buffering LD probes spanning the visible light spectrum have been developed. Using Structured Illumination Microscopy (SIM), the role of LD dynamics have been tracked during cellular ferroptosis with the secretion, storage, and degradation of overexpressed ACSL3 proteins. It is found that LDs serve as storage sites for these proteins through membrane fusion, and further degrade overexpressed proteins via interactions with organelles like lysosomes or through lipophagy, thereby maintaining cellular homeostasis.

Adaptive-modulated fast fluctuation super-resolution microscopy

Fluorescence microscopy has significantly advanced biological imaging at the nanoscale, particularly with the advent of super-resolution microscopy (SRM), which transcends the Abbe diffraction limit. Most cutting-edge SR methods require high-precision optical setups, which constrain the widespread adoption of SRM. Fluorescence fluctuation-based SRM (FF-SRM) can break the diffraction limit without complex optical components, making it particularly well-suited for biological imaging. However, conventional FF-SRM methods, such as super-resolution optical fluctuation imaging (SOFI), still require specific fluorescent molecular blinking properties. Instead of enhancing the intrinsic blinking characteristics by finding specific fluorescent markers, employing optical methods such as spatial light modulation to adjust the excitation light field allows for easier and more flexible matching of the on-time ratio with the analysis of temporal stochastic intensity fluctuations. Nevertheless, the specific parameters of the modulation patterns have not been thoroughly explored, despite their crucial influence on the reconstruction quality. Herein, we propose adaptive-modulated fast fluctuation super-resolution microscopy. Our method demonstrates theoretically and experimentally that restricting the size of modulation units in a certain range ensures better image quality with fewer artifacts and signal losses. We find it still significantly effective when applied to other FF-SRM. Overall, the further development of the adaptive modulation technique has made it more stable in behavior and maintained high-quality imaging, presenting broader prospects for super resolution imaging based on statistical analysis.

Addressable scanning multifocal structured illumination microscopy using acousto-optic deflectors.

Multifocal structured illumination microscopy (MSIM) is a popular super-resolution imaging technique known for its good probe compatibility, low laser power requirements, and improved imaging depth, making it widely applicable in biomedical research. However, the speed of MSIM imaging is typically constrained by the approaches employed to generate and scan the laser foci across the sample. In this study, we propose a flexible two-photon excitation MSIM method using a pair of acousto-optic deflectors. By adopting addressable scanning (AS) and synchronized capturing, MSIM super-resolution imaging can be performed in multiple discrete regions of interest (ROIs) within the field of view. Notably, this AS-MSIM scheme not only enhances the speed of MSIM imaging but also alleviates photobleaching and phototoxicity to biological samples. We demonstrate its potential by achieving super-resolution imaging of selected mitochondria within cells at a frame rate of 4 Hz. Furthermore, we deliberate the possibility of even faster imaging.

Personalized Super Resolution with Face Prior

Face Super-Resolution (FSR) is a critical technology in computer vision that aims to reconstruct high-resolution facial images from low-resolution inputs. Despite recent advancements, current FSR methods struggle to accurately reconstruct personalized and detailed features. This paper proposes a novel FSR approach that addresses these challenges through a personalized feature extraction and fusion framework. Our method integrates a U-Net based downsampling mechanism to extract individual- specific features from high-resolution reference images, which are then fused with a pre-trained Generative Adversarial Network (GAN) for enhanced reconstruction. We introduce a comprehensive loss function that combines reconstruction, adversarial, facial component, and identity preservation losses to guide the learning process. Extensive experiments on the augmented FFHQ dataset demonstrate that our approach significantly improves the reconstruction of rich facial features, particularly for older individuals, outperforming existing state-of-the-art methods in both quantitative metrics and qualitative visual assessments.