Super-resolution Research Articles

Clickable HaloTag ligands for live cell labeling and imaging

Self-labeling protein (SLP) tags, such as HaloTag, have gained considerable interest as advanced tools for live cell labeling. However, the chloroalkane-based substrates that can be directly used for protein labeling are limited. Here, we report two bioorthogonal small molecule linkers, chloroalkane-tetrazine (CA-Tz) and chloroalkane-azide (CA-N3), which can penetrate cell membranes and facilitate click chemistry-based labeling in live cells. We compare their labeling capability using two clickable silicon rhodamine dyes (SiR-PEG3-TCO and SiR-PEG4-DBCO). Confocal imaging results demonstrate that using CA-Tz and SiR-PEG3-TCO dye exhibits superior intracellular labeling with low nonspecific signals. We subsequently compared the photostability of SiR dyes with that of green fluorescent proteins (mEmerald). Total internal reflection fluorescence (TIRF) imaging indicates that SiR dyes exhibit superior photostability under identical excitation conditions, making them suitable for long-term cell imaging. Furthermore, SiR dyes labeling also shows high structure retention for the fourth-order super-resolution optical fluctuation imaging (SOFI) compared to fluorescent proteins. This study presents clickable HaloTag linkers as effective tools for live cell labeling and imaging, highlighting the high-quality labeling of chloroalkane linkers and clickable dyes for live cell imaging.

Airborne Multi-Channel Forward-Looking Radar Super-Resolution Imaging Using Improved Fast Iterative Interpolated Beamforming Algorithm

Radar forward-looking imaging is critical in many civil and military fields, such as aircraft landing, autonomous driving, and geological exploration. Although the super-resolution forward-looking imaging algorithm based on spectral estimation has the potential to discriminate multiple targets within the same beam, the estimation of the angle and magnitude of the targets are not accurate due to the influence of sidelobe leakage. This paper proposes a multi-channel super-resolution forward-looking imaging algorithm based on the improved Fast Iterative Interpolated Beamforming (FIIB) algorithm to solve the problem. First, the number of targets and the coarse estimates of angle and magnitude are obtained from the iterative adaptive approach (IAA). Then, the accurate estimates of angle and magnitude are achieved by the strategy of iterative interpolation and leakage subtraction in FIIB. Finally, a high-resolution forward-looking image is obtained through non-coherent accumulation. The simulation results of point targets and scenes show that the proposed algorithm can distinguish multiple targets in the same beam, effectively improve the azimuthal resolution of forward-looking imaging, and attain the accurate reconstruction of point targets and the contour reconstruction of extended targets.

Long-Axial-Range Double-Helix Point Spread Functions for 3D Volumetric Super-Resolution Imaging.

Single-molecule localization microscopy (SMLM) is a powerful tool for observing structures beyond the diffraction limit of light. Combining SMLM with engineered point spread functions (PSFs) enables 3D imaging over an extended axial range, as has been demonstrated for super-resolution imaging of various cellular structures. However, super-resolving structures in 3D in thick samples, such as whole mammalian cells, remains challenging as it typically requires acquisition and postprocessing stitching of multiple slices to cover the entire sample volume or more complex analysis of the data. Here, we demonstrate how the imaging and analysis workflows can be simplified by 3D single-molecule super-resolution imaging with long-axial-range double-helix (DH)-PSFs. First, we experimentally benchmark the localization precisions of short- and long-axial-range DH-PSFs at different signal-to-background ratios by imaging fluorescent beads. The performance of the DH-PSFs in terms of achievable resolution and imaging speed was then quantified for 3D single-molecule super-resolution imaging of mammalian cells by DNA-PAINT imaging of nuclear lamina protein lamin B1 in U-2 OS cells. Furthermore, we demonstrate how the use of a deep-learning-based algorithm allows the localization of dense emitters, drastically improving the achievable imaging speed and resolution. Our data demonstrate that using long-axial-range DH-PSFs offers stitching-free, 3D super-resolution imaging of whole mammalian cells, simplifying the experimental and analysis procedures for obtaining volumetric nanoscale structural information.

GAN-Based Dual Image Super Resolution for Satellite Imagery Decreasing Radiometric Uncertainty

Abstract. Super resolution for satellite imagery possesses specific challenges due to its unique feature geometry compared to classic computer vision. Specifically, satellite images contain a high amount of relatively small, distributed high-frequency features that are hard to preserve when sampling up to a higher resolution. General adversarial networks (GANs) are suitable for super resolution tasks but need special attention concerning the geometric and radiometric accuracy of the results. We propose a network for versatile satellite imagery super resolution (VSISR) that focuses on high-frequency detail preservation and radiometric consistency. As a novelty, it is able to utilize a reference high resolution image during inference to lower hallucinations without altering the source image’s radiometric characteristics.A GAN that is already handling well high frequencies in standard computer vision cases is adapted for satellite imagery and supplemented with a mixed pixel approach for data augmentation. Training on a diverse RGB dataset from four satellite missions results in a versatile super resolution model that is optimized to preserve radiometric features and minimize hallucinations. Compared to other neural networks for satellite image super resolution in their respective datasets, VSISR performs in the middle regarding mean PSNR (25.30 dB) and SSIM (0.8098). Nevertheless, it is the first time that, using mixed pixel training on a comparably small dataset, this versatility concerning the satellite data source is achieved while maintaining their unique radiometric traits.

Submicrometre spatiotemporal characterization of the Toxoplasma adhesion strategy for gliding motility.

Toxoplasma gondii is a protozoan apicomplexan parasite that uses an adhesion-dependent mode of motility termed gliding to access host cells and disseminate into tissues. Previous studies on Apicomplexa motile morphotypes, including the T. gondii tachyzoite, have identified a cortical actin-myosin motor system that drives the rearward translocation of transmembrane adhesins, thus powering forward movement. However, this model is currently questioned. Here, combining micropatterning and tunable surface chemistry (to edit parasite surface ligands) with flow force and live or super-resolution imaging, we show that tachyzoites build only one apical anchoring contact with the substrate, over which it slides. Furthermore, we show that glycosaminoglycan-parasite interactions are sufficient to promote such force-productive contact and find that the apicobasal flow is set up independent of adhesin release and surface interactions. These findings should enable further characterization of the molecular functions at the T. gondii-substrate mechanosensitive interface and their comparison across apicomplexans.

Piezo2 interacts with E-cadherin in specialized gastrointestinal epithelial mechanoreceptors.

Piezo2 is a mechanically gated ion channel most commonly expressed by specialized mechanoreceptors, such as the enteroendocrine cells (EECs) of the gastrointestinal epithelium. A subpopulation of EECs expresses Piezo2 and functionally resembles the skin's touch sensors, called Merkel cells. Low-magnitude mechanical stimuli delivered to the mucosal layer are primarily sensed by mechanosensitive EECs in a process we term "gut touch." Piezo2 transduces cellular forces into ionic currents, a process that is sensitive to bilayer tension and cytoskeletal depolymerization. E-cadherin is a widely expressed protein that mediates cell-cell adhesion in epithelia and interacts with scaffold proteins that anchor it to actin fibers. E-cadherin was shown to interact with Piezo2 in immortalized cell models. We hypothesized that the Piezo2-E-cadherin interaction is important for EEC mechanosensitivity. To test this, we used super-resolution imaging, co-immunoprecipitation, and functional assays in primary tissues from mice and gut organoids. In tissue EECs and intestinal organoids, we observed multiple Piezo2 cellular pools, including one that overlaps with actin and E-cadherin at the cells' lateral walls. Further, E-cadherin co-immunoprecipitated with Piezo2 in the primary colonic epithelium. We found that E-cadherin knockdown decreases mechanosensitive calcium responses in mechanically stimulated primary EECs. In all, our results demonstrate that Piezo2 localizes to the lateral wall of EECs, where it physically interacts with E-cadherin and actin. They suggest that the Piezo2-E-cadherin-actin interaction is important for mechanosensitivity in the gut epithelium and possibly in tissues where E-cadherin and Piezo2 are co-expressed in epithelial mechanoreceptors, such as skin, lung, and bladder.

Learning super-resolution and pyramidal convolution residual network for vehicle re-identification

Vehicle re-identification (Vehicle Re-ID) aims at retrieving and tracking the specified target vehicle with multiple other cameras, which can provide help in checking violations and catching fugitives, but there are still the following problems that need to be solved urgently. First, the existing collected Vehicle Re-ID data often have low resolution and blur in local regions, so that the Vehicle Re-ID algorithm cannot accurately extract subtle feature representations. In addition, small features are easy to cause the disappearance of features under the operation of a large convolution kernel, which makes the model unable to capture and learn subtle features, resulting in inaccurate judgment of vehicles. In this study, we propose a Vehicle Re-ID method based on super resolution and pyramidal convolution residual network. Firstly, a super-resolution image generation network leveraging generative adversarial networks (GANs) is proposed. This network employs both content loss and adversarial loss as optimization criteria, ensuring an efficient transformation from a low-resolution image into a super-resolution counterpart, while meticulously preserving intricate high-frequency details. Then, multi levels of pyramidal convolution operations are designed to generate multi-scale features, which can capture information on different scales. Moreover, the concept of residual learning is applied between the multi levels of pyramidal convolution operations to expedite model optimization and enhance recognition capabilities. Ultimately, the double pyramidal convolutions are meticulously employed on both the original image and the super-resolution image, yielding low-noise feature representations and intricate semantic information respectively. By seamlessly fusing these two diverse sources of information, the resultant combined features exhibit heightened discrimination capabilities and significantly bolster the robustness of image features. In order to verify the effectiveness of the proposed method, extensive experiments are carried out on VeRi-776 and VehicleID datasets. The experimental results show that the method proposed in this paper effectively captures the detail information of vehicle images, accurately distinguishes the subtle differences between different vehicles of the same type, and is superior to state-of-the-art methods.

Super resolution reconstruction of fluorescence microscopy images by a convolutional network with physical priors

Super-solution fluorescence microscopy, such as single-molecule localization microscopy (SMLM), is effective in observing subcellular structures and achieving excellent enhancement in spatial resolution in contrast to traditional fluorescence microscopy. Recently, deep learning has demonstrated excellent performance in SMLM in solving the trade-offs between spatiotemporal resolution, phototoxicity, and signal intensity. However, most of these researches rely on sufficient and high-quality datasets. Here, we propose a physical priors-based convolutional super-resolution network (PCSR), which incorporates a physical-based loss term and an initial optimization process based on the Wiener filter to create excellent super-resolution images directly using low-resolution images. The experimental results demonstrate that PCSR enables the achievement of a fast reconstruction time of 100 ms and a high spatial resolution of 10 nm by training on a limited dataset, allowing subcellular research with high spatiotemporal resolution, low cell phototoxic illumination, and high accessibility. In addition, the generalizability of PCSR to different live cell structures makes it a practical instrument for diverse cell research.

Disentangled feature fusion network for lightweight image super-resolution

Recently, the quality of generated images in image super-resolution (SR) has significantly improved due to the widespread application of convolutional neural networks. Existing super-resolution methods often overlook the importance of network width and instead prioritize designing deeper network structures to improve performance. Deeper networks, however, demand greater computational resources and are more challenging to train, making them less suitable for devices with limited performance. To address this issue, we propose a novel method called the disentangled feature fusion network (DFFN). Specifically, we construct dual feature extraction streams (DFES) using the strategies of feature disentanglement and parameter sharing, which increase the network's width without increasing the number of parameters. To facilitate the interaction of information between the dual feature extraction streams, we design a feature interaction module (FIM) that separately enhances high and low-frequency features. Furthermore, a feature fusion module (FFM) is presented to efficiently fuse different levels of high and low-frequency feature information. The proposed DFFN integrates DFES, FIM, and FFM, which not only widens the network without increasing parameters but also minimizes the loss of crucial feature information. Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms other state-of-the-art lightweight super-resolution methods, achieving a superior balance between parameter quantity, reconstruction performance, and model complexity. The code is available at https://github.com/zhoujy-aust/DFFN.

Deep Variational Network Toward Blind Image Restoration.

Blind image restoration (IR) is a common yet challenging problem in computer vision. Classical model-based methods and recent deep learning (DL)-based methods represent two different methodologies for this problem, each with their own merits and drawbacks. In this paper, we propose a novel blind image restoration method, aiming to integrate both the advantages of them. Specifically, we construct a general Bayesian generative model for the blind IR, which explicitly depicts the degradation process. In this proposed model, a pixel-wise non-i.i.d. Gaussian distribution is employed to fit the image noise. It is with more flexibility than the simple i.i.d. Gaussian or Laplacian distributions as adopted in most of conventional methods, so as to handle more complicated noise types contained in the image degradation. To solve the model, we design a variational inference algorithm where all the expected posteriori distributions are parameterized as deep neural networks to increase their model capability. Notably, such an inference algorithm induces a unified framework to jointly deal with the tasks of degradation estimation and image restoration. Further, the degradation information estimated in the former task is utilized to guide the latter IR process. Experiments on two typical blind IR tasks, namely image denoising and super-resolution, demonstrate that the proposed method achieves superior performance over current state-of-the-arts.

Novel Intercellular Spread Mode of Respiratory Syncytial Virus Contributes to Neutralization Escape.

Developing widely used respiratory syncytial virus (RSV) vaccines remains a significant challenge, despite the recent authorization of two pre-F vaccines for elderly adults. Previous reports have suggested that even when vaccine-induced immunity generates high titers of potent neutralizing antibodies targeting the pre-F protein, it may not fully inhibit breakthrough of RSV infections. This incomplete inhibition of RSV breakthrough infections can lead to an increased risk of enhanced respiratory disease (ERD) in vaccinated individuals. The reasons why potent neutralizing antibodies cannot fully prevent RSV breakthrough infections are not yet clear. In an attempt to explain this phenomenon, we investigated the effect of potent neutralizing antibodies on the intercellular spread of RSV. Our findings indicated that a specific titer of potent neutralizing antibodies, such as 5C4, could block certain modes of intercellular spread, such as the diffusion of cell-free virions and the delivery of virions through filopodia. However, these antibodies did not fully inhibit the entire process of intercellular spread. Through the use of super-resolution imaging techniques, we observed a novel and efficient spread mode called the transition of viral materials through intercellular nanotubes (TVMIN), independent of virions and insensitive to the presence of antibodies. TVMIN allowed RSV-infected cells to directly transfer viral materials to neighboring cells via intercellular nanotubes that are rich in microfilaments. TVMIN began as early as 5 hours post-infection (h.p.i.) and rapidly initiated infection in recipient cells. Our data provided new insights into the intercellular spread of RSV and might help explain the occurrence of breakthrough infections.

Representative Image Outpainting and Image Super-Resolution Methods Based on Deep Learning

The image generation based on deep learning is a technology that can generate new images or outpaint old images or improve visual effect of old images through learning from input data, according to deep learning structure. The representative technologies of image generation are image outpainting and image super-resolution. Deep learning is widely utilized in the field of computer vision. Facing different needs, it is essential to choose proper ways. This essay reviews several representative and new methods of image outpainting and image super-resolution. Compared with the results generated by old methods, the results generated by new image outpainting methods introduced in this essay have greater precision and clarity, the methods of image super-resolution that are suitable for all fields and professional fields can learn from each others dataset manually to train and develop. There is still unimaginable prospect of deep learning in the field of image outpainting and image super-resolution. Therefore, it is worthy of attention.

Simultaneous single image super‐resolution and blind Gaussian denoising via slim ghost full‐frequency residual blocks

AbstractGiven that super‐resolution (SR) aims to recover lost information, and low‐resolution (LR) images in real‐world conditions might be corrupted with multiple degradations, considering basic bicubic down‐sampling as the sole degradation significantly limits the performance of most existing SR models. This paper presents a model for simultaneous super‐resolution and blind additive white Gaussian noise (AWGN) denoising with two components (netdeg and netSR) that is based on a generative adversarial network (GAN) to achieve detailed results. netdeg, featuring residual and innovative cost‐effective ghost residual blocks with a frequency separation module for obtaining long‐range information, blindly restores a clean version of the LR image. netSR leverages slim ghost full‐frequency residual blocks to process low‐frequency (LF) and high‐frequency (HF) information via static large convolutions and pixel‐wise highlighted input‐adaptive dynamic convolutions, respectively. To address the susceptibility of dynamic layers to noise and preserve feature diversity while reducing model’s costs, static and dynamic layer features are combined and highlighted. Diverse and non‐redundant features are then processed using ghost‐style blocks. The proposed model achieves comparable SR results in bicubic down‐sampling scenarios, outperform existing SR methods in the complex task of concurrent SR and AWGN denoising, and demonstrate robustness in handling images corrupted with varying levels of AWGN.

Efficient Super-resolution of Phase Images Encoded with Random Phase Mask by machine learning techniques

Efficient Super-resolution of Phase Images Encoded with Random Phase Mask by machine learning techniques

Super-resolution imaging quality enhancement method for distributed array infrared camera

Abstract To address issues related to low resolution and high noise in infrared cameras, a distributed array infrared camera imaging system utilizing four cameras is proposed. The four cameras are arranged in an unconstrained array, and the combination algorithm of Projections onto Convex Sets (POCS) and Real-Enhanced Super-Resolution Generative Adversarial Networks (Real-ESRGAN) is applied to achieve high-quality super-resolution infrared imaging. The wavelet fusion algorithm is used to preprocess four low-resolution infrared images to reduce noise. Then, the POCS algorithm is used to reconstruct the preprocessed image. Finally, the Real-ESRGAN is employed for image reconstruction, resulting in an ultra-high-resolution infrared image. The results show that compared to single infrared camera imaging, the resolution of images reconstructed using the distributed infrared camera array is increased by 0.58 times, with the Modulation Transfer Function (MTF) increased by 1.2 times. Additionally, the entropy is increased by 18.87%, the standard deviation is increased by 13.51%, and the Naturalness Image Quality Evaluator (NIQE) is reduced by 18.87%. This demonstrates a significant enhancement in the super-resolution imaging quality of the distributed infrared camera array.

High-Fidelity and High-Efficiency Talking Portrait Synthesis With Detail-Aware Neural Radiance Fields.

In this paper, we propose a novel rendering framework based on neural radiance fields (NeRF) named HH-NeRF that can generate high-resolution audio-driven talking portrait videos with high fidelity and fast rendering. Specifically, our framework includes a detail-aware NeRF module and an efficient conditional super-resolution module. Firstly, a detail-aware NeRF is proposed to efficiently generate a high-fidelity low-resolution talking head, by using the encoded volume density estimation and audio-eye-aware color calculation. This module can capture natural eye blinks and high-frequency details, and maintain a similar rendering time as previous fast methods. Secondly, we present an efficient conditional super-resolution module on the dynamic scene to directly generate the high-resolution portrait with our low-resolution head. Incorporated with the prior information, such as depth map and audio features, our new proposed efficient conditional super resolution module can adopt a lightweight network to efficiently generate realistic and distinct high-resolution videos. Extensive experiments demonstrate that our method can generate more distinct and fidelity talking portraits on high resolution (900 × 900) videos compared to state-of-the-art methods. Our code is available at https://github.com/muyuWang/HHNeRF.

Comparative analysis of accelerated gradient algorithms for convex optimization: high and super resolution ODE approach

We investigate convex differentiable optimization and explore the temporal discretization of damped inertial dynamics driven by the gradient of the objective function. This leads to three accelerated gradient algorithms: Nesterov Accelerated Gradient (NAG), Ravine Accelerated Gradient (RAG), and (IGAHD). The latter was introduced by discretizing inertial dynamics with Hessian-driven damping to attenuate inherent oscillations in inertial methods. By analysing the high-resolution ODEs of order p = 0,1,2 for these algorithms, we gain insights into their similarities and differences. All three algorithms share the same low-resolution ODE of order 0, which is the dynamic proposed as a continuous surrogate for (NAG). To differentiate Nesterov from Ravine, we refine the comparison and demonstrate distinct high-resolution ODEs of order 2 in h (termed super-resolution). The corresponding Taylor expansions in h reveal matching terms of order 1 but differing terms of order 2. To the best of our knowledge, this result is completely new and emphasizes the need to avoid confusion between the Ravine and Nesterov methods in the literature. We present numerical experiments to illustrate our theoretical results. Performance profiles, measuring the number of iterations, indicate that (IGAHD) outperforms both (NAG) and (RAG) methods. (RAG) exhibits a slight advantage over (NAG) in terms of the average number of iterations. When considering CPU-time, both (RAG) and (NAG) outperform (IGAHD). All three algorithms exhibit similar behaviour when evaluating based on gradient norms.

A Feature-Driven Inception Dilated Network for Infrared Image Super-Resolution Reconstruction

Image super-resolution (SR) algorithms based on deep learning yield good visual performances on visible images. Due to the blurred edges and low contrast of infrared (IR) images, methods transferred directly from visible images to IR images have a poor performance and ignore the demands of downstream detection tasks. Therefore, an Inception Dilated Super-Resolution (IDSR) network with multiple branches is proposed. A dilated convolutional branch captures high-frequency information to reconstruct edge details, while a non-local operation branch captures long-range dependencies between any two positions to maintain the global structure. Furthermore, deformable convolution is utilized to fuse features extracted from different branches, enabling adaptation to targets of various shapes. To enhance the detection performance of low-resolution (LR) images, we crop the images into patches based on target labels before feeding them to the network. This allows the network to focus on learning the reconstruction of the target areas only, reducing the interference of background areas in the target areas’ reconstruction. Additionally, a feature-driven module is cascaded at the end of the IDSR network to guide the high-resolution (HR) image reconstruction with feature prior information from a detection backbone. This method has been tested on the FLIR Thermal Dataset and the M3FD Dataset and compared with five mainstream SR algorithms. The final results demonstrate that our method effectively maintains image texture details. More importantly, our method achieves 80.55% mAP, outperforming other methods on FLIR Dataset detection accuracy, and with 74.7% mAP outperforms other methods on M3FD Dataset detection accuracy.

Clog P-Guided Development of Multi-Colored Buffering Fluorescent Probes for Super-Resolution Imaging of Lipid Droplet Dynamics.

Super-resolution fluorescence imaging of live cells increasingly demands fluorescent probes capable of multi-color and long-term dynamic imaging. Understanding the mechanisms of probe-target recognition is essential for the engineered development of such probes. In this study, it is discovered that the molecular lipid solubility parameter, Clog P, determines the staining performance of fluorescent dyes on lipid droplets (LDs). Fluorescent dyes with Clog P values between 2.5 and 4 can form buffering pools outside LDs, replacing photobleached dyes within LDs to maintain constant fluorescence intensity in LDs, thereby enabling dynamic super-resolution imaging of LDs. Guided by Clog P, four different colored buffering LD probes spanning the visible light spectrum have been developed. Using Structured Illumination Microscopy (SIM), the role of LD dynamics have been tracked during cellular ferroptosis with the secretion, storage, and degradation of overexpressed ACSL3 proteins. It is found that LDs serve as storage sites for these proteins through membrane fusion, and further degrade overexpressed proteins via interactions with organelles like lysosomes or through lipophagy, thereby maintaining cellular homeostasis.

Adaptive-modulated fast fluctuation super-resolution microscopy

Fluorescence microscopy has significantly advanced biological imaging at the nanoscale, particularly with the advent of super-resolution microscopy (SRM), which transcends the Abbe diffraction limit. Most cutting-edge SR methods require high-precision optical setups, which constrain the widespread adoption of SRM. Fluorescence fluctuation-based SRM (FF-SRM) can break the diffraction limit without complex optical components, making it particularly well-suited for biological imaging. However, conventional FF-SRM methods, such as super-resolution optical fluctuation imaging (SOFI), still require specific fluorescent molecular blinking properties. Instead of enhancing the intrinsic blinking characteristics by finding specific fluorescent markers, employing optical methods such as spatial light modulation to adjust the excitation light field allows for easier and more flexible matching of the on-time ratio with the analysis of temporal stochastic intensity fluctuations. Nevertheless, the specific parameters of the modulation patterns have not been thoroughly explored, despite their crucial influence on the reconstruction quality. Herein, we propose adaptive-modulated fast fluctuation super-resolution microscopy. Our method demonstrates theoretically and experimentally that restricting the size of modulation units in a certain range ensures better image quality with fewer artifacts and signal losses. We find it still significantly effective when applied to other FF-SRM. Overall, the further development of the adaptive modulation technique has made it more stable in behavior and maintained high-quality imaging, presenting broader prospects for super resolution imaging based on statistical analysis.