Super-resolution Research Articles

Localization of albumin with correlative super resolution light- and electron microscopy in the kidney

The functioning of vertebrate life relies on renal filtration of surplus fluid and elimination of low-molecular-weight waste products, while keeping serum proteins in the blood. In disease, however, there is leak of serum proteins and tracing them to identify the leaking position within tissue with a nanometer resolution poses a significant challenge. Correlative microscopy integrates the specificity of fluorescent protein labeling into high-resolution electron micrographs. Using chemical tagging of albumin with synthetic fluorophores we achieve protein-specific labeling that preserve their post-embedding fluorescence after high-pressure freezing and freeze-substitution of murine kidney tissue. Using advanced registration techniques for super-resolution correlative light and electron microscopy, we can localize the labeled albumin with a high precision in the x-y plane of electron micrographs and cartograph its distribution. Thereby we can quantify the albumin concentration and measure a linear reduction gradient across the kidney filtration barrier. Our study shows the feasibility of combining different microscopy contrasts for tracing fluorescently labeled protein markers with super resolution in various tissue samples and opens new perspectives for correlative imaging in volume electron microscopy.

An intramolecular charge transfer based fluorescent probe for imaging of OCl–

The discovery and utilization of new fluorescent chromophore is indispensable to exploit high performance probes for biological research. Stokes shift is one of the most important properties of chromophore accounting for super-resolution fluorescence imaging. Intramolecular charge transfer (ICT) is one of the fundamental mechanisms for fluorescence that accompanied by large Stokes shifts. Based on the conformational changes between ground and excited states, ICT models can be divided into two types: conformation-steady ICT, whose conformation remains unchanged, and conformation-changeable ICT, which is characterized by the rotation of the chromophore around an axis upon excitation. Herein, we report a new chromophore whose donor and acceptor parts took a butterfly geometry with a dihedral angle of 21° in ground state and a planar conformation upon photo excitation. The bent conformation might be ascribed to the extra conjugated double bond, which made the coplanarity of the chromophore in ground state get worse. The chromophore shows a remarkable Stokes shift over 150 nm and a high fluorescence quantum yieldof 0.62. The limit of detection is 41 nM, which enabled the imaging of basal as well as induced OCl– in different cells. Moreover, the pronounced spectroscopic properties ensure the in vivo monitoring of OCl– in arthritic mice. This finding would shed light on the exploitation of small molecule probes based on new fluorescence chromophore for precise biological imaging.

SeBIR: Semantic-guided burst image restoration

Burst image restoration methods offer the possibility of recovering faithful scene details from multiple low-quality snapshots captured by hand-held devices in adverse scenarios, thereby attracting increasing attention in recent years. However, individual frames in a burst typically suffer from inter-frame misalignments, leading to ghosting artifacts. Besides, existing methods indiscriminately handle all burst frames, struggling to seamlessly remove the corrupted information due to the neglect of multi-frame spatio-temporal varying degradation. To alleviate these limitations, we propose a general semantic-guided model named SeBIR for burst image restoration incorporating the semantic prior knowledge of Segment Anything Model (SAM) to enable adaptive recovery. Specifically, instead of relying solely on a single aligning scheme, we develop a joint implicit and explicit strategy that sufficiently leverages semantic knowledge as guidance to achieve inter-frame alignment. To further adaptively modulate and aggregate aligned features with spatio-temporal disparity, we elaborate a semantic-guided fusion module using the intermediate semantic features of SAM as an explicit guide to weaken the inherent degradation and strengthen the valuable complementary information across frames. Additionally, a semantic-guided local loss is designed to boost local consistency and image quality. Extensive experiments on synthetic and real-world datasets demonstrate the superiority of our method in both quantitative and qualitative evaluations for burst super-resolution, burst denoising, and burst low-light image enhancement tasks.

Super-resolution electromagnetic imaging with an optical frequency comb sampling via multi-dimensional signal separation

Electromagnetic sources show wide distribution, broad frequency coverage, and numerous quantities, posing challenges for traditional sensing techniques to achieve ultra-wideband, large-scale detection and localization. The “electromagnetic eye” imaging technique, inspired by the human eye, utilizes a Luneberg lens and a wideband optoelectronic sensing array as the electromagnetic “lens” and “retina,” respectively. This technique utilizes femtosecond optical pulse sampling reception to down-convert wideband signals, facilitating rapid, large range, and wideband sensing of multiple targets in complex electromagnetic environments. However, the limited aperture of the Luneberg lens results in diffraction-limited blurring, and optical down-conversion may lead to spectral aliasing, causing time-frequency-space overlap and reduced system resolution. In this paper, the frequency variation of the point spread function (PSF) in the wideband degraded images is analyzed, and a multi-dimensional joint super-resolution algorithm is proposed, which involves joint time-frequency-space diagonalization of eigenmatrices based on convolutional mixing array model. The concept is demonstrated through a four-sources imaging simulation achieving 2° resolution, breaking the Rayleigh limit 7.25 times. Furthermore, experimental results show 4-10 GHz imaging breaks the Rayleigh limit 4.5 times.

Identification of Novel Therapeutic Targets for Alport Syndrome Using Near Super-Resolution Imaging of Podocytes

Identification of Novel Therapeutic Targets for Alport Syndrome Using Near Super-Resolution Imaging of Podocytes

Super-Resolution Surface-Enhanced Raman Scattering: Perspectives on the Past, Present, and Future.

Super-resolution surface-enhanced Raman scattering (SERS) allows researchers to overcome the resolution limit of far field optical microscopy and peer into electromagnetic hot spots with nanoscale resolution. By localizing the signal from single (or few) molecules on the surface of plasmonic nanoparticle aggregates, relationships between the spatial origin of the SERS signal, local electromagnetic field enhancements, and SERS intensity can be determined. This Perspective describes the successes and challenges of super-resolution SERS, from the earliest mapping of single-molecule SERS hot spots to the current state-of-the-art, while highlighting open questions and future opportunities to advance the field. Comparisons with fluorescence-based super-resolution imaging are discussed to help frame the unique challenges associated with performing SERS in the super-resolution regime.

Application of improved deep learning algorithm in obstacle avoidance of UAV

Abstract To enhance the intelligent obstacle avoidance capabilities of unmanned aerial vehicles (UAVs), we introduce a refined version of the You Only Look Once (YOLO) algorithm, termed Improved YOLO (I-YOLO). This novel algorithm not only maintains the swift detection speed inherent in the original YOLO but also augments its feature enhancement network. Specifically, we propose a three-branch parallel feature pyramid network (TBPFP) to capture richer semantic information pertaining to the context of small obstacles. Furthermore, we incorporate a generative adversarial network (GAN) in a cascaded manner to synthesize more realistic super-resolution images, thereby bolstering detection accuracy. Experimental evaluations demonstrate that, in comparison to the baseline YOLO algorithm, our I-YOLO variant exhibits superior detection precision and feature extraction capabilities in the perception of intricate environmental targets. While there is a modest decrement in detection speed, it remains adequate to fulfill real-time requirements, particularly in the context of video stream processing.

Rotation Equivariant Proximal Operator for Deep Unfolding Methods in Image Restoration.

The deep unfolding approach has attracted significant attention in computer vision tasks, which well connects conventional image processing modeling manners with more recent deep learning techniques. Specifically, by establishing a direct correspondence between algorithm operators at each implementation step and network modules within each layer, one can rationally construct an almost "white box" network architecture with high interpretability. In this architecture, only the predefined component of the proximal operator, known as a proximal network, needs manual configuration, enabling the network to automatically extract intrinsic image priors in a data-driven manner. In current deep unfolding methods, such a proximal network is generally designed as a CNN architecture, whose necessity has been proven by a recent theory. That is, CNN structure substantially delivers the translational symmetry image prior, which is the most universally possessed structural prior across various types of images. However, standard CNN-based proximal networks have essential limitations in capturing the rotation symmetry prior, another universal structural prior underlying general images. This leaves a large room for further performance improvement in deep unfolding approaches. To address this issue, this study makes efforts to suggest a high-accuracy rotation equivariant proximal network that effectively embeds rotation symmetry priors into the deep unfolding framework. Especially, we deduce, for the first time, the theoretical equivariant error for such a designed proximal network with arbitrary layers under arbitrary rotation degrees. This analysis should be the most refined theoretical conclusion for such error evaluation to date and is also indispensable for supporting the rationale behind such networks with intrinsic interpretability requirements. Through experimental validation on different vision tasks, including blind image super-resolution, medical image reconstruction, and image de-raining, the proposed method is validated to be capable of directly replacing the proximal network in current deep unfolding architecture and readily enhancing their state-of-the-art performance. This indicates its potential usability in general vision tasks.

Customizing subwavelength stochastic light fields via structured spatial coherence in a high-NA resonator

Interaction of light in the high-numerical aperture (high-NA) systems is crucial for theoretical advances and applications such as superresolution imaging and optical nanofabrication. High coherence is demanded at this scale for intensity and spin profile sculpturing, since the underlying physics being wave interference. Here we report that, even for low-coherence light, 3D light features in a nanometer range can be generated by employing structured coherence states of the light beam in a high-NA resonator system. The generated structures, e.g., 3D helix intensity and transverse spin texture, can survive in rather incoherent optical fields at nanoscale. We also found that, counterintuitively, despite the substantial decrease in spatial coherence of the light field, the longitudinal electric field component and transverse spin density are instead enhanced. The applications of the observed twisted spectral density and spin structures may range from high-resolution imaging to metrology.

Multi-contrast image super-resolution with deformable attention and neighborhood-based feature aggregation (DANCE): Applications in anatomic and metabolic MRI

Multi-contrast magnetic resonance imaging (MRI) reflects information about human tissues from different perspectives and has wide clinical applications. By utilizing the auxiliary information from reference images (Refs) in the easy-to-obtain modality, multi-contrast MRI super-resolution (SR) methods can synthesize high-resolution (HR) images from their low-resolution (LR) counterparts in the hard-to-obtain modality. In this study, we systematically discussed the potential impacts caused by cross-modal misalignments between LRs and Refs and, based on this discussion, proposed a novel deep-learning-based method with Deformable Attention and Neighborhood-based feature aggregation to be Computationally Efficient (DANCE) and insensitive to misalignments. Our method has been evaluated in two public MRI datasets, i.e., IXI and FastMRI, and an in-house MR metabolic imaging dataset with amide proton transfer weighted (APTW) images. Experimental results reveal that our method consistently outperforms baselines in various scenarios, with significant superiority observed in the misaligned group of IXI dataset and the prospective study of the clinical dataset. The robustness study proves that our method is insensitive to misalignments, maintaining an average PSNR of 30.67 dB when faced with a maximum range of ±9°and ±9 pixels of rotation and translation on Refs. Given our method’s desirable comprehensive performance, good robustness, and moderate computational complexity, it possesses substantial potential for clinical applications.

A super-resolution algorithm of Ghost Imaging using CNN with Grouped orthonormalization algorithm Constraint

Ghost imaging (GI) is capable of reconstructing images under low-light conditions by single-pixel measurements. However, improving image resolution often requires extensive single-pixel sampling, limiting practical applications. Here we propose a super-resolution algorithm of GI using Convolutional neural network with Grouped orthonormalization algorithm Constraint (GICGC), which aims to reconstruct images at super-resolution with strong local regularities and self-similarity. The proposed algorithm, a versatile approach, has been demonstrated to outperform several other widely used GI algorithms in terms of spatial resolution and sampling rate. Our findings are supported by rigorous benchmark tests and experimental validations in challenging environments, including multimode fibers. The imaging experimental results demonstrate that GICGC achieves superior performance in reconstructing image linewidth and contrast, effectively doubling resolution capability by approximately 2.1 times, indicating significant potential in biomedical imaging and other fields. We believe this study represents a novel breakthrough in universal super-resolution ghost imaging and paves the way for its practical applications.

Elaborating the Fluorescence Regulation and Quenching Mechanism of Sulfur-for-Oxygen Replacement for Fluorophores.

Thio-caged fluorophores can be effectively desulfurized into their oxygenated derivatives through visible light, thereby restoring the strong emission of fluorophores, and are applied in the field of live cell super-resolution imaging. Herein, we theoretically investigated the reasons for the low fluorescence quantum yields of a series of thio-caged fluorophores and the underlying reasons for the differences in fluorescence quantum yields of their oxygenated derivatives. The calculation results show that the S atom on the thiocarbonyl group is more likely to excite n electrons to form the nπ* state, which reduces the energy of the nπ* state and leads to fluorescence quenching. In contrast, the O atom on the carbonyl group is more likely to excite π electrons to form ππ* state, which is the main reason for restoring the strong emission of fluorophore. Meanwhile, the calculation results show that the difference of fluorescence intensity caused by oxygenated derivatives is determined by the number of the carbonyl group, which affects the vibronic coupling between ππ* and nπ* states and thereby leads to fluorescence quenching. These results can effectively reveal the fluorescence quenching mechanism of thio-caged fluorophores and the luminescence mechanism of their oxygenated derivatives, and provide correct and guiding design strategies for the development of new thio-caged fluorophores.

Photonic lantern TIRF microscopy for highly efficient, uniform, artifact-free imaging

We report a method for generating uniform, artifact-free total internal reflection fluorescence (TIRF) excitation via a photonic lantern. Our tapered waveguide, consisting of a multimode input and nine few-mode outputs, enables single-shot TIRF illumination from nine azimuthal directions simultaneously without the introduction of nonstationary devices. Utilizing the photonic lantern for multi-beam excitation provides a low-loss mechanism that supports a wide range of light sources, including high-coherence lasers and various wavelengths in the visible spectrum. Our excitation system also allows tuning of the TIRF penetration depth. The high-quality excitation produced by photonic lantern TIRF (PL-TIRF) enables unbiased imaging across the entire illumination field-of-view. The simplicity and robustness of our technique provides advantages over other TIRF approaches, which often have complicated setups with scanning devices or other impracticalities. In this paper we discuss the lantern design process, characterize its performance, and demonstrate flat-field super-resolution imaging and shadowless live-cell imaging using PL-TIRF.

Performance Improvement of Rice Leaf Disease Prediction Model using Enhanced Super-Resolution Generative Adversarial Networks

The objective of this paper is to propose a method to improve the performance of Rice Leaf Disease Prediction Deep Learning Model. The primary goal is to empower farmers with precise information about crop disease, enabling them to judiciously apply pesticides suitably to their crop's needs. The method proposed here uses image enhancement using Enhanced Super Resolution Generative Adversarial Network. Experiment was carried out using Rice Leaf Disease dataset which includes 16,000 rice leaf images categorized into four groups. Among these groups, three contain 4,000 images each, showcasing three different diseases and another 4000 images of healthy leaves. The best accuracy of the model came out to be 99.68. Also all individual diseases are identified very well and other model evaluation aspects also giving good results. Average precision obtained is 99.67, average recall value=99.68 and average f1-score came out to be 99.67. Conclusion: The performance results obtained is the highest what has been achieved of all the models that have been developed till now for this task.

Efficient degradation representation learning network for remote sensing image super-resolution

The advancements in convolutional neural networks have led to significant progress in image super-resolution (SR) techniques. Nevertheless, it is crucial to acknowledge that current SR methods operate under the assumption of bicubic downsampling as a degradation factor in low-resolution (LR) images and train models accordingly. However, this approach does not account for the unknown degradation patterns present in real-world scenes. To address this problem, we propose an efficient degradation representation learning network (EDRLN). Specifically, we adopt a contrast learning approach, which enables the model to distinguish and learn various degradation representations in realistic images to obtain critical degradation information. We also introduce streamlined and efficient pixel attention to strengthen the feature extraction capability of the model. In addition, we optimize our model with mutual affine convolution layers instead of ordinary convolution layers to make it more lightweight while minimizing performance loss. Experimental results on remote sensing and benchmark datasets show that our proposed EDRLN exhibits good performance for different degradation scenarios, while the lightweight version minimizes the performance loss as much as possible. The Code will be available at: https://github.com/Leilei11111/EDRLN.

Information sparsity guided transformer for multi-modal medical image super-resolution

Multi-modal medical image super-resolution (SR) plays a vital role in enhancing the resolution of medical images, providing more detailed visuals that aid in accurate clinical diagnosis. Recently, Transformer-based super-resolution methods have significantly promoted performance improvement in this field due to their capacity to capture global dependencies. They usually process all non-overlapping patches as tokens and densely sample these tokens without screening for calculating attention mechanisms. However, this strategy ignores the spatial sparsity of medical images, resulting in redundant or even detrimental computations on less informative regions. Hence, this paper proposes a novel sparsity-guided medical image SR network, namely SG-SRNet, by exploiting the spatial sparsity characteristics of the medical images. SG-SRNet mainly consists of two components: a sparsity mask (SM) generator for image sparsity estimation, and a sparsity-guided Transformer (SGTrans) for high-resolution image reconstruction. Specially, the SM generator generates a sparsity mask by minimizing our cross-sparsity loss which can respond to the informative positions. SGTrans first screens out the informative patches according to the sparsity mask. Then, SGTrans utilizes the designed cluster-based attention to only calculate attention between information-related tokens. We perform comprehensive experiments on three datasets to show that SG-SRNet brings significant performance enhancements with low computational complexity.

Single-molecule detection and super-resolution imaging with a portable and adaptable 3D-printed microscopy platform (Brick-MIC).

Over the past decades, single-molecule and super-resolution microscopy have advanced and represent essential tools for life science research. There is, however, a growing gap between the state of the art and what is accessible to biologists, biochemists, medical researchers, or labs with financial constraints. To bridge this gap, we introduce Brick-MIC, a versatile and affordable open-source 3D-printed microspectroscopy and imaging platform. Brick-MIC enables the integration of various fluorescence imaging techniques with single-molecule resolution within a single platform and exchange between different modalities within minutes. We here present variants of Brick-MIC that facilitate single-molecule fluorescence detection, fluorescence correlation spectroscopy, time-correlated single-photon counting and super-resolution imaging (STORM and PAINT). Detailed descriptions of the hardware and software components, as well as data analysis routines, are provided, to allow non-optics specialists to operate their own Brick-MIC with minimal effort and investments. We foresee that our affordable, flexible, and open-source Brick-MIC platform will be a valuable tool for many laboratories worldwide.

A scalable attention network for lightweight image super-resolution

Modeling long-range dependencies among features has become a consensus to improve the results of single image super-resolution (SISR), which stimulates interest in enlarging the kernel sizes in convolutional neural networks (CNNs). Although larger kernels definitely improve the network performance, network parameters and computational complexities are raised sharply as well. Hence, an optimization of setting the kernel sizes is required to improve the efficiency of the network. In this work, we study the influence of the positions of larger kernels on the network performance, and propose a scalable attention network (SCAN). In SCAN, we propose a depth-related attention block (DRAB) that consists of several multi-scale information enhancement blocks (MIEBs) and resizable-kernel attention blocks (RKABs). The RKAB dynamically adjusts the kernel size concerning the locations of the DRABs in the network. The resizable mechanism allows the network to extract more informative features in shallower layers with larger kernels and focus on useful information in deeper layers with smaller ones, which effectively improves the SR results. Extensive experiments demonstrate that the proposed SCAN outperforms other state-of-the-art lightweight SR methods. Our codes are available at https://github.com/ginsengf/SCAN.

An efficient feature reuse distillation network for lightweight image super-resolution

In recent research, single-image super-resolution (SISR) using deep Convolutional Neural Networks (CNN) has seen significant advancements. While previous methods excelled at learning complex mappings between low-resolution (LR) and high-resolution (HR) images, they often required substantial computational and memory resources. We propose the Efficient Feature Reuse Distillation Network (EFRDN) to alleviate these challenges. EFRDN primarily comprises Asymmetric Convolutional Distillation Modules (ACDM), incorporating the Multiple Self-Calibrating Convolution (MSCC) units for spatial and channel feature extraction. It includes an Asymmetric Convolution Residual Block (ACRB) to enhance the skeleton information of the square convolution kernel and a Feature Fusion Lattice Block (FFLB) to convert low-order input signals into higher-order representations. Introducing a Transformer module for global features, we enhance feature reuse and gradient flow, improving model performance and efficiency. Extensive experimental results demonstrate that EFRDN outperforms existing methods in performance while conserving computing and memory resources.

Optical Nanoscopy of Cytokine-Induced Structural Alterations of the Endoplasmic Reticulum and Golgi Apparatus in Insulin-Secreting Cells.

Pro-inflammatory cytokines play a role in the failure of β cells in type 1 and type 2 diabetes. While existing data from 'omics' experiments allow for some understanding of the molecular mechanisms behind cytokine-induced dysfunction in β cells, no report thus far has provided information on the direct imaging of the β cell landscape with nanoscale resolution following cytokine exposure. In this study, we use Airyscan-based optical super-resolution microscopy of Insulinoma 1E (INS-1E) cells to investigate the structural properties of two subcellular membranous compartments involved in the production, maturation and secretion of insulin-containing granules, the endoplasmic reticulum (ER) and the Golgi apparatus (GA). Our findings reveal that exposure of INS-1E cells to IL-1β and IFN-γ for 24 h leads to significant structural alterations of both compartments. In more detail, both the ER and the GA fragment and give rise to vesicle-like structures with markedly reduced characteristic area and perimeter and increased circularity with respect to the original structures. These findings complement the molecular data collected thus far on these compartments and their role in β cell dysfunction and lay the groundwork for future optical microscopy-based ex vivo and in vivo investigations.