Super-resolution Information Research Articles

Spatial-Spectral Middle Cross-Attention Fusion Network for Hyperspectral Image Superresolution

The spatial and spectral features of hyperspectral images exhibit complementarity, and neglecting them prevents the full exploitation of useful information for superresolution. This article proposes a spatial-spectral middle cross-attention fusion network to explore the spatial-spectral structure correlation. Initially, we learn spatial and spectral features through spatial and spectral branches instead of single ones to reduce information compression. Then, a novel middle-cross attention fusion block that includes middle features fusion strategy and cross-attention is proposed to fuse spatial-spectral features to enhance their mutual effects, which aims to explore the spatial-spectral structural correlations. Finally, we propose a spectral feature compensation mechanism to provide complementary information for adjacent band groups. The experimental results show that the proposed method outperforms state-of-the-art algorithms in object values and visual quality.

Generating 10-Meter Resolution Land Use and Land Cover Products Using Historical Landsat Archive Based on Super Resolution Guided Semantic Segmentation Network

Generating high-resolution land cover maps using relatively lower-resolution remote sensing images is of great importance for subtle analysis. However, the domain gap between real lower-resolution and synthetic images has not been permanently resolved. Furthermore, super-resolution information is not fully exploited in semantic segmentation models. By solving the aforementioned issues, a deeply fused super resolution guided semantic segmentation network using 30 m Landsat images is proposed. A large-scale dataset comprising 10 m Sentinel-2, 30 m Landsat-8 images, and 10 m European Space Agency (ESA) Land Cover Product is introduced, facilitating model training and evaluation across diverse real-world scenarios. The proposed Deeply Fused Super Resolution Guided Semantic Segmentation Network (DFSRSSN) combines a Super Resolution Module (SRResNet) and a Semantic Segmentation Module (CRFFNet). SRResNet enhances spatial resolution, while CRFFNet leverages super-resolution information for finer-grained land cover classification. Experimental results demonstrate the superior performance of the proposed method in five different testing datasets, achieving 68.17–83.29% and 39.55–75.92% for overall accuracy and kappa, respectively. When compared to ResUnet with up-sampling block, increases of 2.16–34.27% and 8.32–43.97% were observed for overall accuracy and kappa, respectively. Moreover, we proposed a relative drop rate of accuracy metrics to evaluate the transferability. The model exhibits improved spatial transferability, demonstrating its effectiveness in generating accurate land cover maps for different cities. Multi-temporal analysis reveals the potential of the proposed method for studying land cover and land use changes over time. In addition, a comparison of the state-of-the-art full semantic segmentation models indicates that spatial details are fully exploited and presented in semantic segmentation results by the proposed method.

Faster structured illumination microscopy using complementary encoding-based compressive imaging

Structured illumination microscopy (SIM) has been widely applied to investigate intricate biological dynamics due to its outstanding super-resolution imaging speed. Incorporating compressive sensing into SIM brings the possibility to further improve the super-resolution imaging speed. Nevertheless, the recovery of the super-resolution information from the compressed measurement remains challenging in experiments. Here, we report structured illumination microscopy with complementary encoding-based compressive imaging (CECI-SIM) to realize faster super-resolution imaging. Compared to the nine measurements to obtain a super-resolution image in a conventional SIM, CECI-SIM can achieve a super-resolution image by three measurements; therefore, a threefold improvement in the imaging speed can be achieved. This faster imaging ability in CECI-SIM is experimentally verified by observing tubulin and actin in mouse embryonic fibroblast cells. This work provides a feasible solution for high-speed super-resolution imaging, which would bring significant applications in biomedical research.

Super-resolution imaging method assisted by a left-handed medium slab based on a neural network.

In this study, a super-resolution imaging method is proposed that combines the physical properties of a left-handed medium (LHM) slab and the mathematical methods of a neural network. Firstly, for the problem of super-resolution information loss in the received scattering field in general scenes, the LHM slab is used to construct a perfect lens to recover the evanescent wave component that carries super-resolution information. Secondly, the compressed sensing (CS) method is applied to image the sparse targets under the LHM environment. However, the perfect focus only occurs in spot or line positions. Therefore, the imaging width of conventional methods is limited and a more powerful mathematical method is needed. Finally, the neural network method is introduced to relax the limitations of target imaging width due to its strong non-linear fitting capability. The simulation results demonstrate that the imaging resolution can reach λ/10 with the assistance of LHM, while the CS method can realize super-resolution imaging of λ/20 based on prior information of spatially sparse targets. In addition, the neural network method proposed in this paper relaxes the limitation of targets, realizing super-resolution imaging of λ/20 for general targets.

Store and Fetch Immediately: Everything Is All You Need for Space-Time Video Super-resolution

Existing space-time video super-resolution (ST-VSR) methods fail to achieve high-quality reconstruction since they fail to fully explore the spatial-temporal correlations, long-range components in particular. Although the recurrent structure for ST-VSR adopts bidirectional propagation to aggregate information from the entire video, collecting the temporal information between the past and future via one-stage representations inevitably loses the long-range relations. To alleviate the limitation, this paper proposes an immediate storeand-fetch network to promote long-range correlation learning, where the stored information from the past and future can be refetched to help the representation of the current frame. Specifically, the proposed network consists of two modules: a backward recurrent module (BRM) and a forward recurrent module (FRM). The former first performs backward inference from future to past, while storing future super-resolution (SR) information for each frame. Following that, the latter performs forward inference from past to future to super-resolve all frames, while storing past SR information for each frame. Since FRM inherits SR information from BRM, therefore, spatial and temporal information from the entire video sequence is immediately stored and fetched, which allows drastic improvement for ST-VSR. Extensive experiments both on ST-VSR and space video super-resolution (S-VSR) as well as time video super-resolution (T-VSR) have demonstrated the effectiveness of our proposed method over other state-of-the-art methods on public datasets. Code is available https://github.com/hhhhhumengshun/SFI-STVR

Application of Super-resolution SPEED Microscopy in the Study of Cellular Dynamics.

Super-resolution imaging techniques have broken the diffraction-limited resolution of light microscopy. However, acquiring three-dimensional (3D) super-resolution information about structures and dynamic processes in live cells at high speed remains challenging. Recently, the development of high-speed single-point edge-excitation subdiffraction (SPEED) microscopy, along with its 2D-to-3D transformation algorithm, provides a practical and effective approach to achieving 3D subdiffraction-limit information in subcellular structures and organelles with rotational symmetry. One of the major benefits of SPEED microscopy is that it does not rely on complex optical components and can be implemented on a standard, inverted epifluorescence microscope, simplifying the process of sample preparation and the expertise requirement. SPEED microscopy is specifically designed to obtain 2D spatial locations of individual immobile or moving fluorescent molecules inside submicrometer biological channels or cavities at high spatiotemporal resolution. The collected data are then subjected to postlocalization 2D-to-3D transformation to obtain 3D super-resolution structural and dynamic information. In recent years, SPEED microscopy has provided significant insights into nucleocytoplasmic transport across the nuclear pore complex (NPC) and cytoplasm-cilium trafficking through the ciliary transition zone. This Review focuses on the applications of SPEED microscopy in studying the structure and function of nuclear pores.

G-Loss: A loss function with gradient information for super-resolution

G-Loss: A loss function with gradient information for super-resolution

Super-Resolution Multicomponent Joint-Interferometric Fabry–Perot-Based Technique

We propose a new spectral super-resolution technique combined with a Fabry–Perot interferometer (FPI) and an interferometric hyperspectral imager. To overcome the limitation of the maximal optical path difference (OPD) on the spectral resolution, the object spectrum is periodically modulated based on the FPI, and an optical Fourier transform of the modulated spectrum information is performed using a double-beam interferometer to obtain an interferogram. Drawing on the concept of nonlinear structured light microscopy, the displacement of the high-frequency interference information in the interferogram after adding the FPI is analyzed to restore the high-frequency interference information and improve the spectral resolution. The optical system has a compact structure with little impact on complexity, spectral range, or luminous flux. Our simulation results show that this method can realize multicomponent joint-interference imaging to obtain spectral super-resolution information. The effects of the FPI’s reflectance and interval are analyzed, and the reflectance needs to be within 20~80% and the interval must be as close as possible to the maximum optical range of the interferometer. Compared with previous, related innovations, this innovation has the advantages of higher system stability, higher data utilization, and better suitability for interferometric imaging spectrometers.

Obtaining 3D super-resolution images by utilizing rotationally symmetric structures and 2D-to-3D transformation

Super-resolution imaging techniques have provided unprecedentedly detailed information by surpassing the diffraction-limited resolution of light microscopy. However, in order to derive high quality spatial resolution, many of these techniques require high laser power, extended imaging time, dedicated sample preparation, or some combination of the three. These constraints are particularly evident when considering three-dimensional (3D) super-resolution imaging. As a result, high-speed capture of 3D super-resolution information of structures and dynamic processes within live cells remains both desirable and challenging. Recently, a highly effective approach to obtain 3D super-resolution information was developed that can be employed in commonly available laboratory microscopes. This development makes it both scientifically possible and financially feasible to obtain super-resolution 3D information under certain conditions. This is accomplished by converting 2D single-molecule localization data captured at high speed within subcellular structures and rotationally symmetric organelles. Here, a high-speed 2D single-molecule tracking and post-localization technique, known as single-point edge-excitation sub-diffraction (SPEED) microcopy, along with its 2D-to-3D transformation algorithm is detailed with special emphasis on the mathematical principles and Monte Carlo simulation validation of the technique.

Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes.

The goal when imaging bioprocesses with optical microscopy is to acquire the most spatiotemporal information with the least invasiveness. Deep neural networks have substantially improved optical microscopy, including image super-resolution and restoration, but still have substantial potential for artifacts. In this study, we developed rationalized deep learning (rDL) for structured illumination microscopy and lattice light sheet microscopy (LLSM) by incorporating prior knowledge of illumination patterns and, thereby, rationally guiding the network to denoise raw images. Here we demonstrate that rDL structured illumination microscopy eliminates spectral bias-induced resolution degradation and reduces model uncertainty by five-fold, improving the super-resolution information by more than ten-fold over other computational approaches. Moreover, rDL applied to LLSM enables self-supervised training by using the spatial or temporal continuity of noisy data itself, yielding results similar to those of supervised methods. We demonstrate the utility of rDL by imaging the rapid kinetics of motile cilia, nucleolar protein condensation during light-sensitive mitosis and long-term interactions between membranous and membrane-less organelles.

Reconstruction of finite deep sub-wavelength nanostructures by Mueller-matrix scattered-field microscopy.

Computational super-resolution is a novel approach to break the diffraction limit. The Mueller matrix, which contains full-polarization information about the morphology and structure of a sample, can add super-resolution information and be a promising way to further enhance the resolution. Here we proposed a new approach called Mueller-matrix scattered-field microscopy (MSM) that relies on a computational reconstruction strategy to quantitatively determine the geometrical parameters of finite deep sub-wavelength nanostructures. The MSM adopts a high numerical-aperture objective lens to collect a broad range of spatial frequencies of the scattered field of a sample in terms of Mueller-matrix images. A rigorous forward scattering model is established for MSM, which takes into account the vectorial nature of the scattered field when passing through the imaging system and the effect of defocus in the measurement process. The experimental results performed on a series of isolated Si lines have demonstrated that MSM can resolve a feature size of λ/16 with a sub-7 nm accuracy. The MSM is fast and has a great measurement accuracy for nanostructures, which is expected to have a great potential application for future nanotechnology and nanoelectronics manufacturing.

Angular‐flexible network for light field image super‐resolution

Light field (LF) cameras capture scenes from multiple views and provide additional angular information for image super-resolution (SR). Existing CNN-based LF image SR methods commonly develop specific models for different angular resolutions. However, since the angular resolution can vary significantly with different LF devices, these methods have limited flexibility for real-world applications. Here, an angular-flexible network to use a single model to super-resolve LF images of arbitrary angular resolution is proposed. In this method, spatial and angular feature extractors are designed to achieve angular-flexible feature extraction, and develop a decouple-and-fuse module for SR reconstruction. Moreover, a mixed-angular-resolution training strategy is proposed to further enhance the angular flexibility. Experimental results on five public datasets demonstrate the state-of-the-art performance of the method. Source codes are available at https://github.com/ZhengyuLiang24/LF-AFnet. Here, an angular-flexible network for light field image super-resolution is proposed. The network can handle LFs captured by different kinds of devices with arbitrary angular resolutions. Experimental results demonstrate the effectiveness and superior performance of the method.

Fast-switchable, high diffraction-efficiency ferroelectric liquid crystal Fibonacci grating.

Low-voltage fast switchable 1D and 2D Fibonacci grating (FbG), using an electrically suppressed helix ferroelectric liquid crystal (ESHFLC), with high diffraction efficiency for a super-resolution imaging system in far-field are disclosed in this paper. Specifically, the polarization-independent two-domain (0, π) structure is well designed based on photoalignment technology to maximize the total diffraction efficiency that can reach 97.4% (1st order:8.5%, 2nd order: 30%). Apart from that, the FLC gratings offer two tunable states: non-diffractive and diffractive states. Derived from the fast-response property of ferroelectric liquid crystal material, the switching speed of the 1D and 2D ESHFLC-FbG is 103µs at 4 V of the driving voltage. Furthermore, this system achieves the high-resolving power of (λ/2.25) for object detection based upon the intensity map received behind 1D ESHFLC-FbG at far-field. Contribution from the quasi-periodic FbG's special ability to translate the super-resolution information (including at evanescent wave) into the detectable far-field region. Concisely, the proposed ESHFLC-FbG can be a promising candidate for a super-resolution imaging system, superstructure fibre sensor, and other photonic applications.

Bidirectional 3D Quasi-Recurrent Neural Network for Hyperspectral Image Super-Resolution

Hyperspectral imaging is unable to acquire images with high resolution in both spatial and spectral dimensions yet, due to physical hardware limitations. It can only produce low spatial resolution images in most cases and thus hyperspectral image (HSI) spatial super-resolution is important. Recently, deep learning-based methods for HSI spatial super-resolution have been actively exploited. However, existing methods do not focus on structural spatial-spectral correlation and global correlation along spectra, which cannot fully exploit useful information for super-resolution. Also, some of the methods are straightforward extension of RGB super-resolution methods, which have fixed number of spectral channels and cannot be generally applied to hyperspectral images whose number of channels varies. Furthermore, unlike RGB images, existing HSI datasets are small and limit the performance of learning-based methods. In this article, we design a bidirectional 3D quasi-recurrent neural network for HSI super-resolution with arbitrary number of bands. Specifically, we introduce a core unit that contains a 3D convolutional module and a bidirectional quasi-recurrent pooling module to effectively extract structural spatial-spectral correlation and global correlation along spectra, respectively. By combining domain knowledge of HSI with a novel pretraining strategy, our method can be well generalized to remote sensing HSI datasets with limited number of training data. Extensive evaluations and comparisons on HSI super-resolution demonstrate improvements over state-of-the-art methods, in terms of both restoration accuracy and visual quality.

Experimental evaluation of super-resolution imaging and magnification choice in single-particle cryo-EM

The resolution of cryo-EM reconstructions is fundamentally limited by the Nyquist frequency, which is half the sampling frequency of the detector and depends upon the magnification used. In principle, super-resolution imaging should enable reconstructions to surpass the physical Nyquist limit by increasing sampling frequency, yet there are few reports of reconstructions that do so. Here we directly examine the contribution of super-resolution information, obtained with the K3 direct electron detector using a 2-condenser microscope, to single-particle cryo-EM reconstructions surpassing the physical Nyquist limit. We also present a comparative analysis of a sample imaged at four different magnifications. This analysis demonstrates that lower magnifications can be beneficial, despite the loss of higher resolution signal, due to the increased number of particle images obtained. To highlight the potential utility of lower magnification data collection, we produced a 3.5 Å reconstruction of jack bean urease with particles from a single micrograph.

High-speed super-resolution imaging of rotationally symmetric structures using SPEED microscopy and 2D-to-3D transformation.

Various super-resolution imaging techniques have been developed to break the diffraction-limited resolution of light microscopy. However, it still remains challenging to obtain three-dimensional (3D) super-resolution information of structures and dynamic processes in live cells at high speed. We recently developed high-speed single-point edge-excitation sub-diffraction (SPEED) microscopy and its two-dimensional (2D)-to-3D transformation algorithm to provide an effective approach to achieving 3D sub-diffraction-limit information in subcellular structures and organelles that have rotational symmetry. In contrast to most other 3D super-resolution microscopy or 3D particle-tracking microscopy approaches, SPEED microscopy does not depend on complex optical components and can be implemented onto a standard inverted epifluorescence microscope. SPEED microscopy is specifically designed to obtain 2D spatial locations of individual immobile or moving fluorescent molecules inside sub-micrometer biological channels or cavities at high spatiotemporal resolution. After data collection, post-localization 2D-to-3D transformation is applied to obtain 3D super-resolution structural and dynamic information. The complete protocol, including cell culture and sample preparation (6-7 d), SPEED imaging (4-5 h), data analysis and validation through simulation (5-13 h), takes ~9 d to complete.

Light Field Image Super-Resolution Using Deformable Convolution.

Light field (LF) cameras can record scenes from multiple perspectives, and thus introduce beneficial angular information for image super-resolution (SR). However, it is challenging to incorporate angular information due to disparities among LF images. In this paper, we propose a deformable convolution network (i.e., LF-DFnet) to handle the disparity problem for LF image SR. Specifically, we design an angular deformable alignment module (ADAM) for feature-level alignment. Based on ADAM, we further propose a collect-and-distribute approach to perform bidirectional alignment between the center-view feature and each side-view feature. Using our approach, angular information can be well incorporated and encoded into features of each view, which benefits the SR reconstruction of all LF images. Moreover, we develop a baseline-adjustable LF dataset to evaluate SR performance under different disparity variations. Experiments on both public and our self-developed datasets have demonstrated the superiority of our method. Our LF-DFnet can generate high-resolution images with more faithful details and achieve state-of-the-art reconstruction accuracy. Besides, our LF-DFnet is more robust to disparity variations, which has not been well addressed in literature.

Electron-event representation data enable efficient cryoEM file storage with full preservation of spatial and temporal resolution.

Direct detector device (DDD) cameras have revolutionized electron cryomicroscopy (cryoEM) with their high detective quantum efficiency (DQE) and output of movie data. A high ratio of camera frame rate (frames per second) to camera exposure rate (electrons per pixel per second) allows electron counting, which further improves the DQE and enables the recording of super-resolution information. Movie output also allows the correction of specimen movement and compensation for radiation damage. However, these movies come at the cost of producing large volumes of data. It is common practice to sum groups of successive camera frames to reduce the final frame rate, and therefore the file size, to one suitable for storage and image processing. This reduction in the temporal resolution of the camera requires decisions to be made during data acquisition that may result in the loss of information that could have been advantageous during image analysis. Here, experimental analysis of a new electron-event representation (EER) data format for electron-counting DDD movies is presented, which is enabled by new hardware developed by Thermo Fisher Scientific for their Falcon DDD cameras. This format enables the recording of DDD movies at the raw camera frame rate without sacrificing either spatial or temporal resolution. Experimental data demonstrate that themethod retains super-resolution information and allows the correction of specimen movement at the physical frame rate of the camera while maintaining manageable file sizes. The EER format will enable the development of new methods that can utilize the full spatial and temporal resolution of DDD cameras.

Adjustable super-resolution microscopy with diffractive spot array illumination

Diffractive super-resolution spot arrays offer the possibility of adjustable super-resolution microscopic imaging. By inserting a phase-only diffractive optical element (DOE) into the illumination system of a standard microscope, super-resolution information of the sample can be obtained by spots. Here, we report an adjustable super-resolution microscopy (ASM) that the imaging resolution and the number of spot arrays can be adjusted by DOEs. The results of 3 × 3 and 5 × 5 spot arrays with 50% and 70% of the Airy spot size are, respectively, realized to support the flexibility of the ASM. The resolution test target was used as a sample to show the ASM can achieve about double-resolution experimentally, illuminated by a uniform 3 × 3 spot array with 50% of the Airy spot size under a small numerical aperture objective. Moreover, imaging of cellular mitochondria was performed, substantially realized resolution beyond the diffraction limit. The ability to adjust the super-resolution of microscopy using DOEs is of great importance for further nanoscale imaging.

基于对焦清晰度的双分辨相机变焦算法研究

The dual resolution camera combines large field of view and high resolution. The coaxial optical fixed structure avoids many problems caused by the moving parts of the zoom lens, and has potential application value in deep space detection target tracking and smart terminals such as mobile phones. The existing dual resolution image zoom algorithm based on deep learning has slow speed, no increase in information, poor adaptability of image network structure and forgery of image information restoration. To join a solution in normal image algorithm based on depth information was tentatively proposed in this paper. The feasibility of using image focus sharpness as depth information into the dual resolution zoom algorithm was demonstrated, the accuracy and effect of focus depth information detection was explored by Laplace image evaluation, and the dual resolution zoom algorithm of deep learning and traditional methods based on depth information was tested. The new excellent zoom algorithm used the space during normal focus of the camera which did not affect the imaging speed of normal images. The algorithm complexity is reduced by 60%, the computational memory overhead is reduced by 35%. The spatial relationship super-resolution information is more realistic and reliable, and the image result evaluation is improved by 10% to 50%.