Final Super-resolution Image Research Articles

FACAM: A Fast and Accurate Clustering Analysis Method for Protein Complex Quantification in Single Molecule Localization Microscopy

Single molecule localization microscopy (SMLM) enables the analysis and quantification of protein complexes at the nanoscale. Using clustering analysis methods, quantitative information about protein complexes (for example, the size, density, number, and the distribution of nearest neighbors) can be extracted from coordinate-based SMLM data. However, since a final super-resolution image in SMLM is usually reconstructed from point clouds that contain millions of localizations, current popular clustering methods are not fast enough to enable daily quantification on such a big dataset. Here, we provide a fast and accurate clustering analysis method called FACAM, which is modified from the Alpha Shape method (a point dataset analysis method used in many fields). By taking advantage of parallel computation, FACAM is able to process millions of localizations in less than an hour, which is at least 10 times faster than the popular DBSCAN method. Furthermore, FACAM adaptively determines the segmentation threshold, and thus overcomes the problem of user-defined parameters. Using simulation and experimental datasets, we verified the advantages of FACAM over other reported clustering methods (including Ripley’s H, DBSCAN, and ClusterViSu).

Superresolution Composite SAR Imaging Method by Hierarchical Pattern With Attributed Scattering Priors

Compared with the ideal point scattering model, the attributed scattering center (ASC) model provides concise and physically relevant description of the man-made targets, such as the vehicles and ships, giving an efficient way to interpret the measurements of high-frequency and wide-angle synthetic aperture radar (SAR). Since the ASC model is complicated, it gives rise to a high-dimensional model parameter estimation problem, leading to a heavy computational burden. Aimed at this problem, in this article, a novel superresolution composite SAR imaging algorithm is proposed based on sparse representation, in which the ASC attributes are estimated by a hierarchical pattern for reducing the computational cost. In the proposed algorithm, the ASC model parameters, classified into three categories, are estimated coarsely via minimum variance pursuit (MVP). Then these coarse estimates are refined jointly using the consensus alternating direction method of multipliers (CADMM) optimization for polarimetric measurements. The spectrum extrapolation and replacement are performed for superresolution and reducing the impact of model mismatch, followed by a fast Fourier transform (FFT)-based operation to efficiently obtain the final superresolution image. The proposed approach not only possesses the ability of superresolution reconstruction with the target nonideal scattering features’ enhancement but also offers the physically relevant attributes on the premise of estimation accuracy improvement and calculation capacity reduction. Extensive experiments are conducted to corroborate the effectiveness of the proposed algorithm.

Endoscopic Ultrasound Localization Microscopy for the Evaluation of the Microvasculature of Gastrointestinal Tract Tumors in Rabbits.

The morphological and hemodynamic characterization of the microvascular network around the gastrointestinal (GI) tract can be of significant clinical value for the early diagnosis and treatment of GI tract cancer. Ultrasound localization microscopy (ULM) imaging has been demonstrated to be capable of resolving the microvascular network. However, the endoscopic application of ULM imaging techniques is still unknown. In this study, an endoscopic ultrasound localization microscopy (e-ULM) imaging technique was developed to evaluate the changes of microvasculature during GI tract tumor growth. A customized circular array transducer (center frequency: 6.8 MHz) and the coherent diverging wave compounding method were used to generate B-mode images. Spatiotemporal singular value decomposition processing was used to eliminate the background signals before signal localizations. The centroids of spatially isolated signals were localized and summed to generate the final super-resolution image. The final microvasculature map of a rabbit GI tract tumor reveals that e-ULM can be used to surpass the diffraction limit in traditional endoscopic ultrasound (EUS) imaging. Furthermore, it is observed that data from different stages of tumor growth exhibit significant differences in microvascular pattern and density. Our study demonstrated the implementation and application of an in vivo e-ULM imaging technique for the evaluation of the microvasculature of GI tumors. The efficient e-ULM imaging technique shows potential for use in the detection of GI tract tumor microcirculation changes and subsequent diagnosis of GI tract cancer.

Development of Deep-Learning-Based Single-Molecule Localization Image Analysis.

Recent developments in super-resolution fluorescence microscopic techniques (SRM) have allowed for nanoscale imaging that greatly facilitates our understanding of nanostructures. However, the performance of single-molecule localization microscopy (SMLM) is significantly restricted by the image analysis method, as the final super-resolution image is reconstructed from identified localizations through computational analysis. With recent advancements in deep learning, many researchers have employed deep learning-based algorithms to analyze SMLM image data. This review discusses recent developments in deep-learning-based SMLM image analysis, including the limitations of existing fitting algorithms and how the quality of SMLM images can be improved through deep learning. Finally, we address possible future applications of deep learning methods for SMLM imaging.

Influence of drift correction precision on super-resolution localization microscopy.

Super-resolution localization microscopy (SRLM) breaks the diffraction limit successfully and improves the resolution of optical imaging systems by nearly an order of magnitude. However, SRLM typically takes several minutes or longer to collect a sufficient number of image frames that are required for reconstructing a final super-resolution image. During this long image acquisition period, system drift should be tightly controlled to ensure the imaging quality; thus, several drift correction methods have been developed. However, it is still unclear whether the performance of these methods is able to ensure sufficient image quality in SRLM. Without a clear answer to this question, it is hard to choose a suitable drift correction method for a specific SRLM experiment. In this paper, we use both theoretical analysis and simulation to investigate the relationship among drift correction precision, localization precision, and position estimation precision. We propose a concept of relative localization precision for evaluating the effect of drift correction on imaging resolution, which would help to select an appropriate drift correction method for a specific experiment.

An Adaptive Texture Enhancement Algorithm for AR Live Screen Based on Approximate Matching

In order to improve the visual effect of AR live video detail texture, an adaptive enhancement algorithm based on approximate matching is proposed. According to the local self-similarity of the original image, the best matching block of the initial super-resolution image block is obtained. After extracting its high-frequency information, the improved singular value decomposition (SVD) is used to embed the watermark into the original super-resolution gray image; And through the watermark extraction and reconstruction matrix, the effective high-frequency detail texture information is obtained, then the final super-resolution image is obtained, and the AR live video detail texture enhancement is completed. The experimental results show that the peak signal-to-noise ratio, structure similarity and feature similarity of the algorithm are high, and the reconstruction effect is good; After detail texture enhancement, the detail texture clarity and edge sharpness of AR live picture are better, and the best visual effect is achieved.

Automated Image Super Resolution with the Aid of Activation Function Optimized Deep CNN and Adaptive Wavelet Lifting Approach

Diverse image super-resolution (SR) techniques have been implemented to reconstruct the high-resolution (HR) images from input images through lower spatial resolutions. However, the evaluation of the perceptual quality of SR images remains an important and complex research problem. This paper proposes a new image SR model with the intention of attaining maximum Peak Signal-to-Noise Ratio (PSNR). The conversion of low-resolution (LR) images from the HR images is performed by bicubic interpolation-based downsampling and upsampling. Then, the four sub-bands of LR and HR images are generated by the novel Adaptive Wavelet Lifting approach, in which the filter modes are optimized using the proposed SA-CBO. From this technique, LR wavelet sub-bands (LRSB) for LR images and HR wavelet sub-bands (HRSB) for HR images are formed. With the help of the LRSB and HRSB images, the residual images are formed by the adoption of the optimized Activation function and optimized hidden neurons in a deep convolutional neural network (CNN). The improvement in both the adaptive wavelet lifting approach and deep CNN is made by the self-adaptive-colliding bodies optimization (SA-CBO). Finally, the inverse adaptive wavelet lifting approach is used to produce the final SR image. Experimental results on publicly available SR image quality databases confirm the effectiveness and generalization ability of the proposed method compared with the traditional image quality assessment algorithms.

Resampling and super-resolution of hexagonally sampled images using deep learning

Super-resolution (SR) aims to increase the resolution of imagery. Applications include security, medical imaging, and object recognition. We propose a deep learning-based SR system that takes a hexagonally sampled low-resolution image as an input and generates a rectangularly sampled SR image as an output. For training and testing, we use a realistic observation model that includes optical degradation from diffraction and sensor degradation from detector integration. Our SR approach first uses non-uniform interpolation to partially upsample the observed hexagonal imagery and convert it to a rectangular grid. We then leverage a state-of-the-art convolutional neural network (CNN) architecture designed for SR known as Residual Channel Attention Network (RCAN). In particular, we use RCAN to further upsample and restore the imagery to produce the final SR image estimate. We demonstrate that this system is superior to applying RCAN directly to rectangularly sampled LR imagery with equivalent sample density. The theoretical advantages of hexagonal sampling are well known. However, to the best of our knowledge, the practical benefit of hexagonal sampling in light of modern processing techniques such as RCAN SR is heretofore untested. Our SR system demonstrates a notable advantage of hexagonally sampled imagery when employing a modified RCAN for hexagonal SR.

Image super resolution model enabled by wavelet lifting with optimized deep convolutional neural network

AbstractThis paper plans to develop an intelligent super resolution model with the linkage of Wavelet lifting scheme and Deep learning algorithm. Before initiating the resolution procedure, the entire HR images are converted into Low Resolution (LR) images using bicubic interpolation‐based downsampling and upsampling. Further, the Wavelet lifting scheme helps to generate the four subbands of each image like LR wavelet Sub‐Bands for LR images, and High Resolution (HR) wavelet Sub‐Bands for HR images. The residual image is generated by taking the difference between the LR wavelet Sub‐Bands and HR wavelet Sub‐Bands images. The proposed model involves two main phases: Training phase and Testing. The training phase trains the residual image of all images by Deep Convolutional Neural Network with LR wavelet Sub‐Bands as input and residual image as target. On the other hand, in testing phase, the LR wavelet Sub‐Bands query image is subjected to Deep Convolutional Neural Network, which outputs the concerned residual image. This generated residual image is summed with LR wavelet Sub‐Bands image, followed by inverse wavelet lifting scheme to obtain the final super resolution image. The main contribution of this paper is to improve the conventional Deep Convolutional Neural Network by optimizing the number of hidden layer, and hidden neurons using modified Whale Optimization Algorithm called Average Fitness Enabled Whale Optimization Algorithm by considering the objective of maximizing the Peak Signal‐to‐Noise Ratio. Finally, the proposed method achieves an improved quality of the results which is comparable the existing models.

Enhancing Blurred Low-Resolution Images via Exploring the Potentials of Learning-Based Super-Resolution

This paper aims to propose a candidate solution to the challenging task of single-image blind super-resolution (SR), via extensively exploring the potentials of learning-based SR schemes in the literature. The task is formulated into an energy functional to be minimized with respect to both an intermediate super-resolved image and a nonparametric blur-kernel. The functional includes a so-called convolutional consistency term which incorporates a nonblind learning-based SR result to better guide the kernel estimation process, and a bi-[Formula: see text]-[Formula: see text]-norm regularization imposed on both the super-resolved sharp image and the nonparametric blur-kernel. A numerical algorithm is deduced via coupling the splitting augmented Lagrangian (SAL) and the conjugate gradient (CG) method. With the estimated blur-kernel, the final SR image is reconstructed using a simple TV-based nonblind SR method. The proposed blind SR approach is demonstrated to achieve better performance than [T. Michaeli and M. Irani, Nonparametric Blind Super-resolution, in Proc. IEEE Conf. Comput. Vision (IEEE Press, Washington, 2013), pp. 945–952.] in terms of both blur-kernel estimation accuracy and image ehancement quality. In the meanwhile, the experimental results demonstrate surprisingly that the local linear regression-based SR method, anchored neighbor regression (ANR) serves the proposed functional more appropriately than those harnessing the deep convolutional neural networks.

Structure-resolving index: an efficient criterion for ending image acquisition in super-resolution localization microscopy

In super-resolution localization microscopy, a final super-resolution image is built upon a huge amount of raw image frames (typically >10,000). To guarantee the quality of a final super-resolution image, researchers tend to collect as many raw images as possible, leading to excess data volume, which causes a significant waste of data acquisition time and computation resources. Here we present the structure-resolving index (SRI), an efficient criterion to end image acquisition automatically. Simulated and experimental data show that SRI can be used as a superior ending criterion to minimize the acquisition of excess data volume while ensuring the resolving ability of a final super-resolution image.

Deep Distillation Recursive Network for Remote Sensing Imagery Super-Resolution

Deep convolutional neural networks (CNNs) have been widely used and achieved state-of-the-art performance in many image or video processing and analysis tasks. In particular, for image super-resolution (SR) processing, previous CNN-based methods have led to significant improvements, when compared with shallow learning-based methods. However, previous CNN-based algorithms with simple direct or skip connections are of poor performance when applied to remote sensing satellite images SR. In this study, a simple but effective CNN framework, namely deep distillation recursive network (DDRN), is presented for video satellite image SR. DDRN includes a group of ultra-dense residual blocks (UDB), a multi-scale purification unit (MSPU), and a reconstruction module. In particular, through the addition of rich interactive links in and between multiple-path units in each UDB, features extracted from multiple parallel convolution layers can be shared effectively. Compared with classical dense-connection-based models, DDRN possesses the following main properties. (1) DDRN contains more linking nodes with the same convolution layers. (2) A distillation and compensation mechanism, which performs feature distillation and compensation in different stages of the network, is also constructed. In particular, the high-frequency components lost during information propagation can be compensated in MSPU. (3) The final SR image can benefit from the feature maps extracted from UDB and the compensated components obtained from MSPU. Experiments on Kaggle Open Source Dataset and Jilin-1 video satellite images illustrate that DDRN outperforms the conventional CNN-based baselines and some state-of-the-art feature extraction approaches.

Super-resolution based on self-example learning and guided filtering

A sparse representation (using self-example dictionary learning)-based framework for denoising and super-resolution (SR) is proposed. The proposed scheme makes use of fast nonnegative orthogonal matching pursuit for the sparse coding. The dictionary learning is implemented using the K-singular value decomposition. The scheme preprocesses the low-resolution noisy image with a denoising algorithm. The SR versions of the noisy and denoised images are computed by self-example learning algorithm. The resultant SR images are combined through guided (edge preserving and scale aware) filtering technique that preserves high-frequency textural information to obtain a final SR image. Quantitative analysis and visual results demonstrate the significance of proposed scheme.

Super-Resolution Digital Pathology Image Processing of Bone Marrow Aspirate and Cytology Smears and Tissue Sections

Background: Accurate digital pathology image analysis depends on high-quality images. As such, it is imperative to obtain digital images with high resolution for downstream data analysis. While hematoxylin and eosin (H&E)-stained tissue section slides from solid tumors contain three-dimensional information, these data have been ignored in digital pathology. In addition, in cytology and bone marrow aspirate smears, the three-dimensional nature of the specimen has precluded efficient analysis of such morphologic data. An individual image snapshot at a single focal distance is often not sufficient for accurate diagnoses and multiple whole-slide images at different focal distances are necessary for diagnostics. Materials and Methods: We describe a novel computational pipeline and processing program for obtaining a super-resolved image from multiple static images at different z-planes in overlapping but separate frames. This program, MULTI-Z, performs image alignment, Gaussian smoothing, and Laplacian filtering to construct a final super-resolution image from multiple images. Results: We applied this algorithm and program to images of cytology and H&E-stained sections and demonstrated significant improvements in both resolution and image quality by objective data analyses (24% increase in sharpness and focus). Conclusions: With the use of our program, super-resolved images of cytology and H&E-stained tissue sections can be obtained to potentially allow for more optimal downstream computational analysis. This method is applicable to whole-slide scanned images.

Improved Imaging Performance in Super-Resolution Localization Microscopy by YALL1 Method

In super-resolution localization microscopy, e.g., stochastic optical reconstruction microscopy or photoactivated localization microscopy, a long acquisition time is required because of stochastic imaging nature, which limits its application in dynamic imaging for live cell. To overcome the limitation, one approach based on compressed sensing (CS) has been used in the previous reports. However, the imaging performance obtained by this method may be affected due to the use of interior point method (IPM). To address the problem, in this paper, we introduce an alternative CS reconstruction method and apply the recently developed YALL1 (your algorithm for L1 norm problems) method to super-resolution imaging model. Two types of numerical simulation experiments were performed to evaluate the performance of the proposed method. In case 1, the microscopy data from a single frame was simulated, which was used to evaluate the performance of YALL1 in single-emitter detection. In case 2, the dynamic microscopy data from a series of time points was generated, which was used to evaluate the performance of YALL1 in resolving fine structures. The results show that compared with the previous reported IPM method, the localization accuracy of super-resolution is improved by the proposed YALL1 method, even if there is high emitter density and noise in measurement data. In addition, the imaging time can also be reduced, because fewer imaging cycles are required for reconstructing the final super-resolution image by YALL1 method. Hence, the technique provides the potential in imaging fast cellular processes.

Image super-resolution by estimating the enhancement weight of self example and external missing patches

Image super-resolution (SR) is the process of generating a high-resolution (HR) image using one or more low-resolution (LR) inputs. Many SR methods have been proposed, but generating the small-scale structure of an SR image remains a challenging task. We hence propose a single-image SR algorithm that combines the benefits of both internal and external SR methods. First, we estimate the enhancement weights of each LR-HR image patch pair. Next, we multiply each patch by the estimated enhancement weight to generate an initial SR patch. We then employ a method to recover the missing information from the high-resolution patches and create that missing information to generate a final SR image. We then employ iterative back-projection to further enhance visual quality. The method is compared qualitatively and quantitatively with several state-of-the-art methods, and the experimental results indicate that the proposed framework provides high contrast and better visual quality, particularly for non-smooth texture areas.

Detail enhancement of image super-resolution based on detail synthesis

In this paper, a detail-enhancement and super-resolution algorithm based on detail synthesis is proposed. The novelty of this algorithm is in combining local self-similarity search and singular value decomposition of patches together to obtain details with more natural high-frequency. The proposed algorithm improves the facet or line phenomenon on edges and areas that have rich texture. The algorithm firstly searches for an image patch and extracts the high-frequency components based on a local self-similarity of the original, low-resolution image. The matrix of the high-frequency block is then decomposed into two sub-spaces by the singular value decomposition and the pseudo high-frequency is removed by a soft threshold. Then, the high-frequency block is reconstructed using effective singular values. The final super-resolution image is restored by the detail synthesis with the initial super-resolution image. The experimental results show that the proposed method can significantly remove the artificial effect of facet or line phenomenon caused by pseudo high-frequency. Moreover, the method is also applicable to other super-resolution algorithm in detail enhancement.

Improved superresolution microscopy imaging by sparse deconvolution with an interframe penalty

Penalized regression with a combination of sparseness and an interframe penalty is explored for image deconvolution in wide‐field single‐molecule fluorescence microscopy. The aim is to reconstruct superresolution images, which can be achieved by averaging the positions and intensities of individual fluorophores obtained from the analysis of successive frames. Sparsity of the fluorophore distribution in the spatial domain is obtained with an L0‐norm penalty on estimated fluorophore intensities, effectively constraining the number of fluorophores per frame. Simultaneously, continuity of the fluorophore localizations in the time mode is obtained by penalizing the total numbers of pixel status changes between successive frames. We implemented the interframe penalty in a sparse deconvolution algorithm (sparse image deconvolution and reconstruction) for improved imaging of densely labeled biological samples. For simulated and real biological data, we show that more accurate estimates of the final superresolution images of cellular structures can be obtained.

Structural anisotropy quantification improves the final superresolution image of localization microscopy.

Superresolution localization microscopy initially produces a dataset of fluorophore coordinates instead of a conventional digital image. Therefore, superresolution localization microscopy requires additional data analysis to present a final superresolution image. However, methods of employing the structural information within the localization dataset to improve the data analysis performance remain poorly developed. Here, we quantify the structural information in a localization dataset using structural anisotropy, and propose to use it as a figure of merit for localization event filtering. With simulated as well as experimental data of a biological specimen, we demonstrate that exploring structural anisotropy has allowed us to obtain superresolution images with a much cleaner background.

Self Super-resolution for Magnetic Resonance Images.

It is faster and therefore cheaper to acquire magnetic resonance images (MRI) with higher in-plane resolution than through-plane resolution. The low resolution of such acquisitions can be increased using post-processing techniques referred to as super-resolution (SR) algorithms. SR is known to be an ill-posed problem. Most state-of-the-art SR algorithms rely on the presence of external/training data to learn a transform that converts low resolution input to a higher resolution output. In this paper an SR approach is presented that is not dependent on any external training data and is only reliant on the acquired image. Patches extracted from the acquired image are used to estimate a set of new images, where each image has increased resolution along a particular direction. The final SR image is estimated by combining images in this set via the technique of Fourier Burst Accumulation. Our approach was validated on simulated low resolution MRI images, and showed significant improvement in image quality and segmentation accuracy when compared to competing SR methods. SR of FLuid Attenuated Inversion Recovery (FLAIR) images with lesions is also demonstrated.