Microscopic Imaging Modalities Research Articles

Unpaired data training enables super-resolution confocal microscopy from low-resolution acquisitions.

Supervised deep-learning models have enabled super-resolution imaging in several microscopic imaging modalities, increasing the spatial lateral bandwidth of the original input images beyond the diffraction limit. Despite their success, their practical application poses several challenges in terms of the amount of training data and its quality, requiring the experimental acquisition of large, paired databases to generate an accurate generalized model whose performance remains invariant to unseen data. Cycle-consistent generative adversarial networks (cycleGANs) are unsupervised models for image-to-image translation tasks that are trained on unpaired datasets. This paper introduces a cycleGAN framework specifically designed to increase the lateral resolution limit in confocal microscopy by training a cycleGAN model using low- and high-resolution unpaired confocal images of human glioblastoma cells. Training and testing performances of the cycleGAN model have been assessed by measuring specific metrics such as background standard deviation, peak-to-noise ratio, and a customized frequency content measure. Our cycleGAN model has been evaluated in terms of image fidelity and resolution improvement using a paired dataset, showing superior performance than other reported methods. This work highlights the efficacy and promise of cycleGAN models in tackling super-resolution microscopic imaging without paired training, paving the path for turning home-built low-resolution microscopic systems into low-cost super-resolution instruments by means of unsupervised deep learning.

A generalized deep neural network approach for improving resolution of fluorescence microscopy images

Deep learning is capable of greatly promoting the progress of super-resolution imaging technology in terms of imaging and reconstruction speed, imaging resolution, and imaging flux. This paper proposes a deep neural network based on a generative adversarial network (GAN). The generator employs a U-Net-based network, which integrates DenseNet for the downsampling component. The proposed method has excellent properties, for example, the network model is trained with several different datasets of biological structures; the trained model can improve the imaging resolution of different microscopy imaging modalities such as confocal imaging and wide-field imaging; and the model demonstrates a generalized ability to improve the resolution of different biological structures even out of the datasets. In addition, experimental results showed that the method improved the resolution of caveolin-coated pits (CCPs) structures from 264[Formula: see text]nm to 138[Formula: see text]nm, a 1.91-fold increase, and nearly doubled the resolution of DNA molecules imaged while being transported through microfluidic channels.

Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration.

Fluorescence microscopy-based image restoration has received widespread attention in the life sciences and has led to significant progress, benefiting from deep learning technology. However, most current task-specific methods have limited generalizability to different fluorescence microscopy-based image restoration problems. Here, we seek to improve generalizability and explore the potential of applying a pretrained foundation model to fluorescence microscopy-based image restoration. We provide a universal fluorescence microscopy-based image restoration (UniFMIR) model to address different restoration problems, and show that UniFMIR offers higher image restoration precision, better generalization and increased versatility. Demonstrations on five tasks and 14 datasets covering a wide range of microscopy imaging modalities and biological samples demonstrate that the pretrained UniFMIR can effectively transfer knowledge to a specific situation via fine-tuning, uncover clear nanoscale biomolecular structures and facilitate high-quality imaging. This work has the potential to inspire and trigger new research highlights for fluorescence microscopy-based image restoration.

A unified microstructure segmentation approach via human-in-the-loop machine learning

Microstructure segmentation, a technique for extracting structural statistics from microscopy images, is an essential step for establishing quantitative structure–property relationships in a broad range of materials research areas. However, the task is challenging due to the morphological complexity and diversity of structural constituents. While recent breakthroughs in deep learning have led to significant progress in microstructure segmentation, there remain two fundamental challenges to its further diffusion: prohibitive annotation costs and an unreliable decision-making process. To tackle these challenges, we propose a human-in-the-loop machine learning framework for unified microstructure segmentation, which leverages recent advances in both weakly supervised learning and active learning techniques. The key idea behind our approach lies in the integration of human and machine capabilities to make not only precise but also reliable microstructure segmentation at minimal annotation costs. Extensive experiments demonstrate the generality of our framework across different material classes, structural constituents, and microscopic imaging modalities.

A phantom for the quantitative determination and improvement of the spatial resolution in slice-selective 2D-FT magnetic resonance micro-imaging and -microscopy based on Deep X-ray Lithography (DXRL)

Introduction: The most important assessed quality-control (QC) criteria for improvements in high-resolution imaging are represented by the contrast-to-noise-ratio and spatial resolution. Ultra-High-Field (UHF) Magnetic-Resonance-scanners (B ≥ 7 T) for medical research allowed for the improvement in spatial resolution up to the microimaging and nominal microscopy range [pixel-size: ps < (100 μm)2], even in-vivo on humans just recently. Preclinical MRI- and dedicated MR-microscopy (MRM) scanners already allow for microimaging and MRM (1-256 μm) but lack a sensible spatial resolution phantom for QC and performance improvements in hardware, pulse-sequencing and MRprotocols. In most scientific MRI articles, the spatial resolution is characterized by the ps, though this measurement parameter only limits the actual resolution.Methods: Here the Modulation-Transfer-Function (MTF) is used as evaluation concept for the determination of the spatial resolution in MRM using simple intensity profiles. The resolution limit is defined using a critical modulation-level. In approaching visual impressions on spatial resolution an additional criterion derived from the Modulation-depth-to-Noise-Ratio (MNR) is proposed. A practical method for assessment based on a concrete phantom design and its realization is shown.Results: The phantom design consists of several sets of fine grids, specifically featuring high structural anisotropy for optimum SNR and CNR, with different spatial periods ranging from a1 = 256 μm down to a8 = 2 μm, not only for a quick visual qualitative check, but also for quantification of resolution using the MTF for two different spatial encodings in two orthogonal in-plane directions. The challenging demands on the manufacturing technology especially with regard to the aspect-ratio are approached using Deep-X-Ray-Lithography (DXRL) relying on the high brilliance of Synchroton-radiation. Smallest grid plates with width of 4 μm corresponding to 125 line pairs/mm at a plate depth of 100 μm were achieved.Discussion: MR-microscopic images, originating from a microscopy insert on a human UHF-MR-scanner, were used for demonstration of the evaluation process with two independent resolution-criteria. The developed prototype offers unique possibilities for quantitative resolution QC on UHF human and preclinical MR-scanners. Such a resolution-phantom might be very important for the improvement of MR-pulse-sequences, MR-protocols and even hardware. In principle the phantom can also be used for other microscopic imaging-modalities as for instance μCT and Optical-Coherence-Tomography (OCT).

The hair cell analysis toolbox is a precise and fully automated pipeline for whole cochlea hair cell quantification.

Our sense of hearing is mediated by sensory hair cells, precisely arranged and highly specialized cells subdivided into outer hair cells (OHCs) and inner hair cells (IHCs). Light microscopy tools allow for imaging of auditory hair cells along the full length of the cochlea, often yielding more data than feasible to manually analyze. Currently, there are no widely applicable tools for fast, unsupervised, unbiased, and comprehensive image analysis of auditory hair cells that work well either with imaging datasets containing an entire cochlea or smaller sampled regions. Here, we present a highly accurate machine learning-based hair cell analysis toolbox (HCAT) for the comprehensive analysis of whole cochleae (or smaller regions of interest) across light microscopy imaging modalities and species. The HCAT is a software that automates common image analysis tasks such as counting hair cells, classifying them by subtype (IHCs versus OHCs), determining their best frequency based on their location along the cochlea, and generating cochleograms. These automated tools remove a considerable barrier in cochlear image analysis, allowing for faster, unbiased, and more comprehensive data analysis practices. Furthermore, HCAT can serve as a template for deep learning-based detection tasks in other types of biological tissue: With some training data, HCAT's core codebase can be trained to develop a custom deep learning detection model for any object on an image.

Strategies to convert organic fluorophores into red/near-infrared emitting analogues and their utilization in bioimaging probes.

Organic fluorophores aided by current microscopy imaging modalities are essential for studying biological systems. Recently, red/near-infrared emitting fluorophores have attracted great research efforts, as they enable bioimaging applications with reduced autofluorescence interference and light scattering, two significant obstacles for deep-tissue imaging, as well as reduced photodamage and photobleaching. Herein, we analyzed the current strategies to convert key organic fluorophores bearing xanthene, coumarin, and naphthalene cores into longer wavelength-emitting derivatives by focussing on their effectiveness and limitations. Together, we introduced typical examples of how such fluorophores can be used to develop molecular probes for biological analytes, along with key sensing features. Finally, we listed several critical issues to be considered in developing new fluorophores.

Fluorescence lifetime image microscopy prediction with convolutional neural networks for cell detection and classification in tissues.

Convolutional neural networks (CNNs) and other deep-learning models have proven to be transformative tools for the automated analysis of microscopy images, particularly in the domain of cellular and tissue imaging. These computer-vision models have primarily been applied with traditional microscopy imaging modalities (e.g. brightfield and fluorescence), likely due to the availability of large datasets in these regimes. However, more advanced microscopy imaging techniques could, potentially, allow for improved model performance in various computational histopathology tasks. In this work, we demonstrate that CNNs can achieve high accuracy in cell detection and classification without large amounts of data when applied to histology images acquired by fluorescence lifetime imaging microscopy (FLIM). This accuracy is higher than what would be achieved with regular single or dual-channel fluorescence images under the same settings, particularly for CNNs pretrained on publicly available fluorescent cell or general image datasets. Additionally, generated FLIM images could be predicted from just the fluorescence image data by using a dense U-Net CNN model trained on a subset of ground-truth FLIM images. These U-Net CNN generated FLIM images demonstrated high similarity to ground truth and improved accuracy in cell detection and classification over fluorescence alone when used as input to a variety of commonly used CNNs. This improved accuracy was maintained even when the FLIM images were generated by a U-Net CNN trained on only a few example FLIM images.

Visualization of brain microvasculature and blood flow in vivo: Feasibility study using confocal laser endomicroscopy.

Qualitative and quantitative analyses of blood flow in normal and pathologic brain and spinal cord microvasculature were performed using confocal laser endomicroscopy (CLE). Blood flow in cortical, dural, and spinal cord microvasculature was assessed in vivo in swine. We assessed microvasculature under normal conditions and after vessel occlusion, brain injury due to cold or surgical trauma, and cardiac arrest. Tumor-associated microvasculature was assessed in vivo and ex vivo in 20 patients with gliomas. We observed erythrocyte flow in vessels 5-500µm in diameter. Thrombosis, flow arrest and redistribution, flow velocity changes, agglutination, and cells rolling were assessed in normal and injured brain tissue. Microvasculature in in vivo CLE images of gliomas was classified as normal in 68% and abnormal in 32% of vessels on the basis of morphological appearance. Dural lymphatic channels were discriminated from blood vessels. Microvasculature CLE imaging was possible for up to 30minutes after a 1mg/kg intravenous dose of fluorescein. CLE imaging allows assessment of cerebral and tumor microvasculature and blood flow alterations with subcellular resolution intraoperative imaging demonstrating precise details of real-time cell movements. Research and clinical scenarios may benefit from this novel intraoperative in vivo microscopic fluorescence imaging modality.

Macroscopic and microscopic imaging modalities for diagnosis and monitoring of urogenital schistosomiasis.

Urogenital schistosomiasis remains a major global challenge. Optimal management of this infection depends upon imaging-based assessment of sequelae. Although established imaging modalities such as ultrasonography, plain radiography, magnetic resonance imaging (MRI), narrow band imaging, and computerized tomography (CT) have been used to determine tissue involvement by urogenital schistosomiasis, newer refinements in associated technologies may lead to improvements in patient care. Moreover, application of investigational imaging methods such as confocal laser endomicroscopy and two-photon microscopy in animal models of urogenital schistosomiasis are likely to contribute to our understanding of this infection's pathogenesis. This review discusses prior use of imaging in patients with urogenital schistosomiasis and experimentally infected animals, the advantages and limitations of these modalities, the latest radiologic developments relevant to this infection, and a proposed future diagnostic standard of care for management of afflicted patients.

Deep Learning for Virtual Histological Staining of Bright-Field Microscopic Images of Unlabeled Carotid Artery Tissue

PurposeHistological analysis of artery tissue samples is a widely used method for diagnosis and quantification of cardiovascular diseases. However, the variable and labor-intensive tissue staining procedures hinder efficient and informative histological image analysis.ProceduresIn this study, we developed a deep learning-based method to transfer bright-field microscopic images of unlabeled tissue sections into equivalent bright-field images of histologically stained versions of the same samples. We trained a convolutional neural network to build maps between the unstained images and histologically stained images using a conditional generative adversarial network model.ResultsThe results of a blind evaluation by board-certified pathologists illustrate that the virtual staining and standard histological staining images of rat carotid artery tissue sections and those involving different types of stains showed no major differences. Quantification of virtual and histological H&E staining in carotid artery tissue sections showed that the relative errors of intima thickness, intima area, and media area were lower than 1.6 %, 5.6 %, and 12.7 %, respectively. The training time of deep learning network was 12.857 h with 1800 training patches and 200 epoches.ConclusionsThis virtual staining method significantly mitigates the typically laborious and time-consuming histological staining procedures and could be augmented with other label-free microscopic imaging modalities.

Quantitative Spatial Analysis of Metabolic Heterogeneity Across in vivo and in vitro Tumor Models.

Metabolic preferences of tumor cells vary within a single tumor, contributing to tumor heterogeneity, drug resistance, and patient relapse. However, the relationship between tumor treatment response and metabolically distinct tumor cell populations is not well-understood. Here, a quantitative approach was developed to characterize spatial patterns of metabolic heterogeneity in tumor cell populations within in vivo xenografts and 3D in vitro cultures (i.e., organoids) of head and neck cancer. Label-free images of cell metabolism were acquired using two-photon fluorescence lifetime microscopy of the metabolic co-enzymes NAD(P)H and FAD. Previous studies have shown that NAD(P)H mean fluorescence lifetimes can identify metabolically distinct cells with varying drug response. Thus, density-based clustering of the NAD(P)H mean fluorescence lifetime was used to identify metabolic sub-populations of cells, then assessed in control, cetuximab-, cisplatin-, and combination-treated xenografts 13 days post-treatment and organoids 24 h post-treatment. Proximity analysis of these metabolically distinct cells was designed to quantify differences in spatial patterns between treatment groups and between xenografts and organoids. Multivariate spatial autocorrelation and principal components analyses of all autofluorescence intensity and lifetime variables were developed to further improve separation between cell sub-populations. Spatial principal components analysis and Z-score calculations of autofluorescence and spatial distribution variables also visualized differences between models. This analysis captures spatial distributions of tumor cell sub-populations influenced by treatment conditions and model-specific environments. Overall, this novel spatial analysis could provide new insights into tumor growth, treatment resistance, and more effective drug treatments across a range of microscopic imaging modalities (e.g., immunofluorescence, imaging mass spectrometry).

Quadrature demodulation in line-scan focal modulation microscopy for imaging three-dimensional zebrafish neural structure.

Visualizing biological processes in neuroscience requires in vivo functional imaging at single-neuron resolution, high image acquisition speed and strong optical sectioning ability. However, due to light scattering of in tissue, very often conventional wide-field fluorescence microscopes are unable to resolve cells in the presence of a strong out-of-focus background. Line-scan focal modulation microscopy enables high temporal resolution and good optical sectioning ability at the same time. Here we demonstrate a quadrature demodulation method to extract the focal information with an extended frequency bandwidth and therefore higher spatial resolution. The performance of the demodulation scheme in line-scan focal modulation microscope has been evaluated by performing imaging experiments with fluorescence beads and zebrafish neural structure. Reduced background, reduced artifacts and more detailed morphological information are evident in the obtained images.

Super-Resolution Scanning Transmission X-Ray Imaging Using Single Biconcave Parabolic Refractive Lens Array

A new super resolution imaging technique which potentially enables sub-µm spatial resolution, using a detector of pixels much larger than the spatial resolution, is proposed. The method utilizes sample scanning through a large number of identical X-ray microprobes periodically spaced (the period corresponds to a multiple of the pixel size), which reduces drastically the scanning time. The information about the sample illuminated by the microprobes is stored by large detector pixels. Using these data and sample position information, a super-resolution image reconstruction is performed. With a one-dimensional (1D) high aspect ratio nickel single lens array designed for theoretically expected sub-µm microprobes at 17 keV and fabricated by deep X-ray lithography and electroforming technique, 2 µm X-ray microprobes with a period of 10 µm were achieved. We performed a first experiment at KARA synchrotron facility, and it was demonstrated that the smallest structure of a test pattern with a size of 1.5 µm could be easily resolved by using images generated from a detector having a pixel size of 10.4 µm. This new approach has a great potential for providing a new microscopic imaging modality with a large field of view and short scan time.

Focus on lasers, imaging, and microscopy

Focus on lasers, imaging, and microscopy

Awake Mouse Imaging: From Two-Photon Microscopy to Blood Oxygen Level–Dependent Functional Magnetic Resonance Imaging

BackgroundFunctional magnetic resonance imaging (fMRI) in awake behaving mice is well positioned to bridge the detailed cellular-level view of brain activity, which has become available owing to recent advances in microscopic optical imaging and genetics, to the macroscopic scale of human noninvasive observables. However, though microscopic (e.g., two-photon imaging) studies in behaving mice have become a reality in many laboratories, awake mouse fMRI remains a challenge. Owing to variability in behavior among animals, performing all types of measurements within the same subject is highly desirable and can lead to higher scientific rigor. MethodsWe demonstrated blood oxygenation level–dependent fMRI in awake mice implanted with long-term cranial windows that allowed optical access for microscopic imaging modalities and optogenetic stimulation. We started with two-photon imaging of single-vessel diameter changes (n = 1). Next, we implemented intrinsic optical imaging of blood oxygenation and flow combined with laser speckle imaging of blood flow obtaining a mesoscopic picture of the hemodynamic response (n = 16). Then we obtained corresponding blood oxygenation level–dependent fMRI data (n = 5). All measurements could be performed in the same mice in response to identical sensory and optogenetic stimuli. ResultsThe cranial window did not deteriorate the quality of fMRI and allowed alternation between imaging modalities in each subject. ConclusionsThis report provides a proof of feasibility for multiscale imaging approaches in awake mice. In the future, this protocol could be extended to include complex cognitive behaviors translatable to humans, such as sensory discrimination or attention.

Visualization of internal in situ cell structure by atomic force microscopy.

Light and electron microscopy have been used to study cell structure for many years, but atomic force microscopy is a more recent technique used to analyze cells, mainly due to the absence of techniques to prepare the samples. Isolated molecules or organelles, whole cells, and to a lesser extent in situ cell structure have been observed by different atomic force microscopy imaging modes. Here, we review efforts intended to analyze in situ the cell structures using approaches involving imaging of the surface of semithin sections of samples embedded in resin and sections prepared with an ultramicrotome. The results of such studies are discussed in relation to their implications to analyze the fine structure of organelles at the nanoscale in situ at enhanced resolution compared to light microscopy.

Deep brain optical coherence tomography angiography in mice: in vivo, noninvasive imaging of hippocampal formation

The hippocampus is associated with memory and navigation, and the rodent hippocampus provides a useful model system for studying neurophysiology such as neural plasticity. Vascular changes at this site are closely related to brain diseases, such as Alzheimer’s disease, dementia, and epilepsy. Vascular imaging around the hippocampus in mice may help to further elucidate the mechanisms underlying these diseases. Optical coherence tomography angiography (OCTA) is an emerging technology that can provide label-free blood flow information. As the hippocampus is a deep structure in the mouse brain, direct in vivo visualisation of the vascular network using OCTA and other microscopic imaging modalities has been challenging. Imaging of blood vessels in the hippocampus has been performed using multiphoton microscopy; however, labelling with fluorescence probes is necessary when using this technique. Here, we report the use of label-free and noninvasive microvascular imaging in the hippocampal formation of mice using a 1.7-μm swept-source OCT system. The imaging results demonstrate that the proposed system can visualise blood flow at different locations of the hippocampus corresponding with deep brain areas.

Learning to transfer microscopy image modalities

Phase Contrast and Differential Interference Contrast (DIC) microscopy are two popular noninvasive techniques for monitoring live cells. Each of these two imaging modalities has its own advantages and disadvantages to visualize specimens, so biologists need these two complementary modalities together to analyze specimens. In this paper, we propose a novel data-driven learning method capable of transferring microscopy images from one imaging modality to the other imaging modality, reflecting the characteristics of specimens from different perspectives. For example, given a Phase Contrast microscope, we can transfer its images to the corresponding DIC images without using DIC microscope, vice versa. The preliminary experiments demonstrate that the image transfer approach can provide biologists a computational way to switch between microscopy imaging modalities, so biologists can combine the advantages of different imaging modalities to better visualize and analyze specimens over time, without purchasing all types of microscopy imaging modalities or switching between imaging systems back-and-forth during time-lapse experiments.

A proof-of-concept study for developing integrated two-photon microscopic and magnetic resonance imaging modality at ultrahigh field of 16.4 tesla

Functional magnetic resonance imaging (fMRI) based on the blood oxygen level dependent (BOLD) contrast has gained a prominent position in neuroscience for imaging neuronal activity and studying effective brain connectivity under working state and functional connectivity at resting state. However, the fundamental questions in regards to fMRI technology: how the BOLD signal inferences the underlying microscopic neuronal activity and physiological changes and what is the ultimate specificity of fMRI for functional mapping of microcircuits, remain unanswered. The capability of simultaneous fMRI measurement and functional microscopic imaging in a live brain thus holds the key to link the microscopic and mesoscopic neural dynamics to the macroscopic brain activity at the central nervous system level. Here we report the first demonstration to integrate high-resolution two-photon fluorescence microscopy (TPM) with a 16.4 tesla MRI system, which proves the concept and feasibility for performing simultaneous high-resolution fMRI and TPM imaging at ultrahigh magnetic field.