Microscopy Image Data Research Articles

Applying Machine Learning to Healthcare Operations Management: CNN-Based Model for Malaria Diagnosis.

The purpose of this study is to explore how machine learning technologies can improve healthcare operations management. A machine learning-based model to solve a specific medical problem is developed to achieve this research purpose. Specifically, this study presents an AI solution for malaria infection diagnosis by applying the CNN (convolutional neural network) algorithm. Based on malaria microscopy image data from the NIH National Library of Medicine, a total of 24,958 images were used for deep learning training, and 2600 images were selected for final testing of the proposed diagnostic architecture. The empirical results indicate that the CNN diagnostic model correctly classified most malaria-infected and non-infected cases with minimal misclassification, with performance metrics of precision (0.97), recall (0.99), and f1-score (0.98) for uninfected cells, and precision (0.99), recall (0.97), and f1-score (0.98) for parasite cells. The CNN diagnostic solution rapidly processed a large number of cases with a high reliable accuracy of 97.81%. The performance of this CNN model was further validated through the k-fold cross-validation test. These results suggest the advantage of machine learning-based diagnostic methods over conventional manual diagnostic methods in improving healthcare operational capabilities in terms of diagnostic quality, processing costs, lead time, and productivity. In addition, a machine learning diagnosis system is more likely to enhance the financial profitability of healthcare operations by reducing the risk of unnecessary medical disputes related to diagnostic errors. As an extension for future research, propositions with a research framework are presented to examine the impacts of machine learning on healthcare operations management for safety and quality of life in global communities.

Enhanced bare-bones particle swarm optimization based evolving deep neural networks

In this research, we propose a variant of the Bare-Bones Particle Swarm Optimization (BBPSO) algorithm for hyper-parameter selection and deep architecture generation for image, audio and video classification tasks. Since the search process of the original BBPSO model is guided by a single leader and the particles’ personal best experiences, there is a lack of interactions pertaining to the neighbouring elite solutions. To overcome this limitation, we propose a versatile search process for a modified BBPSO model that incorporates a number of effective components and operations. These include the neighbouring and global best signals, search actions with Cauchy/Levy scale factors, sub-dimension operations guided by the local and global elite solutions, and a Levy-driven local search mechanism. Moreover, root-finding algorithms are employed which use informative mathematical principles to estimate new root offspring for leader/particle enhancement. A reinforcement learning algorithm is subsequently used to identify the optimal sequential deployment of these numerical analysis methods to increase robustness. Several medical imaging data sets, i.e., ISIC 2017, PH2 and Dermofit skin lesion databases, the ALL-IDB2 microscopic blood image data set, the MURA musculoskeletal radiographic database, the CK + facial expression data set, as well as the Coswara respiratory audio data set and UCF101 video action data set, are employed for evaluation. The proposed BBPSO-optimized Convolutional Neural Network (CNN), bidirectional Long Short-Term Memory (BiLSTM) with attention mechanism, and CNN-BiLSTM models outperform those devised by other PSO and BBPSO variants, as well as state-of-the-art existing studies, significantly, for image, audio respiratory abnormality and realistic video action recognition.

Deterioration Effect of Sandstone Tensile Strength and Its Mesoscopic Mechanism under Dry-wet Cycles

The rock mass in the hydro-fluctuating zone of the reservoir bank slope is under dry-wet cycles for a long time, which will cause the deterioration of rock mass and induce geological disasters. In this study, a series of dry-wet cycle tests on the argillaceous quartz sandstone in the Three Gorges Reservoir area was carried out. Then, after different dry-wet cycles, the sandstone specimens were used to conduct the Brazilian splitting, scanning electron microscope, and 3D laser scanning tests. Herein, we provided detailed physical and microscopy image data to analyze the deterioration effect of tensile strength and mesostructure deterioration process of sandstone. With the increase of dry-wet cycles, the tensile strength of sandstone initially decreases rapidly, and then the decline rate tends to slow down. The evolution laws of fractal dimension and porosity are also significantly consistent with the deterioration of tensile strength. Moreover, further mesostructural analysis has revealed the repeated “absorption and swelling-dehydration and contraction” of clay minerals. This results in the breakage of framework mineral quartz and the expansion and connectivity of internal cracks, which ultimately deteriorates sandstone’s tensile strength.

Darfix - data analysis for dark-field X-ray microscopy.

A Python package for the analysis of dark-field X-ray microscopy (DFXM) and rocking curve imaging (RCI) data is presented. DFXM is a non-destructive diffraction imaging technique that provides three-dimensional maps of lattice strain and orientation. The darfix package enables fast processing and visualization of these data, providing the user with the essential tools to extract information from the acquired images in a fast and intuitive manner. These data processing and visualization tools can be either imported as library components or accessed through a graphical user interface as an Orange add-on. In the latter case, the different analysis modules can be easily chained to define computational workflows. Operations on larger-than-memory image sets are supported through the implementation of online versions of the data processing algorithms, effectively trading performance for feasibility when the computing resources are limited. The software can automatically extract the relevant instrument angle settings from the input files' metadata. The currently available input file format is EDF and in future releases HDF5 will be incorporated.

Interoperable slide microscopy viewer and annotation tool for imaging data science and computational pathology

The exchange of large and complex slide microscopy imaging data in biomedical research and pathology practice is impeded by a lack of data standardization and interoperability, which is detrimental to the reproducibility of scientific findings and clinical integration of technological innovations. We introduce Slim, an open-source, web-based slide microscopy viewer that implements the internationally accepted Digital Imaging and Communications in Medicine (DICOM) standard to achieve interoperability with a multitude of existing medical imaging systems. We showcase the capabilities of Slim as the slide microscopy viewer of the NCI Imaging Data Commons and demonstrate how the viewer enables interactive visualization of traditional brightfield microscopy and highly-multiplexed immunofluorescence microscopy images from The Cancer Genome Atlas and Human Tissue Atlas Network, respectively, using standard DICOMweb services. We further show how Slim enables the collection of standardized image annotations for the development or validation of machine learning models and the visual interpretation of model inference results in the form of segmentation masks, spatial heat maps, or image-derived measurements.

AtomVision: A Machine Vision Library for Atomistic Images.

Computer vision techniques have immense potential for materials design applications. In this work, we introduce an integrated and general-purpose AtomVision library that can be used to generate and curate microscopy image (such as scanning tunneling microscopy and scanning transmission electron microscopy) data sets and apply a variety of machine learning techniques. To demonstrate the applicability of this library, we (1) establish an atomistic image data set of about 10 000 materials with large structural and chemical diversity, (2) develop and compare convolutional and atomistic line graph neural network models to classify the Bravais lattices, (3) demonstrate the application of fully convolutional neural networks using U-Net architecture to pixelwise classify atom versus background, (4) use a generative adversarial network for super resolution, (5) curate an image data set on the basis of natural language processing using an open-access arXiv data set, and (6) integrate the computational framework with experimental microscopy images for Rh, Fe3O4, and SnS systems. The AtomVision library is available at https://github.com/usnistgov/atomvision.

EMDS-7: Environmental microorganism image dataset seventh version for multiple object detection evaluation.

Nowadays, the detection of environmental microorganism indicators is essential for us to assess the degree of pollution, but the traditional detection methods consume a lot of manpower and material resources. Therefore, it is necessary for us to make microbial data sets to be used in artificial intelligence. The Environmental Microorganism Image Dataset Seventh Version (EMDS-7) is a microscopic image data set that is applied in the field of multi-object detection of artificial intelligence. This method reduces the chemicals, manpower and equipment used in the process of detecting microorganisms. EMDS-7 including the original Environmental Microorganism (EM) images and the corresponding object labeling files in ".XML" format file. The EMDS-7 data set consists of 41 types of EMs, which has a total of 2,65 images and 13,216 labeled objects. The EMDS-7 database mainly focuses on the object detection. In order to prove the effectiveness of EMDS-7, we select the most commonly used deep learning methods (Faster-Region Convolutional Neural Network (Faster-RCNN), YOLOv3, YOLOv4, SSD, and RetinaNet) and evaluation indices for testing and evaluation. EMDS-7 is freely published for non-commercial purpose at: https://figshare.com/articles/dataset/EMDS-7_DataSet/16869571.

MO-SOD: Micro-Oxidation Small Object Detection Model for Oxygen-Free Copper Surfaces Based on Microscopic Imaging System.

Micro-oxidation is a fatal problem for some precision oxygen-free copper materials, and it is difficult to detect with the naked eyes. However, manual inspection using microscope equipment is expensive, subjective, and time-consuming. The automatic high-definition micrograph system equipped with micro-oxidation detection algorithm can detect more quickly, efficiently, and accurately. In this study, a micro-oxidation small objection detection model, MO-SOD, is proposed to detect the oxidation degree on oxygen-free copper surface based on microimaging system. This model is developed for rapid detection on the robot platform combined with high-definition microphotography system. The proposed MO-SOD model consists of three modules: small target feature extraction layer, key small object attention pyramid integration layer, and anchor-free decoupling detector. The small object feature extraction layer focuses on the local features of small object to improve the perception of micro-oxidation spots and also takes the global features into account to reduce the impact of noisy background on feature extraction. Key small object attention pyramid integration block couples key small object feature attention and pyramid to detect the micro-oxidation spots in the image. The performance of MO-SOD model is further improved by combining the anchor-free decoupling detector. In addition, the loss function is improved to combine CIOU loss and focal loss to achieve effective micro-oxidation detection. The MO-SOD model is trained and tested from three oxidation levels in an oxygen-free copper surface microscope image data set. The test results show that the average accuracy (mAP) of MO-SOD model is 82.96%, which is superior to other most advanced detectors.

MIDLS - Membrane detection in cryo-EM using deep level sets.

Studying membrane proteins (MP) with their membranes is more biologically relevant and allows for functional and structural studies to be performed in the same system. However, the presence of membranes in cryogenic-Electron Microscopy (cryo-EM) imaging data limits particle identification and alignment. The variability and strong contrast of membranes override the weaker particle signal. To address these limitations, researchers are forced to either extract MPs from their membrane before imaging, preventing functional studies; or digitally subtract membranes, requiring user intervention to detect membranes.

Super-resolved SIM imaging revealed hierarchical organization of the ryanodine receptor network in 3 dimensions in rabbit sinoatrial node cells.

The heartbeat is initiated by specialized sinoatrial node cells (SANCs). The rate of the heartbeat is regulated by Ca released from the sarcoplasmic reticulum (SR) via ryanodine receptors (RyRs) interacting with voltage and Ca-dependent membrane ion currents in SANCs. The released Ca takes the form of localized Ca releases (LCRs) that propagate for limited distances in wave-like manner. A computational model of SANCs, previously developed in our laboratory, simulates LCR synchronization mechanism as hierarchical clustering of surface RyRs based on confocal microscopy imaging data obtained in tangential sections of SANCs. The true detailed distribution of RyR clusters in 3D, however, remains unknown. Here, we applied a series of new computational approaches to experimental data obtained by super-resolved structured illumination microscopy in rabbit SANCs, in which RyRs were labeled by specific antibodies. Our approach consists of automated segmentation of RyR clusters with globally optimal detection and segmentation of objects with constant internal intensity distribution, followed by object-based colocalization analysis in 3D. Results are further refined with watershed transformations and computational geometry. Examinations of the images have revealed a hierarchical architecture of RyR network (positions and sizes of submembrane RyR clusters). After constructing a mathematical convex hull around the cell using RyR clusters as nodes, clusters were grouped based on their volume. The distribution of RyR clusters exhibited a wide range of sizes from 0.75 × 10−3 µm3 to 0.50 µm3 and cluster histograms followed a Poisson-like distribution. These new data have been incorporated in a new, realistic 3D model of SANC, in which heterogeneous RyR clusters facilitate LCR propagation. Impairment of the complex, hierarchical structure of RyR clusters may lead to sinus node dysfunction in aging and disease.

X-ray micro-tomographic imaging and modelling of saline ice properties in concrete frost salt scaling experiments

Frost salt scaling of concrete is related to cyclic freezing and melting of a few millimeter thick deicer solution on the surface of the concrete. It is almost absent when pure water is freezing and reaches a maximum at a so-called pessimum concentration that for NaCl is around 3%. Different mechanisms have been suggested to explain this pessimum and frost salt scaling in general, ranging from the transport of moisture and growth of ice within the pore space of concrete (“cryogenic suction”) to crack formation in the saline ice layer followed by spalling off the surface (“glue-spall”). Though in these theories the saline ice layer, that forms in concrete frost salt scaling experiments, plays a major role, so far little is known about its properties. We present a characterisation and an analysis of the microstructure of this saline ice layer by means of 3D X-ray microtomography. We found that the morphology of the saline ice is very similar to young, columnar sea ice, with lamellae of ice and brine oriented in the direction of freezing. On the basis of the microscopic 3D image data, we formulated percolation-based models of macroscopic properties (e.g., strength, thermal expansion coefficient, porosity metrics) relevant for different proposed frost salt scaling mechanisms. Model results and observations suggest that the ice growth velocity, direction and confinement, have a major impact on the pore structure of saline ice, thereby governing both mechanical and transport properties. These properties in turn are expected to affect proposed frost salt scaling mechanisms of concrete. The microstructure length scales in the ice-brine composite (lamellar spacing, pore width) are comparable to those for concrete (air void spacing and size), suggesting complex poro-mechanical interaction at the interface of concrete and saline ice. The results highlight the importance of studying saline ice properties to improve predictions of frost salt scaling processes.

Second Harmonic Generation Morphometry of Muscle Cytoarchitecture in Living Cells.

The architectural structure of cells is essential for the cells' function, which becomes especially apparent in the highly "structure functionally" tuned skeletal muscle cells. Here, structural changes in the microstructure can have a direct impact on performance parameters, such as isometric or tetanic force production. The microarchitecture of the actin-myosin lattice in muscle cells can be detected noninvasively in living cells and in 3D by using second harmonic generation (SHG) microscopy, forgoing the need to alter samples by introducing fluorescent probes into them. Here, we provide tools and step-by-step protocols to guide the processes of obtaining SHG microscopy image data from samples, as well as extracting characteristic values from the image data to quantify the cellular microarchitecture using characteristic patterns of myofibrillar lattice alignments.

Electron Tomography and Machine Learning for Understanding the Highly Ordered Structure of Leafhopper Brochosomes.

Insects known as leafhoppers (Hemiptera: Cicadellidae) produce hierarchically structured nanoparticles known as brochosomes that are exuded and applied to the insect cuticle, thereby providing camouflage and anti-wetting properties to aid insect survival. Although the physical properties of brochosomes are thought to depend on the leafhopper species, the structure-function relationships governing brochosome behavior are not fully understood. Brochosomes have complex hierarchical structures and morphological heterogeneity across species, due to which a multimodal characterization approach is required to effectively elucidate their nanoscale structure and properties. In this work, we study the structural and mechanical properties of brochosomes using a combination of atomic force microscopy (AFM), electron microscopy (EM), electron tomography, and machine learning (ML)-based quantification of large and complex scanning electron microscopy (SEM) image data sets. This suite of techniques allows for the characterization of internal and external brochosome structures, and ML-based image analysis methods of large data sets reveal correlations in the structure across several leafhopper species. Our results show that brochosomes are relatively rigid hollow spheres with characteristic dimensions and morphologies that depend on leafhopper species. Nanomechanical mapping AFM is used to determine a characteristic compression modulus for brochosomes on the order of 1-3 GPa, which is consistent with crystalline proteins. Overall, this work provides an improved understanding of the structural and mechanical properties of leafhopper brochosomes using a new set of ML-based image classification tools that can be broadly applied to nanostructured biological materials.

Characterizing Nanoparticle Mass Distributions Using Charge-Independent Nanoresonator Mass Spectrometry

Due to their unique size-dependent properties, nanoparticles (NPs) have many industrial and biomedical applications. Although NPs are generally characterized based on the size or morphological analysis, the mass of whole particles can be of interest as it represents the total amount of material in the particle regardless of shape, density, or elemental composition. In addition, the shape of nonspherical NPs presents a conceptual challenge, making them difficult to characterize in terms of size or morphological characteristics. Here, we used a novel nano-electro-mechanical sensor mass spectrometry (NEMS-MS) technology to characterize the mass distributions of various NPs. For standard spherical gold NPs, mass distributions covered the range from ∼5 to 250 MDa (8 to ∼415 attograms). Applying the density of gold (19.3 g/cm3) and assuming perfect sphericity, these mass measurements were used to compute the equivalent diameters of the NPs. The sizes determined agreed well with the transmission electron microscopy (TEM) imaging data, with deviations of ∼1.4%. Subsequently, we analyzed the mass distribution of ∼50 nm synthetic silicon dioxide particles, having determined their size by electron microscopy (SEM and TEM). Their estimated density was in line with the literature values derived from differential mobility analyzer and aerosol particle mass analyzer data. Finally, we examined the intact gold nanotetrapods and obtained a mass distribution revealing their controlled polydispersity. The presence of polyethylene glycol coating was also quantified and corroborated nuclear magnetic resonance observations. Our results demonstrate the potential of NEMS-MS-based measurements as an effective means to characterize NPs, whatever their composition, shape or density.

Cellular adaptations leading to coral fragment attachment on artificial substrates in Acropora millepora (Am-CAM)

Reproductive propagation by asexual fragmentation in the reef-building coral Acropora millepora depends on (1) successful attachment to the reef substrate through modification of soft tissues and (2) a permanent bond with skeletal encrustation. Despite decades of research examining asexual propagation in corals, the initial response, cellular reorganisation, and development leading to fragment substrate attachment via a newly formed skeleton has not been documented in its entirety. Here, we establish the first "coral attachment model" for this species ("Am-CAM") by developing novel methods that allow correlation of fluorescence and electron microscopy image data with in vivo microscopic time-lapse imagery. This multi-scale imaging approach identified three distinct phases involved in asexual propagation: (1) the contact response of the coral fragment when contact with the substrate, followed by (2) fragment stabilisation through anchoring by the soft tissue, and (3) formation of a "lappet-like appendage" structure leading to substrate bonding of the tissue for encrustation through the onset of skeletal calcification. In developing Am-CAM, we provide new biological insights that can enable reef researchers, managers and coral restoration practitioners to begin evaluating attachment effectiveness, which is needed to optimise species-substrate compatibility and achieve effective outplanting.

Decoding the Mechanisms of Phase Transitions from In Situ Microscopy Observations.

Analysis of the temperature- and stimulus-dependent imaging data toward elucidation of the physical transformations is an ubiquitous problem in multiple fields. Here, temperature-induced phase transition in BaTiO3 is explored using the machine learning analysis of domain morphologies visualized via variable-temperature scanning transmission electron microscopy (STEM) imaging data. This approach is based on the multivariate statistical analysis of the time or temperature dependence of the statistical descriptors of the system, derived in turn from the categorical classification of observed domain structures or projection on the continuous parameter space of the feature extraction-dimensionality reduction transform. The proposed workflow offers a powerful tool for the exploration of the dynamic data based on the statistics of image representation as a function of the external control variable to visualize the transformation pathways during phase transitions and chemical reactions. This can include the mesoscopic STEM data as demonstrated here, but also optical, chemical imaging, etc., data. It can further be extended to the higher dimensional spaces, for example, analysis of the combinatorial libraries of materials compositions.

A Hitchhiker's guide through the bio-image analysis software universe.

Modern research in the life sciences is unthinkable without computational methods for extracting, quantifying and visualising information derived from microscopy imaging data of biological samples. In the past decade, we observed a dramatic increase in available software packages for these purposes. As it is increasingly difficult to keep track of the number of available image analysis platforms, tool collections, components and emerging technologies, we provide a conservative overview of software that we use in daily routine and give insights into emerging new tools. We give guidance on which aspects to consider when choosing the platform that best suits the user's needs, including aspects such as image data type, skills of the team, infrastructure and community at the institute and availability of time and budget.

Human Blastocyst Components Detection Using Multiscale Aggregation Semantic Segmentation Network for Embryonic Analysis.

Infertility is one of the most important health concerns worldwide. It is characterized by not being successful of pregnancy after some periods of periodic unprotected sexual intercourse. In vitro fertilization (IVF) is an assisted reproduction technique that efficiently addresses infertility. IVF replaces the actual mode of reproduction through a manual procedure wherein embryos are cultivated in a controlled laboratory environment until they reach the blastocyst stage. The standard IVF procedure includes the transfer of one or two blastocysts from several blastocysts that are grown in a controlled environment. The morphometric properties of blastocysts with their compartments such as trophectoderm (TE), zona pellucida (ZP), inner cell mass (ICM), and blastocoel (BL), are analyzed through manual microscopic analysis to predict viability. Deep learning has been extensively used for medical diagnosis and analysis and can be a powerful tool to automate the morphological analysis of human blastocysts. However, the existing approaches are inaccurate and require extensive preprocessing and expensive architectures. Thus, to cope with the automatic detection of blastocyst components, this study proposed a novel multiscale aggregation semantic segmentation network (MASS-Net) that combined four different scales via depth-wise concatenation. The extensive use of depthwise separable convolutions resulted in a decrease in the number of trainable parameters. Further, the innovative multiscale design provided rich spatial information of different resolutions, thereby achieving good segmentation performance without a very deep architecture. MASS-Net utilized 2.06 million trainable parameters and accurately detects TE, ZP, ICM, and BL without using preprocessing stages. Moreover, it can provide a separate binary mask for each blastocyst component simultaneously, and these masks provide the structure of each component for embryonic analysis. Further, the proposed MASS-Net was evaluated using publicly available human blastocyst (microscopic) imaging data. The experimental results revealed that it can effectively detect TE, ZP, ICM, and BL with mean Jaccard indices of 79.08, 84.69, 85.88%, and 89.28%, respectively, for embryological analysis, which was higher than those of the state-of-the-art methods.

Retraction Note to: Computational intelligence on image classification methods for microscopic image data

Retraction Note to: Computational intelligence on image classification methods for microscopic image data

Abstract 481: Biophysical models of pattern formation in bone marrow endothelial networks

Abstract In this work we aim to understand how cellular aggregation and angiogenic pattern formation develop in bone marrow endothelial networks by integrating spatial-temporal mathematical models with theory regarding how cellular metabolism scales with cell number. Presently, efforts to model angiogenic pattern formation fail to accurately predict spatial characteristics often observed. Biological allometries, such as the scaling of metabolism to mass or cell number, have been hypothesized to result from natural selection to maximize how vascular networks fill space yet minimize internal transport distances and resistance to motion. Metabolic scaling theory argues that two guiding principles—optimization of motion and space-filling fractal distributions—describe a diversity of biological networks and predict how the geometry of these networks influences organismal metabolism. Recently, metabolic scaling theory has been applied to study tumor growth rates, offering a mechanistic link between tumor growth and vascular patterning. Remarkably, there exists two classes of spatial-temporal mathematical models where equivalent angiogenic pattern formation occurs. These two classes are determined primarily by either cellular interactions as proposed by Hillen and Painter, or biomechanical forces as proposed by Manoussaki and Murray. Examples of underlying mechanisms are chemotactic quorum-sensing and resource competition for the former, and advection-diffusion driven stresses and strains in the extra-cellular matrix for the latter. We compare these models by examining theoretical predictions for pattern forming criteria with numerical simulations. Furthermore, we identify experimentally testable predictions for angiogenic pattern formation related to the spatial configuration of the extra-cellular matrix. We test these predictions against in vitro confocal microscopy imaging data for bone marrow endothelial networks with experimental control over adhesion molecules and the resulting network structure. Specifically, we measure time-dependent geometric features in the highly reticulated endothelial networks related to steady-state equilibration, connectivity and space-filling. Example metrics are the scaling of network edge lengths and enclosed areas within the extra-cellular matrix that indicate changes to network morphology as adhesion molecules are present or absent. These metrics are informed by metabolic scaling theory and are predicted by the steady-state solutions of the underlying models. By conducting metric and model selection, this work identifies data-driven mechanisms for angiogenic pattern formation and quantifies normality in bone-marrow endothelial networks. This is an essential step for future work in quantifying and understanding abnormality in bone marrow endothelial networks. Citation Format: Alexander B. Brummer, Young-Woong Kim, Nadia Carlesso, Russell C. Rockne. Biophysical models of pattern formation in bone marrow endothelial networks [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 481.