Machine Learning-based Image Processing Research Articles

Image recognition-based petal arrangement estimation.

Flowers exhibit morphological diversity in the number and positional arrangement of their floral organs, such as petals. The petal arrangements of blooming flowers are represented by the overlap position relation between neighboring petals, an indicator of the floral developmental process; however, only specialists are capable of the petal arrangement identification. Therefore, we propose a method to support the estimation of the arrangement of the perianth organs, including petals and tepals, using image recognition techniques. The problem for realizing the method is that it is not possible to prepare a large number of image datasets: we cannot apply the latest machine learning based image processing methods, which require a large number of images. Therefore, we describe the tepal arrangement as a sequence of interior-exterior patterns of tepal overlap in the image, and estimate the tepal arrangement by matching the pattern with the known patterns. We also use methods that require less or no training data to implement the method: the fine-tuned YOLO v5 model for flower detection, GrubCut for flower segmentation, the Harris corner detector for tepal overlap detection, MAML-based interior-exterior estimation, and circular permutation matching for tepal arrangement estimation. Experimental results showed good accuracy when flower detection, segmentation, overlap location estimation, interior-exterior estimation, and circle permutation matching-based tepal arrangement estimation were evaluated independently. However, the accuracy decreased when they were integrated. Therefore, we developed a user interface for manual correction of the position of overlap estimation and interior-exterior pattern estimation, which ensures the quality of tepal arrangement estimation.

Railroad Sleeper Condition Monitoring Using Non-Contact in Motion Ultrasonic Ranging and Machine Learning-Based Image Processing.

An ultrasonic sonar-based ranging technique is introduced for measuring full-field railroad crosstie (sleeper) deflections. Tie deflection measurements have numerous applications, such as detecting degrading ballast support conditions and evaluating sleeper or track stiffness. The proposed technique utilizes an array of air-coupled ultrasonic transducers oriented parallel to the tie, capable of "in-motion" contactless inspections. The transducers are used in pulse-echo mode, and the distance between the transducer and the tie surface is computed by tracking the time-of-flight of the reflected waveforms from the tie surface. An adaptive, reference-based cross-correlation operation is used to compute the relative tie deflections. Multiple measurements along the width of the tie allow the measurement of twisting deformations and longitudinal deflections (3D deflections). Computer vision-based image classification techniques are also utilized for demarcating tie boundaries and tracking the spatial location of measurements along the direction of train movement. Results from field tests, conducted at walking speed at a BNSF train yard in San Diego, CA, with a loaded train car are presented. The tie deflection accuracy and repeatability analyses indicate the potential of the technique to extract full-field tie deflections in a non-contact manner. Further developments are needed to enable measurements at higher speeds.

Computational control of gene expression in individual yeast using reactive microscopy.

Biotechnology is rapidly improving, largely due to our ability to manipulate genetic material and efficiently measure biological processes. This talk will describe our advances in computational methods and experimental platforms to probe and control stochastic biological processes within individual cells under a microscope. I will describe a novel optogenetics-based approach in which cells are segmented and controlled in real time. An optogenetically regulated gene in each yeast cell under the microscope is modeled using the Chemical Master Equation and controlled independently using a digital micromirror device to send light to only that cell. I will briefly describe the open source python-based software we developed, called MicroMator, that enabled this work, and show how we used it for optogenetically controlled recombination and adaptive illumination of a photobleaching fluorophore. This work brings together stochastic modeling of gene regulation, machine learning based image processing, and automated microscopy to provide methods for understanding basic cellular behavior.

A New Automation System for Equipment Status and Efficiency Detection with Machine Learning Based Image Processing

Overall equipment effectiveness (OEE) is a necessary metric for monitoring and improving production processes in industry [Nakajima, 1988]. In order to make the OEE calculation properly, one needs to digitize the accurate data coming from the production line on the shopfloor, which is a challenge itself. The solution we present provides the accurate collection of production line status and OEE data required for monitoring and decision making. The problems found in existing solutions are overcome with advanced analytical methods such as video image processing and deep learning / machine learning. There are many solutions in the literature using traditional image processing approaches [Dalal, 2005] or machine learning methods [Felzenszwalb, 2010] to solve the object detection problem in the video. In recent years, deep learning methods have also yielded successful results in object detection [Girshick, 2014]. The innovative aspect of the solution we offer is that it is a system that learns patterns that may be different for each production line, and automatically predicts the production line status.

Intelligent framework for mineral segmentation and fluid-accessible surface area analysis in scanning electron microscopy

Imaging is powerful means of sample characterization where mineral abundances and surface areas can be quantified from mineral maps. Images are typically manually processed by domain experts, which is time-consuming, labor intensive, and subjective. Emerging techniques, such as machine learning based image processing, can potentially address these limitations and accelerate image processing but the performance of these models for accurate sample characterization and surface area analysis has not been completely evaluated. This study evaluates the potential of Random Forest and U-Net machine learning methods for mineral characterization and surface area analysis of six sandstone samples. Various input variable sets including filter extracted features, scanning electron microscopy (SEM) backscatter electron (BSE) images and SEM-energy dispersive x-ray spectroscopy images (EDS) images were considered. The evaluation was conducted by providing an intelligent framework that not only evaluates the accuracy of prediction for each pixel but also investigates the accuracy of predicted neighboring pixels. In addition, a new methodology is proposed to distinguish the more susceptible places to dissolution on the surface of a given mineral using a ranked mineral dissolution risk assessment map. The results showed both methods had an acceptable performance with the U-Net model outperforming Random Forest. Both methods showed an improved accuracy when filter extracted features were added to the dataset as input variables. The models’ performance predicting mineral abundances and accessibility agreed well with ground truth data for majority classes (e.g., quartz) compared to minority classes. Finally, the proposed methodology was shown to reliably identify the locations susceptible for dissolution indicated via proposed risk assessment maps. The intelligent segmentation and surface area analysis framework is a promising tool for accelerating the processing of SEM data and reactivity assessment of samples.

Machine learning-based image processing in materials science and engineering: A review

Machine learning (ML) is playing a great role in every sphere of life including the area of materials science and engineering. At times, the experimental and computational data together create a complex scenario for interpretation giving rise to the requirement of ML. The incorporation of ML in the development of material science has a promised future to accelerate the research through automatic data interpretation with the help of different models like classification, Regression, Clustering, etc. Therefore, it becomes quick and easy to analyze the data sets with greater accuracy as compared to the analysis done by human researchers. This paper gives an overview of the applications and applicability of Machine Learning being used in various areas of materials development. The various image processing techniques which can be applied in material science are discussed in detail. In addition to that, the paper gives detail as to how a model can be trained with a smaller number of datasets. Subsequently, the different data ecosystems which can be useful for data collection and preparation before selecting any Machine Learning model are highlighted. Furthermore, the various opportunities and challenges that material scientists have been facing in the development of new materials are also outlined for a better understanding of ML-based image processing in material science and engineering.

Membrane-Based In-Gel Loop-Mediated Isothermal Amplification (mgLAMP) System for SARS-CoV-2 Quantification in Environmental Waters.

Since the COVID-19 pandemic is expected to become endemic, quantification of severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) in ambient waters is critical for environmental surveillance and for early detection of outbreaks. Herein, we report the development of a membrane-based in-gel loop-mediated isothermal amplification (mgLAMP) system that is designed for the rapid point-of-use quantification of SARS-CoV-2 particles in environmental waters. The mgLAMP system integrates the viral concentration, in-assay viral lysis, and on-membrane hydrogel-based RT-LAMP quantification using enhanced fluorescence detection with a target-specific probe. With a sample-to-result time of less than 1 h, mgLAMP successfully detected SARS-CoV-2 below 0.96 copies/mL in Milli-Q water. In surface water, the lowest detected SARS-CoV-2 concentration was 93 copies/mL for mgLAMP, while the reverse transcription quantitative polymerase chain reaction (RT-qPCR) with optimal pretreatment was inhibited at 930 copies/mL. A 3D-printed portable device is designed to integrate heated incubation and fluorescence illumination for the simultaneous analysis of nine mgLAMP assays. Smartphone-based imaging and machine learning-based image processing are used for the interpretation of results. In this report, we demonstrate that mgLAMP is a promising method for large-scale environmental surveillance of SARS-CoV-2 without the need for specialized equipment, highly trained personnel, and labor-intensive procedures.

Quantitative plaque assessment by coronary computed tomography angiography: An up-to-date review

Abstract Coronary computed tomography angiography has an emerging role in the diagnostic workup of coronary artery disease. Due to its high sensitivity and negative predictive value, coronary computed tomography angiography can rule out obstructive coronary artery diseases and substitute invasive coronary angiography in many cases. In addition, coronary computed tomography angiography provides a unique information beyond stenosis grading as it can visualize atherosclerosis and quantify its extent. Qualitative and quantitative plaque assessment provides an incremental value in the prediction of future major adverse cardiac events. Moreover, determining adverse plaque features has a potential to identify advanced atherosclerosis and patients at increased risk of acute coronary syndrome. Nevertheless, challenges may emerge with the process of quantifying coronary plaques due to limited reproducibility, lack of automated, standardized and validated techniques. Therefore, reliable quantified data are scarce due to the various computed tomography scanners and software platforms and investigations with small sample sizes. Radiomics and machine learning-based image processing methods are relatively new in the field of cardiovascular plaque imaging. These techniques hold the promise to improve diagnostic performance, reproducibility and prognostic value of computed tomography based plaque assessment.

Multimodal imaging and machine learning to enhance microscope images of shale

A machine learning based image processing workflow is presented to enhance shale source rock microscopic images obtained using diverse imaging platforms. Images were acquired from a 30μm diameter cylindrical Vaca Muerta shale sample using both nondestructive Transmission X-ray Microscopy (TXM, alternately referred to as nano computed tomography) and destructive Focused Ion Beam-Scanning Electron Microscopy (FIB-SEM). Output cross-sectional images from each modality were aligned using a combination of manual and automated registration techniques to create a registered image dataset. We then apply this dataset for two image processing tasks: prediction of image cross sections with SEM-like resolution from nondestructive TXM data and repair of charged region artifacts (localized accumulation of electrons) within SEM images. The image processing algorithms for both tasks use deep learning models, specifically image-to-image Convolutional Neural Networks (CNNs) and conditional Generative Adversarial Networks (cGANs). In the image enhancement tasks, we are able to achieve significant qualitative and quantitative improvement in TXM images. The best model reaches an average Peak Signal to Noise Ratio (PSNR) of 15.8 dB. Conditioning on TXM data is also shown to reduce artifacts from SEM charging, achieving an average PSNR of 25.8 dB. Our results suggest that properly trained and validated networks are capable of significant enhancement of images obtained using nondestructive techniques, thereby improving interpretation of two- and three-dimensional images while preserving samples for future use.

Abstract 314: Micropatterning supported method for cancer spheroid assays under flow and high throughput CAR-T cell potency assay with single cell resolution

Abstract Tumor spheroids, organoids, and cell culture models demand defined cell-cell and cell-matrix interactions, ideally with specific cell attachment on adhesion sites for multicellular spheroids or single cells. Our bioinert based micropatterning technology allows for the creation of adhesion spots while the surrounding passivated area is biologically inert, not interfering with the assay.We use this micro-patterning of adhesion ligands on highly passivated surfaces to generate homogenously distributed spheroids within a microfluidic channel system which can be perfused. Spheroids can either be directly generated by a simple injection of a large number of cells and a self-sorting process or by the addition of pre-formed spheroids, which will bind to the patterned adhesion sites. Perfusion of the spheroid chip enables the homogenous supply of all spheroids with nutrients and oxygen, and also the application of a defined shear stress, metabolite analysis together with toxicological screenings or even co-culture of multiple spheroid types. Applying a flow increases the overall spheroid growth, but condenses the cell aggregate, as well as influences the drug triggered protein expression response towards a more in vivo-like reaction.We also use this technique to generate arrays of homogenously distributed single cancer cells to evaluate cytotoxic T cell activity on a single cell level. By optical analysis combined with machine-learning based image processing of cell / spheroid arrays we obtained efficient label-free analysis of each cancer cell - T cell interaction over time. Citation Format: Elias Horn, Miriam Balles, Angelika J. Fischbeck, Jan Schwarz, Dane A. Thyssen, David W. Drell, Elfriede Noessner, Roman Zantl. Micropatterning supported method for cancer spheroid assays under flow and high throughput CAR-T cell potency assay with single cell resolution [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 314.

Machine learning based image processing technology application in bunch longitudinal phase information extraction

We report on the application of machine learning (ML) methods to extract longitudinal phase information such as parameters of the synchrotron damping oscillation. Parameters of the synchrotron damping oscillation are important for the evaluation of machine status and bunch stability. It is of concern to extract these parameters with high-speed and high-precision. The previous methods, such as multiparameter nonlinear fitting and table look-up, are slower and easily fall into local optimal solutions. Our approach based on ML-image processing consists of training a virtual diagnostic to predict parameters using the beam position monitor (BPM) electrical signal data as inputs. We find that when the noise of data is large, our ML-model can still get better results than other methods, an important step toward on-line multiparameter extraction from multidimensional raw data.

Simple agarose micro-confinement array and machine-learning-based classification for analyzing the patterned differentiation of mesenchymal stem cells

The geometrical confinement of small cell colonies gives differential cues to cells sitting at the periphery versus the core. To utilize this effect, for example to create spatially graded differentiation patterns of human mesenchymal stem cells (hMSCs) in vitro or to investigate underpinning mechanisms, the confinement needs to be robust for extended time periods. To create highly repeatable micro-fabricated structures for cellular patterning and high-throughput data mining, we employed here a simple casting method to fabricate more than 800 adhesive patches confined by agarose micro-walls. In addition, a machine learning based image processing software was developed (open code) to detect the differentiation patterns of the population of hMSCs automatically. Utilizing the agarose walls, the circular patterns of hMSCs were successfully maintained throughout 15 days of cell culture. After staining lipid droplets and alkaline phosphatase as the markers of adipogenic and osteogenic differentiation, respectively, the mega-pixels of RGB color images of hMSCs were processed by the software on a laptop PC within several minutes. The image analysis successfully showed that hMSCs sitting on the more central versus peripheral sections of the adhesive circles showed adipogenic versus osteogenic differentiation as reported previously, indicating the compatibility of patterned agarose walls to conventional microcontact printing. In addition, we found a considerable fraction of undifferentiated cells which are preferentially located at the peripheral part of the adhesive circles, even in differentiation-inducing culture media. In this study, we thus successfully demonstrated a simple framework for analyzing the patterned differentiation of hMSCs in confined microenvironments, which has a range of applications in biology, including stem cell biology.

Automated discrimination of dicentric and monocentric chromosomes by machine learning-based image processing.

Dose from radiation exposure can be estimated from dicentric chromosome (DC) frequencies in metaphase cells of peripheral blood lymphocytes. We automated DC detection by extracting features in Giemsa-stained metaphase chromosome images and classifying objects by machine learning (ML). DC detection involves (i) intensity thresholded segmentation of metaphase objects, (ii) chromosome separation by watershed transformation and elimination of inseparable chromosome clusters, fragments and staining debris using a morphological decision tree filter, (iii) determination of chromosome width and centreline, (iv) derivation of centromere candidates, and (v) distinction of DCs from monocentric chromosomes (MC) by ML. Centromere candidates are inferred from 14 image features input to a Support Vector Machine (SVM). Sixteen features derived from these candidates are then supplied to a Boosting classifier and a second SVM which determines whether a chromosome is either a DC or MC. The SVM was trained with 292 DCs and 3135 MCs, and then tested with cells exposed to either low (1 Gy) or high (2-4 Gy) radiation dose. Results were then compared with those of 3 experts. True positive rates (TPR) and positive predictive values (PPV) were determined for the tuning parameter, σ. At larger σ, PPV decreases and TPR increases. At high dose, for σ = 1.3, TPR = 0.52 and PPV = 0.83, while at σ = 1.6, the TPR = 0.65 and PPV = 0.72. At low dose and σ = 1.3, TPR = 0.67 and PPV = 0.26. The algorithm differentiates DCs from MCs, overlapped chromosomes and other objects with acceptable accuracy over a wide range of radiation exposures.