ML Training Research Articles

Machine learning for nano-level defect detection in aligned random carbon nanotubes-reinforced electrically conductive nanocomposite

Machine learning allows fast nano-scale defect detection in polymer-impregnated aligned carbon nanotube (CNT) nanocomposites. Digital twins were populated by TEM-validated geometry; considered defects were flat cracks and close-to-spherical voids. Finite-element analysis of piezoresistive response was conducted by embedment of CNT network into matrix. Identification of a defect by change in CNT network piezoresistivity was challenged by: (1) randomness of CNTs’ shapes and placement, ML training happened on random realisations; (2) high strength of CNTs leading to the preservation of conductive paths along CNTs and changes only in conductivities of tunnelling contacts. “Artificial approximation“ was introduced to economise computer time multi-fold: ML was trained on cases with artificially degraded tunnelling conductivities within the defect. Three ML models: XGBoost, fully connected, and convolution neural networks were employed. All models managed the task for near-spherical voids, but performed poorly for flat cracks, due to the limited number of tunnelling contacts in crack volume. When trained on the mixed set of voids and cracks, both neural networks demonstrated the ability to learn the difference and detected even cracks, while XGBoost was not up to the challenge. By metrics, the convolutional neural network demonstrated the highest accuracy of predictions.

Efficient Phase Segmentation of Light-Optical Microscopy Images of Highly Complex Microstructures Using a Correlative Approach in Combination with Deep Learning Techniques

Reliable microstructure characterization is essential for establishing process–microstructure–property links and effective quality control. Traditional manual microstructure analysis often struggles with objectivity, reproducibility, and scalability, particularly in complex materials. Machine learning methods offer a promising alternative but are hindered by the challenge of assigning an accurate and consistent ground truth, especially for complex microstructures. This paper introduces a methodology that uses correlative microscopy—combining light optical microscopy, scanning electron microscopy, and electron backscatter diffraction (EBSD)—to create objective, reproducible pixel-by-pixel annotations for ML training. In a semi-automated manner, EBSD-based annotations are employed to generate an objective ground truth mask for training a semantic segmentation model for quantifying simple light optical micrographs. The training masks are directly derived from raw EBSD data using modern deep learning methods. By using EBSD-based annotations, which incorporate crystallographic and misorientation data, the correctness and objectivity of the training mask creation can be assured. The final approach is capable of reproducibly and objectively differentiating bainite and martensite in optical micrographs of complex quenched steels. Through the reduction in the microstructural evaluation to light optical micrographs as the simplest and most widely used method, this way of quantifying microstructures is characterized by high efficiency as well as good scalability.

Data and resource aware incremental ML training in support of pervasive applications

Data and resource aware incremental ML training in support of pervasive applications

You Can Have Your Cake and Eat It Too: Ensuring Practical Robustness and Privacy in Federated Learning

Inherently, federated learning (FL) robustness is very challenging to guarantee, especially when trying to maintain privacy. Compared to standard ML settings, FL's open training process allows for malicious clients to easily go under the radar. Alongside this, malicious clients can easily collude to attack the training process continuously, and without detection. FL models are also still susceptible to attacks on standard ML training procedures. This massive attack surface makes balancing the tradeoff between utility, practicality, robustness, and privacy extremely challenging. While there have been proposed defenses to attacks using popular privacy-preserving primitives, such as fully homomorphic encryption, they often face trouble balancing an all-important question that is present in all privacy-preserving systems: How much utility and practicality am I willing to give up to ensure privacy and robustness? In this work, we discuss a practical approach towards secure and robust FL and the challenges that face this field of emerging research.

Label-free SERS-ML detection of cocaine trace in human blood plasma

The widespread consumption of cocaine poses a significant threat to modern society. The most effective way to combat this problem is to control the distribution of cocaine, based on its accurate and sensitive detection. Here, we proposed the detection of cocaine in human blood plasma using a combination of surface enhanced Raman spectroscopy and machine learning (SERS-ML). To demonstrate the efficacy of our proposed approach, cocaine was added into blood plasma at various concentrations and drop-deposited onto a specially prepared disposable SERS substrate. SERS substrates were created by deposition of metal nanoclusters on electrospun polymer nanofibers. Subsequently, SERS spectra were measured and as could be expected, the manual distinguishing of cocaine from the spectra proved unfeasible, as its signal was masked by the background signal from blood plasma molecules. To overcome this issue, a database of SERS spectra of cocaine in blood plasma was collected and used for ML training and validation. After training, the reliability of proposed approach was tested on independently prepared samples, with unknown for SERS-ML cocaine presence or absence. As a result, the possibility of rapid determination of cocaine in blood plasma with a probability above 99.5% for cocaine concentrations up to 10-14 M was confirmed. Therefore, it is evident that the proposed approach has the ability to detect trace amounts of cocaine in bioliquids in an express and simple manner.

786: Utilizing Pareto optimization to refine ML training data in prostate radiotherapy planning

786: Utilizing Pareto optimization to refine ML training data in prostate radiotherapy planning

Incorporating Serverless Computing into P2P Networks for ML Training: In-Database Tasks and Their Scalability Implications (Student Abstract)

Distributed ML addresses challenges from increasing data and model complexities. Peer to peer (P2P) networks in distributed ML offer scalability and fault tolerance. However, they also encounter challenges related to resource consumption, and communication overhead as the number of participating peers grows. This research introduces a novel architecture that combines serverless computing with P2P networks for distributed training. Serverless computing enhances this model with parallel processing and cost effective scalability, suitable for resource-intensive tasks. Preliminary results show that peers can offload expensive computational tasks to serverless platforms. However, their inherent statelessness necessitates strong communication methods, suggesting a pivotal role for databases. To this end, we have enhanced an in memory database to support ML training tasks.

Energy and Emissions of Machine Learning on Smartphones vs. the Cloud

A Google case study finds ML training in the cloud can reduce CO 2 e emissions up to 100×.

VeriTrain: Validating MLaaS Training Efforts via Anomaly Detection

Machine learning as a service (MLaaS) offers users the benefit of training state-of-the-art neural network models on fast hardware with low costs. However, it also brings security concerns since the user does not fully trust the cloud. To prove to the user that the ML training results are legitimate, existing approaches mainly adopt cryptographic techniques such as secure multi-party computation, which incur large overheads. In this paper, we model the problem of verifying ML training efforts as an anomaly detection problem. We design a verification system, dubbed <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> , which combines unsupervised anomaly detection approaches and hypothesis testing techniques to verify the legitimacy of training efforts on the MLaaS cloud. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> is run inside trusted execution environments (TEEs) on the same cloud machine to ensure the integrity of its execution. We consider a threat model where the cloud model trainer is a lazy attacker and tries to fool <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> with minimum training effort. We perform extensive evaluations on multiple neural network models and datasets, which shows that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> performs well in detecting parameter updates crafted by the attacker. We also implement <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> with Intel SGX and show that it only incurs moderate overheads.

TCR-H: explainable machine learning prediction of T-cell receptor epitope binding on unseen datasets.

Artificial-intelligence and machine-learning (AI/ML) approaches to predicting T-cell receptor (TCR)-epitope specificity achieve high performance metrics on test datasets which include sequences that are also part of the training set but fail to generalize to test sets consisting of epitopes and TCRs that are absent from the training set, i.e., are 'unseen' during training of the ML model. We present TCR-H, a supervised classification Support Vector Machines model using physicochemical features trained on the largest dataset available to date using only experimentally validated non-binders as negative datapoints. TCR-H exhibits an area under the curve of the receiver-operator characteristic (AUC of ROC) of 0.87 for epitope 'hard splitting' (i.e., on test sets with all epitopes unseen during ML training), 0.92 for TCR hard splitting and 0.89 for 'strict splitting' in which neither the epitopes nor the TCRs in the test set are seen in the training data. Furthermore, we employ the SHAP (Shapley additive explanations) eXplainable AI (XAI) method for post hoc interrogation to interpret the models trained with different hard splits, shedding light on the key physiochemical features driving model predictions. TCR-H thus represents a significant step towards general applicability and explainability of epitope:TCR specificity prediction.

Machine learning approach for the detection of vitamin D level: a comparative study

BackgroundAfter the World Health Organization declared the COVID-19 pandemic, the role of Vitamin D has become even more critical for people worldwide. The most accurate way to define vitamin D level is 25-hydroxy vitamin D(25-OH-D) blood test. However, this blood test is not always feasible. Most data sets used in health science research usually contain highly correlated features, which is referred to as multicollinearity problem. This problem can lead to misleading results and overfitting problems in the ML training process. Therefore, the proposed study aims to determine a clinically acceptable ML model for the detection of the vitamin D status of the North Cyprus adult participants accurately, without the need to determine 25-OH-D level, taking into account the multicollinearity problem.MethodThe study was conducted with 481 observations who applied voluntarily to Internal Medicine Department at NEU Hospital. The classification performance of four conventional supervised ML models, namely, Ordinal logistic regression(OLR), Elastic-net ordinal regression(ENOR), Support Vector Machine(SVM), and Random Forest (RF) was compared. The comparative analysis is performed regarding the model's sensitivity to the participant’s metabolic syndrome(MtS)'positive status, hyper-parameter tuning, sensitivities to the size of training data, and the classification performance of the models.ResultsDue to the presence of multicollinearity, the findings showed that the performance of the SVM(RBF) is obviously negatively affected when the test is examined. Moreover, it can be obviously detected that RF is more robust than other models when the variations in the size of training data are examined. This experiment's result showed that the selected RF and ENOR showed better performances than the other two models when the size of training samples was reduced. Since the multicollinearity is more severe in the small samples, it can be concluded that RF and ENOR are not affected by the presence of the multicollinearity problem. The comparative analysis revealed that the RF classifier performed better and was more robust than the other proposed models in terms of accuracy (0.94), specificity (0.96), sensitivity or recall (0.94), precision (0.95), F1-score (0.95), and Cohen's kappa (0.90).ConclusionIt is evident that the RF achieved better than the SVM(RBF), ENOR, and OLR. These comparison findings will be applied to develop a Vitamin D level intelligent detection system for being used in routine clinical, biochemical tests, and lifestyle characteristics of individuals to decrease the cost and time of vitamin D level detection.

Data Locality for Large scale AI / ML training

Data Locality for Large scale AI / ML training

DOLL

Most large-scale ML implementations scale to large amounts of data by utilizing multiple servers or virtual machines (VMs) that iteratively compute model updates on local data that are periodically synchronized. Due to the complexity of managing the resulting computing infrastructure, many companies run their ML jobs on external cloud providers' servers. However, cloud resources can be expensive, particularly for large ML jobs with long runtimes. A particularly popular method to limit the costs of training ML jobs is to utilize preemptible cloud instances. These may be interrupted at the cloud provider's discretion, but they are significantly (up to 90%) cheaper than conventional on-demand instances. Most studies of these ML methods, however, assume the availability of large datasets at training time. In practice, training data may arrive at irregular intervals and models may be trained online as new data samples arrive, e.g., when monitoring data from IoT sensors. While some software frameworks like Apache Kafka can feed online data arrivals to ML algorithms, they provide little insight into the resulting costs of ML training. We extend prior work on provisioning preemptible instances to analyze available pools of data in order to run online ML on incoming datastreams, which presents new challenges due to the need to carefully handle data arrivals. We design, analyze, and optimize DOLL, which to the best of our knowledge is the first system that provides provable performance guarantees for Distributed OnLine Learning over preemptible instances. Research Challenges and Our Contributions: When pools of data are readily available, the bottleneck to distributed ML training often lies in the time required for each VM to compute its model updates. In our scenario, however, the arrival rate of incoming data may also bottleneck data processing. An intuitive strategy would then be for each VM to process each data point as it arrives. However, since arrivals at different VMs may not be coordinated, synchronizing the model parameters at each VM between data arrivals may introduce additional delays, while asynchronous SGD methods can lead to slow convergence [1]. DOLL uses a batching and grouping process to limit the synchronization delay, which naturally realizes traditional mini-batch SGD so as to provide provable model convergence guarantees. Handling online data arrivals becomes particularly challenging when we use preemptible instances to compute model updates. Existing methods utilizing preemptible instances for ML jobs largely focus on mitigating training interruptions [2] and their effects on model convergence [3]. When used on datastreams, we face an additional challenge of interruptions pausing the data arrival process, which impedes the rate at which we can compute model updates and thus model convergence. Thus, one should ensure that preemptions do not happen "too often," e.g., by computing some updates on on-demand instances. Our work is the first to optimize the number of preemptible VMs used and demonstrate that we can meet ML convergence guarantees.

An adaptive synaptic array using Fowler\u2013Nordheim dynamic analog memory

In this paper we present an adaptive synaptic array that can be used to improve the energy-efficiency of training machine learning (ML) systems. The synaptic array comprises of an ensemble of analog memory elements, each of which is a micro-scale dynamical system in its own right, storing information in its temporal state trajectory. The state trajectories are then modulated by a system level learning algorithm such that the ensemble trajectory is guided towards the optimal solution. We show that the extrinsic energy required for state trajectory modulation can be matched to the dynamics of neural network learning which leads to a significant reduction in energy-dissipated for memory updates during ML training. Thus, the proposed synapse array could have significant implications in addressing the energy-efficiency imbalance between the training and the inference phases observed in artificial intelligence (AI) systems.

Multilingual speech recognition for GlobalPhone languages

In this paper, we present our investigations towards the development of multilingual automatic speech recognition (ML ASR) systems using the GlobalPhone database. In addition to GlobalPhone, we have included 4 Ethiopian languages (Amharic, Oromo, Tigrigna and Wolaytta), as well as Uyghur and English in our investigation. In order to see the impact of language relatedness in ML ASR training, we have analyzed both phonetic overlap and morphological complexity of the languages. Deep Neural Network based ML ASR systems have been developed using ML mix, transfer and multitask learning approaches. Relative word error rate (WER) reductions up to 33.21% have been achieved as a result of using resources of other languages in ML acoustic model training. Our experimental results show that languages with small amounts of monolingual training data benefit a lot from ML training. Moreover, using phonetically related languages in ML training is more beneficiary than using phonetically less related languages. It seems that the nature of the corpus (single or mixed domain, noisy or noise free, etc.) has also an impact in ML training although it is not as important as the phonetic relatedness of the languages.

Machine learning-based diagnosis and risk factor analysis of cardiocerebrovascular disease based on KNHANES

The prevalence of cardiocerebrovascular disease (CVD) is continuously increasing, and it is the leading cause of human death. Since it is difficult for physicians to screen thousands of people, high-accuracy and interpretable methods need to be presented. We developed four machine learning-based CVD classifiers (i.e., multi-layer perceptron, support vector machine, random forest, and light gradient boosting) based on the Korea National Health and Nutrition Examination Survey. We resampled and rebalanced KNHANES data using complex sampling weights such that the rebalanced dataset mimics a uniformly sampled dataset from overall population. For clear risk factor analysis, we removed multicollinearity and CVD-irrelevant variables using VIF-based filtering and the Boruta algorithm. We applied synthetic minority oversampling technique and random undersampling before ML training. We demonstrated that the proposed classifiers achieved excellent performance with AUCs over 0.853. Using Shapley value-based risk factor analysis, we identified that the most significant risk factors of CVD were age, sex, and the prevalence of hypertension. Additionally, we identified that age, hypertension, and BMI were positively correlated with CVD prevalence, while sex (female), alcohol consumption and, monthly income were negative. The results showed that the feature selection and the class balancing technique effectively improve the interpretability of models.

Toward Excellence of Electrocatalyst Design by Emerging Descriptor‐Oriented Machine Learning

AbstractMachine learning (ML) is emerging as a powerful tool for identifying quantitative structure–activity relationships to accelerate electrocatalyst design by learning from historic data without explicit programming. The algorithms, data/database, and descriptors are usually the decisive factors for ML and the descriptors play a pivotal role for electrocatalysis as they contain the essence of catalysis from the physicochemical nature. Despite the considerable research efforts regarding electrocatalyst design with ML, the lack of universal selection tactics for descriptors bridging the gap between structures and activity impedes its wider application. A timely summary of the application of ML in electrocatalyst design helps to deepen the understanding of the nature of descriptors and improve the application scope and design efficiency. This review summarizes the geometrical, electronic, and activity descriptors used as input for ML training and predicting to reveal the general rules for their application in the design of electrocatalysts. In response to the challenges of hydrogen evolution reaction, oxygen evolution reaction, oxygen reduction reaction, CO2 reduction reaction, and nitrogen reduction reaction, the ML application in these areas is tracked for the progress and prospective changes. Additionally, the potential application of the automated design and discovery are discussed for the other well‐known electrocatalytic processes.

Prediction of Cloud Fractional Cover Using Machine Learning

Climate change is stated as one of the largest issues of our time, resulting in many unwanted effects on life on earth. Cloud fractional cover (CFC), the portion of the sky covered by clouds, might affect global warming and different other aspects of human society such as agriculture and solar energy production. It is therefore important to improve the projection of future CFC, which is usually projected using numerical climate methods. In this paper, we explore the potential of using machine learning as part of a statistical downscaling framework to project future CFC. We are not aware of any other research that has explored this. We evaluated the potential of two different methods, a convolutional long short-term memory model (ConvLSTM) and a multiple regression equation, to predict CFC from other environmental variables. The predictions were associated with much uncertainty indicating that there might not be much information in the environmental variables used in the study to predict CFC. Overall the regression equation performed the best, but the ConvLSTM was the better performing model along some coastal and mountain areas. All aspects of the research analyses are explained including data preparation, model development, ML training, performance evaluation and visualization.

Improved prediction of clay soil expansion using machine learning algorithms and meta-heuristic dichotomous ensemble classifiers

Soil swelling-related disaster is considered as one of the most devastating geo-hazards in modern history. Hence, proper determination of a soil’s ability to expand is very vital for achieving a secure and safe ground for infrastructures. Accordingly, this study has provided a novel and intelligent approach that enables an improved estimation of swelling by using kernelised machines (Bayesian linear regression (BLR) & bayes point machine (BPM) support vector machine (SVM) and deep-support vector machine (D-SVM)); (multiple linear regressor (REG), logistic regressor (LR) and artificial neural network (ANN)), tree-based algorithms such as decision forest (RDF) & boosted trees (BDT). Also, and for the first time, meta-heuristic classifiers incorporating the techniques of voting (VE) and stacking (SE) were utilised. Different independent scenarios of explanatory features’ combination that influence soil behaviour in swelling were investigated. Preliminary results indicated BLR as possessing the highest amount of deviation from the predictor variable (the actual swell-strain). REG and BLR performed slightly better than ANN while the meta-heuristic learners (VE and SE) produced the best overall performance (greatest R2 value of 0.94 and RMSE of 0.06% exhibited by VE). CEC, plasticity index and moisture content were the features considered to have the highest level of importance. Kernelized binary classifiers (SVM, D-SVM and BPM) gave better accuracy (average accuracy and recall rate of 0.93 and 0.60) compared to ANN, LR and RDF. Sensitivity-driven diagnostic test indicated that the meta-heuristic models’ best performance occurred when ML training was conducted using k-fold validation technique. Finally, it is recommended that the concepts developed herein be deployed during the preliminary phases of a geotechnical or geological site characterisation by using the best performing meta-heuristic models via their background coding resource.

Explaining multivariate molecular diagnostic tests via Shapley values

BackgroundMachine learning (ML) can be an effective tool to extract information from attribute-rich molecular datasets for the generation of molecular diagnostic tests. However, the way in which the resulting scores or classifications are produced from the input data may not be transparent. Algorithmic explainability or interpretability has become a focus of ML research. Shapley values, first introduced in game theory, can provide explanations of the result generated from a specific set of input data by a complex ML algorithm.MethodsFor a multivariate molecular diagnostic test in clinical use (the VeriStrat® test), we calculate and discuss the interpretation of exact Shapley values. We also employ some standard approximation techniques for Shapley value computation (local interpretable model-agnostic explanation (LIME) and Shapley Additive Explanations (SHAP) based methods) and compare the results with exact Shapley values.ResultsExact Shapley values calculated for data collected from a cohort of 256 patients showed that the relative importance of attributes for test classification varied by sample. While all eight features used in the VeriStrat® test contributed equally to classification for some samples, other samples showed more complex patterns of attribute importance for classification generation. Exact Shapley values and Shapley-based interaction metrics were able to provide interpretable classification explanations at the sample or patient level, while patient subgroups could be defined by comparing Shapley value profiles between patients. LIME and SHAP approximation approaches, even those seeking to include correlations between attributes, produced results that were quantitatively and, in some cases qualitatively, different from the exact Shapley values.ConclusionsShapley values can be used to determine the relative importance of input attributes to the result generated by a multivariate molecular diagnostic test for an individual sample or patient. Patient subgroups defined by Shapley value profiles may motivate translational research. However, correlations inherent in molecular data and the typically small ML training sets available for molecular diagnostic test development may cause some approximation methods to produce approximate Shapley values that differ both qualitatively and quantitatively from exact Shapley values. Hence, caution is advised when using approximate methods to evaluate Shapley explanations of the results of molecular diagnostic tests.