Statistical Machine Learning Methods Research Articles

Single-cell RNA-seq data analysis based on directed graph neural network.

Single-cell RNA sequencing (scRNA-seq) data scale surges with high-throughput sequencing technology development. However, although single-cell data analysis is a powerful tool, various issues have been reported, such as sequencing sparsity and complex differential patterns in gene expression. Statistical or traditional machine learning methods are inefficient, and the accuracy needs to be improved. The methods based on deep learning can not directly process non-Euclidean spatial data, such as cell diagrams. In this study, we have developed graph autoencoders and graph attention network for scRNA-seq analysis based on a directed graph neural network named scDGAE. Directed graph neural networks cannot only retain the connection properties of the directed graph but also expand the receptive field of the convolution operation. Cosine similarity, median L1 distance, and root-mean-squared error are used to measure the gene imputation performance of different methods with scDGAE. Furthermore, adjusted mutual information, normalized mutual information, completeness score, and Silhouette coefficient score are used to measure the cell clustering performance of different methods with scDGAE. Experiment results show that the scDGAE model achieves promising performance in gene imputation and cell clustering prediction on four scRNA-seq data sets with gold-standard cell labels. Furthermore, it is a robust framework that can be applied to general scRNA-Seq analyses.

Multimodal deep learning methods enhance genomic prediction of wheat breeding.

While several statistical machine learning methods have been developed and studied for assessing the genomic prediction (GP) accuracy of unobserved phenotypes in plant breeding research, few methods have linked genomics and phenomics (imaging). Deep learning (DL) neural networks have been developed to increase the GP accuracy of unobserved phenotypes while simultaneously accounting for the complexity of genotype-environment interaction (GE); however, unlike conventional GP models, DL has not been investigated for when genomics is linked with phenomics. In this study we used 2 wheat data sets (DS1 and DS2) to compare a novel DL method with conventional GP models. Models fitted for DS1 were GBLUP, gradient boosting machine (GBM), support vector regression (SVR) and the DL method. Results indicated that for 1 year, DL provided better GP accuracy than results obtained by the other models. However, GP accuracy obtained for other years indicated that the GBLUP model was slightly superior to the DL. DS2 is comprised only of genomic data from wheat lines tested for 3 years, 2 environments (drought and irrigated) and 2-4 traits. DS2 results showed that when predicting the irrigated environment with the drought environment, DL had higher accuracy than the GBLUP model in all analyzed traits and years. When predicting drought environment with information on the irrigated environment, the DL model and GBLUP model had similar accuracy. The DL method used in this study is novel and presents a strong degree of generalization as several modules can potentially be incorporated and concatenated to produce an output for a multi-input data structure.

Structural health monitoring for corrosion induced thickness loss in marine plates subjected to random loads

In the highly corrosive environment where marine structures operate, the current industry practice is to overcompensate for corrosion induced thickness loss (CITL) and replace any parts that score below specified allowable limits during scheduled maintenance. Hence, there would be immediate benefits from the implementation of a Structural Health Monitoring (SHM) system that would allow for predictive, condition-based maintenance. The objective of this study is to investigate and assess the effectiveness of state-of-the-art statistical pattern recognition (SPR) and machine learning (ML) methods in association with alternative sensor grid architectures as an SHM scheme for detecting CITL, under highly variable operational conditions. For this purpose, a simple rectangular plate at different corrosion levels (uniform and pitting) was considered as a reference structural element, which was subjected to a stochastic pressure load. Strain response data were produced using Finite Element (FE) simulations and were treated under a probabilistic framework. Elements from detection theory and ML were taken under consideration in order to construct alternative detectors and assess their performance.

Latent dirichlet allocation for double clustering (LDA-DC): discovering patients phenotypes and cell populations within a single Bayesian framework

BackgroundCurrent clinical routines rely more and more on “omics” data such as flow cytometry data from host and microbiota. Cohorts variability in addition to patients’ heterogeneity and huge dimensions make it difficult to understand underlying structure of the data and decipher pathologies. Patients stratification and diagnostics from such complex data are extremely challenging. There is an acute need to develop novel statistical machine learning methods that are robust with respect to the data heterogeneity, efficient from the computational viewpoint, and can be understood by human experts.ResultsWe propose a novel approach to stratify cell-based observations within a single probabilistic framework, i.e., to extract meaningful phenotypes from both patients and cells data simultaneously. We define this problem as a double clustering problem that we tackle with the proposed approach. Our method is a practical extension of the Latent Dirichlet Allocation and is used for the Double Clustering task (LDA-DC). We first validate the method on artificial datasets, then we apply our method to two real problems of patients stratification based on cytometry and microbiota data. We observe that the LDA-DC returns clusters of patients and also clusters of cells related to patients’ conditions. We also construct a graphical representation of the results that can be easily understood by humans and are, therefore, of a big help for experts involved in pre-clinical research.

Neural Markers of shootig performance: An explore of EEG frequency features during aiming

The frequency-domain characteristics of electroencephalogram (EEG) during shooting used to evaluate and improve shooting performance has become a research hotspot in recent years. However, neural markers that can be effectively and stably used to characterize various shooting performances remain unclear. In this study, a real shooting experiment was designed to study the effective EEG neural markers for different shooting performances. EEG data were obtained during the aiming period,the frequency domain characteristics such as spectrum, power spectrum, and differential entropy corresponding to Delta wave, Theta wave, Alpha wave, Beta wave, and Gamma wave were extracted respectively. Classical statistical analysis and machine learning methods were used to investigate the effectiveness of the characteristics. The results revealed that the overall energy of the EEG signals was low and stable for the participants with excellent shooting performance. The differential entropy feature of the gamma band as a effective neural marker had achieved the best classification accuracy of 74.91% throughout the aiming period. The results verified that the brain state of shooters is stable and their brains exhibit higher cognitive information processing efficiency during accurate shootings. The feasibility of using EEG neural markers to evaluate shooting performance is a novel avenue for the selection and performance evaluation of high-performance shooters.

Neuronal Correlates of Task Irrelevant Distractions Enhance the Detection of Attention Deficit/Hyperactivity Disorder.

Early diagnosis and treatment can reduce the symptoms of Attention Deficit/Hyperactivity Disorder (ADHD) in children, but medical diagnosis is usually delayed. Hence, it is important to increase the efficiency of early diagnosis. Previous studies used behavioral and neuronal data during GO/NOGO task to help detect ADHD and the accuracy differed considerably from 53% to 92%, depending on the employed methods and the number of electroencephalogram (EEG) channels. It remains unclear whether data from a few EEG channels can still lead to a good accuracy of detecting ADHD. Here, we hypothesize that introducing distractions into a VR-based GO/NOGO task can augment the detection of ADHD using 6-channel EEG because children with ADHD are easily distracted. Forty-nine ADHD children and 32 typically developing children were recruited. We use a clinically applicable system with EEG to record data. Statistical analysis and machine learning methods were employed to analyze the data. The behavioral results revealed significant differences in task performance when there are distractions. The presence of distractions leads to EEG changes in both groups, indicating immaturity in inhibitory control. Importantly, the distractions additionally enhanced the between-group differences in NOGO α and γ power, reflecting insufficient inhibition in different neural networks for distraction suppression in the ADHD group. Machine learning methods further confirmed that distractions enhance the detection of ADHD with an accuracy of 85.45%. In conclusion, this system can assist in fast screenings for ADHD and the findings of neuronal correlates of distractions can help design therapeutic strategies.

Sentiment classification of annual reports in Slovak language using symbolic, subsymbolic and statistic methods

Background: Sentiment analysis as a tool for recognizing emotions from bank reports can serve to capture the intentions and opinions of corporations, change the emotional context during turbulent periods, and to verify the compliance of the context of the reports with banking conservatism. The purpose of the presented study is to describe the process of creating an own dataset from annual reports in the Slovak language with marked sentiment and the subsequent search for models capable of classifying the sentiment of reports as positive, negative, or neutral.Methods: Our methodology consists of a series of steps necessary for obtaining the text and its subsequent preprocessing to create a dataset. The study further deals with the iterative search for the best combination of hyperparameters for symbolic, subsymbolic, statistic and homogenous symbolic ensemble models in order to find the best classifying model.Results: The experimental results show that the subsymbolic methods are the most suitable for the classifying sentiment of Slovak annual reports. The proposed model can classify sentiment with 98% accuracy.Conclusion: The classification of sentiment in inflectional languages requires particularly precise pre-preparation to give. For the Slovak language, subsymbolic and statistical machine learning methods work best for the sentiment classification of annual reports.

Statistical machine learning methods applied in the study of web accessibility: a literature review

An accessible website refers to the fact that any person, especially those with physical disabilities, can access the content of the website without problems. The objective of this research is the analysis through literature reviews of machine learning methods applied to the study of accessibility in the web portals of Decentralized Autonomous Governments. In addition, a systematic literature review methodology was used to review more than twenty scientific articles related to keywords such as web accessibility, statistics, and machine learning, among others. In the results obtained, several techniques stand out, especially those of unsupervised learning, since their usefulness was observed in several investigations, improving the analysis and understanding of the data. This research has shown that exciting work can be done on web accessibility in institutions, considering that these studies would significantly contribute to improving access to content.

Integration of machine learning and statistical models for crash frequency modeling

ABSTRACT Crash frequency modeling has been an active research topic in traffic safety, for which various techniques have been proposed that can be loosely classified as either statistical models or machine learning (ML) methods. Statistical models are suitable for drawing inferences and producing relationships that are verifiable by domain experts. However, they generally suffer from low predictive performance due to built-in assumptions about the crash data and adherence to prespecified functional forms. On the other hand, ML methods are data-driven and free from pre-supposed conditions on the dataset, yet they are often not interpretable. In this paper, a combination scheme is proposed to leverage the advantages of both techniques, and it is evaluated using crash data collected from urban highways in the state of Washington. The results show that this combination scheme could significantly improve the predictive performance and model fitness of statistical models without adversely impacting their interpretability.

Single-cell RNA-seq data analysis using graph autoencoders and graph attention networks.

With the development of high-throughput sequencing technology, the scale of single-cell RNA sequencing (scRNA-seq) data has surged. Its data are typically high-dimensional, with high dropout noise and high sparsity. Therefore, gene imputation and cell clustering analysis of scRNA-seq data is increasingly important. Statistical or traditional machine learning methods are inefficient, and improved accuracy is needed. The methods based on deep learning cannot directly process non-Euclidean spatial data, such as cell diagrams. In this study, we developed scGAEGAT, a multi-modal model with graph autoencoders and graph attention networks for scRNA-seq analysis based on graph neural networks. Cosine similarity, median L1 distance, and root-mean-squared error were used to measure the gene imputation performance of different methods for comparison with scGAEGAT. Furthermore, adjusted mutual information, normalized mutual information, completeness score, and Silhouette coefficient score were used to measure the cell clustering performance of different methods for comparison with scGAEGAT. Experimental results demonstrated promising performance of the scGAEGAT model in gene imputation and cell clustering prediction on four scRNA-seq data sets with gold-standard cell labels.

A Comparison between Three Tuning Strategies for Gaussian Kernels in the Context of Univariate Genomic Prediction.

Genomic prediction is revolutionizing plant breeding since candidate genotypes can be selected without the need to measure their trait in the field. When a reference population contains both phenotypic and genotypic information, it is trained by a statistical machine learning method that is subsequently used for making predictions of breeding or phenotypic values of candidate genotypes that were only genotyped. Nevertheless, the successful implementation of the genomic selection (GS) methodology depends on many factors. One key factor is the type of statistical machine learning method used since some are unable to capture nonlinear patterns available in the data. While kernel methods are powerful statistical machine learning algorithms that capture complex nonlinear patterns in the data, their successful implementation strongly depends on the careful tuning process of the involved hyperparameters. As such, in this paper we compare three methods of tuning (manual tuning, grid search, and Bayesian optimization) for the Gaussian kernel under a Bayesian best linear unbiased predictor model. We used six real datasets of wheat (Triticum aestivum L.) to compare the three strategies of tuning. We found that if we want to obtain the major benefits of using Gaussian kernels, it is very important to perform a careful tuning process. The best prediction performance was observed when the tuning process was performed with grid search and Bayesian optimization. However, we did not observe relevant differences between the grid search and Bayesian optimization approach. The observed gains in terms of prediction performance were between 2.1% and 27.8% across the six datasets under study.

A Multi-Trait Gaussian Kernel Genomic Prediction Model under Three Tunning Strategies.

While genomic selection (GS) began revolutionizing plant breeding when it was proposed around 20 years ago, its practical implementation is still challenging as many factors affect its accuracy. One such factor is the choice of the statistical machine learning method. For this reason, we explore the tuning process under a multi-trait framework using the Gaussian kernel with a multi-trait Bayesian Best Linear Unbiased Predictor (GBLUP) model. We explored three methods of tuning (manual, grid search and Bayesian optimization) using 5 real datasets of breeding programs. We found that using grid search and Bayesian optimization improve between 1.9 and 6.8% the prediction accuracy regarding of using manual tuning. While the improvement in prediction accuracy in some cases can be marginal, it is very important to carry out the tuning process carefully to improve the accuracy of the GS methodology, even though this entails greater computational resources.

The Use of Machine Learning to Reduce Overtreatment of the Axilla in Breast Cancer: Retrospective Cohort Study.

Patients with early breast cancer undergoing primary surgery, who have low axillary nodal burden, can safely forego axillary node clearance (ANC). However, routine use of axillary ultrasound (AUS) leads to 43% of patients in this group having ANC unnecessarily, following a positive AUS. The intersection of machine learning with medicine can provide innovative ways to understand specific risks within large patient data sets, but this has not yet been trialed in the arena of axillary node management in breast cancer. The objective of this study was to assess if machine learning techniques could be used to improve preoperative identification of patients with low and high axillary metastatic burden. A single-center retrospective analysis was performed on patients with breast cancer who had a preoperative AUS, and the specificity and sensitivity of AUS were calculated. Standard statistical methods and machine learning methods, including artificial neural network, naive Bayes, support vector machine, and random forest, were applied to the data to see if they could improve the accuracy of preoperative AUS to better discern high and low axillary burden. The study included 459 patients; 142 (31%) had a positive AUS; among this group, 88 (62%) had 2 or fewer macrometastatic nodes at ANC. Logistic regression outperformed AUS (specificity 0.950 vs 0.809). Of all the methods, the artificial neural network had the highest accuracy (0.919). Interestingly, AUS had the highest sensitivity of all methods (0.777), underlining its utility in this setting. We demonstrated that machine learning improves identification of the important subgroup of patients with no palpable axillary disease, positive ultrasound, and more than 2 metastatically involved nodes. A negative ultrasound in patients with no palpable lymphadenopathy is highly indicative of low axillary burden, and it is unclear whether sentinel node biopsy adds value in this situation. Further studies with larger patient numbers focusing on specific breast cancer subgroups are required to refine these techniques in this setting.

Generating Explanations for Conceptual Validation of Graph Neural Networks: An Investigation of Symbolic Predicates Learned on Relevance-Ranked Sub-Graphs

Graph Neural Networks (GNN) show good performance in relational data classification. However, their contribution to concept learning and the validation of their output from an application domain’s and user’s perspective have not been thoroughly studied. We argue that combining symbolic learning methods, such as Inductive Logic Programming (ILP), with statistical machine learning methods, especially GNNs, is an essential forward-looking step to perform powerful and validatable relational concept learning. In this contribution, we introduce a benchmark for the conceptual validation of GNN classification outputs. It consists of the symbolic representations of symmetric and non-symmetric figures that are taken from a well-known Kandinsky Pattern data set. We further provide a novel validation framework that can be used to generate comprehensible explanations with ILP on top of the relevance output of GNN explainers and human-expected relevance for concepts learned by GNNs. Our experiments conducted on our benchmark data set demonstrate that it is possible to extract symbolic concepts from the most relevant explanations that are representative of what a GNN has learned. Our findings open up a variety of avenues for future research on validatable explanations for GNNs.

ILL-IDS: An incremental lifetime learning IDS for VANETs

Vehicle Ad Hoc Network (VANET) is one of the most important approaches for intelligent vehicles to communicate under complex road conditions. However, as a VANET is working under wireless and complex conditions, it is under the threat of various malicious attacks. One efficient solution to counter those attacks is the Intrusion Detection System (IDS), which can detect intrusions to the VANET based on the statistical machine learning method. However, most IDS solutions have a problem that they can only detect the attacks which have already known when the IDS is designed/trained. For unknown attacks, most IDSs become inefficient. A blockchain-based Lifetime Learning IDS (LL-IDS) framework has been designed to address this problem. It applies a blockchain to store uncertain data that an IDS cannot decide and is highly likely to be a new type of attacks. With the help of traditional security agencies such as universities, these uncertain data can be labelled to train the IDS model. The incremental learning models are shown to have a great potential in this scenario. In this paper, we introduces a novel IDS named as incremental IDS based on LL-IDS. Numerical experiments show that the computational time consumption and web payload could be decreased by applying the ILL-IDS to a public VANET dataset.

Development of an Algorithm to Analyze Vacancies in the Labor Market Based on Open-Source Data

In the introductory part of the article, the authors substantiate the relevance of developing methodological tools for analyzing job vacancies in the labor market in the context of the modern technological revolution, which significantly increases requirements for professional knowledge and experience of working personnel and changes the ratio between traditional and new professions.To assess the current situation on the labor market and the demand for currently existing professions, the main section of the published results of the study presents the algorithm for analyzing vacancies using large data arrays from open sources using mathematical and statistical tools and machine learning methods using the Python programming language and the IBM SPSS modeler analytical platform. The algorithm includes: parsing data on vacancies, analyzing vacancies by the main criteria, clustering vacancies by salary level and building a neural network model – a multilayer perceptron of the dependence of salary on a number of predictors. It should be noted that the developed algorithm is universal, because it can be used to analyze big data from any open source at a certain point in time.The results of the analysis will allow researchers and specialists of management structures to more realistically assess the current situation on the labor market, educational institutions will be able to adjust training programs in accordance with the modern requirements of employers, employers will make decisions on the development of competencies in their field of activity and conduct a comparative analysis of demanded vacancies in terms of quantitative and qualitative characteristics, and for the applicant it will be easier to see the demand for vacancies in the labor market and develop new skills.

Multi-trait genome prediction of new environments with partial least squares.

The genomic selection (GS) methodology proposed over 20 years ago by Meuwissen et al. (Genetics, 2001) has revolutionized plant breeding. A predictive methodology that trains statistical machine learning algorithms with phenotypic and genotypic data of a reference population and makes predictions for genotyped candidate lines, GS saves significant resources in the selection of candidate individuals. However, its practical implementation is still challenging when the plant breeder is interested in the prediction of future seasons or new locations and/or environments, which is called the “leave one environment out” issue. Furthermore, because the distributions of the training and testing set do not match, most statistical machine learning methods struggle to produce moderate or reasonable prediction accuracies. For this reason, the main objective of this study was to explore the use of the multi-trait partial least square (MT-PLS) regression methodology for this specific task, benchmarking its performance with the Bayesian Multi-trait Genomic Best Linear Unbiased Predictor (MT-GBLUP) method. The benchmarking process was performed with five actual data sets. We found that in all data sets the MT-PLS method outperformed the popular MT-GBLUP method by 349.8% (under predictor E + G), 484.4% (under predictor E + G + GE; where E denotes environments, G genotypes and GE the genotype by environment interaction) and 15.9% (under predictor G + GE) across traits. Our results provide empirical evidence of the power of the MT-PLS methodology for the prediction of future seasons or new environments. Furthermore, the comparison between single univariate-trait (UT) versus MT for GBLUP and PLS gave an increase in prediction accuracy of MT-GBLUP versus UT-GBLUP, but not for MT-PLS versus UT-PLS.

A Comparison of Three Machine Learning Methods for Multivariate Genomic Prediction Using the Sparse Kernels Method (SKM) Library.

Genomic selection (GS) changed the way plant breeders select genotypes. GS takes advantage of phenotypic and genotypic information to training a statistical machine learning model, which is used to predict phenotypic (or breeding) values of new lines for which only genotypic information is available. Therefore, many statistical machine learning methods have been proposed for this task. Multi-trait (MT) genomic prediction models take advantage of correlated traits to improve prediction accuracy. Therefore, some multivariate statistical machine learning methods are popular for GS. In this paper, we compare the prediction performance of three MT methods: the MT genomic best linear unbiased predictor (GBLUP), the MT partial least squares (PLS) and the multi-trait random forest (RF) methods. Benchmarking was performed with six real datasets. We found that the three investigated methods produce similar results, but under predictors with genotype (G) and environment (E), that is, E + G, the MT GBLUP achieved superior performance, whereas under predictors E + G + genotype environment (GE) and G + GE, random forest achieved the best results. We also found that the best predictions were achieved under the predictors E + G and E + G + GE. Here, we also provide the R code for the implementation of these three statistical machine learning methods in the sparse kernel method (SKM) library, which offers not only options for single-trait prediction with various statistical machine learning methods but also some options for MT predictions that can help to capture improved complex patterns in datasets that are common in genomic selection.

Machine Learning-Based Hyperspectral and RGB Discrimination of Three Polyphagous Fungi Species Grown on Culture Media

In this study, three fungi species (Botrytis cinerea, Rhizoctonia solani, Sclerotinia sclerotiorum) were discriminated using hyperspectral and red-green-blue (RGB) data and machine learning methods. The fungi were incubated at 25 °C for 10 days on potato dextrose agar in Petri dishes. The Hyperspectral data were acquired using an ASD spectroradiometer, which measures reflectance with 3 and 10 nm bandwidths over the range 350–1000 nm and the range 1000–2500 nm, respectively. The RGB images were collected using a digital Canon 450D camera equipped with the DIGIC 3 processor. The research showed the possibility of distinguishing the analysed fungi species based on hyperspectral curves and RGB images and assessing this differentiation using machine learning statistical methods (extreme boosting machine with bootstrap simulation). The best results in analysed fungi discrimination based on hyperspectral data were achieved using the Principal Component Analysis method, in which the average values of recognition and accuracy for all three species were 0.96 and 0.93, respectively. The wavelengths of the shortwave infrared (SWIR) wavelength region appeared to be the most effective in distinguishing B. cinerea-R. solani and B. cinerea-S. sclerotiorum, while of the visible range (VIS) of electromagnetic spectrum in discrimination of R. solani-S. sclerotiorum. The hyperspectral reflectance data were strongly correlated with the intensity of the pixels in the visible range (R2 = 0.894–0.984). The RGB images proved to be successfully used primarily for the identification of R. solani (recognition = 0.90, accuracy = 0.79) and S. sclerotiorum (recognition = 0.84, accuracy = 0.76). The greatest differences in the intensity of the pixels between B. cinerea and R. solani as well as R. solani and S. sclerotiorum occurred in the blue band and in distinguishing B. cinerea and S. sclerotiorum in the red band.

Early prediction of clinical scores for left ventricular reverse remodeling using extreme gradient random forest, boosting, and logistic regression algorithm representations.

ObjectiveAt present, there is no early prediction model of left ventricular reverse remodeling (LVRR) for people who are in cardiac arrest with an ejection fraction (EF) of ≤35% at first diagnosis; thus, the purpose of this article is to provide a supplement to existing research.Materials and methodsA total of 109 patients suffering from heart attack with an EF of ≤35% at first diagnosis were involved in this single-center research study. LVRR was defined as an absolute increase in left ventricular ejection fraction (LVEF) from ≥10% to a final value of >35%, with analysis features including demographic characteristics, diseases, biochemical data, echocardiography, and drug therapy. Extreme gradient boosting (XGBoost), random forest, and logistic regression algorithm models were used to distinguish between LVRR and non-LVRR cases and to obtain the most important features.ResultsThere were 47 cases (42%) of LVRR in patients suffering from heart failure with an EF of ≤35% at first diagnosis after optimal drug therapy. General statistical analysis and machine learning methods were combined to exclude a number of significant feature groups. The median duration of disease in the LVRR group was significantly lower than that in the non-LVRR group (7 vs. 48 months); the mean values of creatine kinase (CK) and MB isoenzyme of creatine kinase (CK-MB) in the LVRR group were lower than those in the non-LVRR group (80.11 vs. 94.23 U/L; 2.61 vs. 2.99 ng/ml; 27.19 vs. 28.54 mm). Moreover, AUC values for our feature combinations ranged from 97 to 94% and to 87% when using the XGBoost, random forest, and logistic regression techniques, respectively. The ablation test revealed that beats per minute (BPM) and disease duration had a greater impact on the model’s ability to accurately forecast outcomes.ConclusionShorter disease duration, slightly lower CK and CK-MB levels, slightly smaller right and left ventricular and left atrial dimensions, and lower mean heart rates were found to be most strongly predictive of LVRR development (BPM).