Conventional Machine Learning Models Research Articles

Testing the applicability and performance of Auto ML for potential applications in diagnostic neuroradiology

To investigate the applicability and performance of automated machine learning (AutoML) for potential applications in diagnostic neuroradiology. In the medical sector, there is a rapidly growing demand for machine learning methods, but only a limited number of corresponding experts. The comparatively simple handling of AutoML should enable even non-experts to develop adequate machine learning models with manageable effort. We aim to investigate the feasibility as well as the advantages and disadvantages of developing AutoML models compared to developing conventional machine learning models. We discuss the results in relation to a concrete example of a medical prediction application. In this retrospective IRB-approved study, a cohort of 107 patients who underwent gross total meningioma resection and a second cohort of 31 patients who underwent subtotal resection were included. Image segmentation of the contrast enhancing parts of the tumor was performed semi-automatically using the open-source software platform 3D Slicer. A total of 107 radiomic features were extracted by hand-delineated regions of interest from the pre-treatment MRI images of each patient. Within the AutoML approach, 20 different machine learning algorithms were trained and tested simultaneously. For comparison, a neural network and different conventional machine learning algorithms were trained and tested. With respect to the exemplary medical prediction application used in this study to evaluate the performance of Auto ML, namely the pre-treatment prediction of the achievable resection status of meningioma, AutoML achieved remarkable performance nearly equivalent to that of a feed-forward neural network with a single hidden layer. However, in the clinical case study considered here, logistic regression outperformed the AutoML algorithm. Using independent test data, we observed the following classification results (AutoML/neural network/logistic regression): mean area under the curve = 0.849/0.879/0.900, mean accuracy = 0.821/0.839/0.881, mean kappa = 0.465/0.491/0.644, mean sensitivity = 0.578/0.577/0.692 and mean specificity = 0.891/0.914/0.936. The results obtained with AutoML are therefore very promising. However, the AutoML models in our study did not yet show the corresponding performance of the best models obtained with conventional machine learning methods. While AutoML may facilitate and simplify the task of training and testing machine learning algorithms as applied in the field of neuroradiology and medical imaging, a considerable amount of expert knowledge may still be needed to develop models with the highest possible discriminatory power for diagnostic neuroradiology.

DI-ADS: A Deep Intelligent Distributed Denial of Service Attack Detection Scheme for Fog-Based IoT Applications

Nowadays, fog computing plays a very vital role in providing many services to end-based IoT (Internet of Things) systems. The end IoT devices communicate with the middle layer fog nodes and to the above cloud layer to process the user tasks. However, this large data communication experiences many security challenges as IoT devices are being compromised and thus the fog nodes at the fog layer are more prone to a very critical attack known as Distributed Denial of Service (DDoS) attack. The attackers or the compromised IoT devices need to be detected well in the network. Deep Learning (DL) plays a prominent role in predicting the end-user behavior by extracting features and classifying the adversary in the network. But, due to IoT device’s constrained nature in computation and storage facilities, DL cannot be administered on those. In this paper, a deep intelligent DDoS attack detection scheme (DI-ADS) is proposed for fog-based IoT applications. The framework mainly uses a deep learning model (DLM) to detect DDoS attacks in the network. The DLM is installed on the computation module of the fog node that predicts the end IoT device behavior. For the selection of the best DLM model at the fog layer, the performance comparison is made on Deep Neural Multilayer Perceptron (DNMLP) and Long Short-Term Memory (LSTM) models along with the conventional machine learning (ML) models such as Support Vector Machine (SVM), K-Nearest Neighbours (KNN), Logistic Regression (LR), and Random Forest (RF). The simulation is performed using the Python Anaconda platform by considering a new DDoS-SDN (Mendeley Dataset) dataset that consists of three DDoS attacks such as TCP Syn, UDP Flood, and ICMP attacks. From the results, DNMLP showed the best accuracy of 99.44% as compared to other DL and ML models. By outperforming nature in the detection of DDoS attacks, DNMLP is considered in the proposed framework for being implemented at the fog layer.

Blood–brain barrier penetration prediction enhanced by uncertainty estimation

Blood–brain barrier is a pivotal factor to be considered in the process of central nervous system (CNS) drug development, and it is of great significance to rapidly explore the blood–brain barrier permeability (BBBp) of compounds in silico in early drug discovery process. Here, we focus on whether and how uncertainty estimation methods improve in silico BBBp models. We briefly surveyed the current state of in silico BBBp prediction and uncertainty estimation methods of deep learning models, and curated an independent dataset to determine the reliability of the state-of-the-art algorithms. The results exhibit that, despite the comparable performance on BBBp prediction between graph neural networks-based deep learning models and conventional physicochemical-based machine learning models, the GROVER-BBBp model shows greatly improvement when using uncertainty estimations. In particular, the strategy combined Entropy and MC-dropout can increase the accuracy of distinguishing BBB + from BBB − to above 99% by extracting predictions with high confidence level (uncertainty score < 0.1). Case studies on preclinical/clinical drugs for Alzheimer’ s disease and marketed antitumor drugs that verified by literature proved the application value of uncertainty estimation enhanced BBBp prediction model, that may facilitate the drug discovery in the field of CNS diseases and metastatic brain tumors.

A physics-constrained long-term production prediction method for multiple fractured wells using deep learning

Production prediction of hydraulic fractured wells is often complicated due to the incomplete understanding of production mechanisms and insufficient amount of available data. As a prevalent approach, machine learning can capture nonlinear patterns of production variation but often yields physically inconsistent predictions due to the lack of involvement of physical laws, especially for out-of-sample scenarios. In this work, we propose a novel physics-constrained deep learning framework for long-term production prediction of multiple fractured wells based on a Bidirectional gated recurrent unit (BiGRU) and Deep hybrid neural network (DHNN) combined neural network (BiGRU-DHNN). The proposed approach can take full advantage of the complementary strengths of physics knowledge and machine learning to better acquire the dynamics of time-series production. Static and dynamic, temporal and spatial data of fractured wells are treated as additional variables to enforce physical constraints on the long-term production forecasting with a multi-step-ahead prediction strategy. Two multi-well field cases are conducted to compare the prediction performance of the proposed physics-constrained BiGRU-DHNN method with some conventional time-series machine learning models, including Recurrent neural network (RNN), Long short-term memory (LSTM), Gated recurrent unit (GRU) and BiGRU, by ignoring physical constraints. The results demonstrate that the proposed method outperforms conventional machine learning methods under the constraints of geological properties, logging curves, well information, fracturing parameters and field operations. By considering the physics-based dependencies between constraints and production, the BiGRU-DHNN model significantly improves the accuracy, robustness and generalization of the long-term production prediction of fractured wells in a gridless and efficient manner. Furthermore, the contribution mechanisms of these constraints to the model prediction performance are also investigated to improve its interpretability, which provides guidance to field personnel on which kind of constraints should be prioritized and addressed if uncertainties exist in the practical application. The findings are expected to provide insight into the integration of physics into machine learning for accurate production prediction and to be employed as a potential tool to support fracturing design optimization.

A Self-Tuning Cyber-Attacks’ Location Identification Approach for Critical Infrastructures

The integration of the communications network and the Internet of Things in today’s critical infrastructures facilitates intelligent and online monitoring of these systems. However, although critical infrastructure’s digitalization brings tremendous advantages and opportunities for remote access and control, it significantly increases cyber-attack’s vulnerability. Therefore, efficient and proper detection and localization of cyber-attack are paramount for the critical infrastructure’s reliable and secure operation. This article proposes a deep learning-based cyber-attack detection and location identification system for critical infrastructures by constructing new representations and model the system behavior using multilayer autoencoders. The results show that the new representations capture the physical relationships among the measurements and have more discriminant power in distinguishing the location of the attack. Furthermore, the proposed method has outperformed conventional machine learning models under various cyber-attack scenarios using real-world data from the gas pipeline and water distribution supervisory control and data acquisition systems.

Landslide Susceptibility Mapping Using Machine Learning: A Literature Survey

Landslide is a devastating natural disaster, causing loss of life and property. It is likely to occur more frequently due to increasing urbanization, deforestation, and climate change. Landslide susceptibility mapping is vital to safeguard life and property. This article surveys machine learning (ML) models used for landslide susceptibility mapping to understand the current trend by analyzing published articles based on the ML models, landslide causative factors (LCFs), study location, datasets, evaluation methods, and model performance. Existing literature considered in this comprehensive survey is systematically selected using the ROSES protocol. The trend indicates a growing interest in the field. The choice of LCFs depends on data availability and case study location; China is the most studied location, and area under the receiver operating characteristic curve (AUC) is considered the best evaluation metric. Many ML models have achieved an AUC value &gt; 0.90, indicating high reliability of the susceptibility map generated. This paper also discusses the recently developed hybrid, ensemble, and deep learning (DL) models in landslide susceptibility mapping. Generally, hybrid, ensemble, and DL models outperform conventional ML models. Based on the survey, a few recommendations and future works which may help the new researchers in the field are also presented.

The scatter in the galaxy–halo connection: a machine learning analysis

ABSTRACT We apply machine learning (ML), a powerful method for uncovering complex correlations in high-dimensional data, to the galaxy–halo connection of cosmological hydrodynamical simulations. The mapping between galaxy and halo variables is stochastic in the absence of perfect information, but conventional ML models are deterministic and hence cannot capture its intrinsic scatter. To overcome this limitation, we design an ensemble of neural networks with a Gaussian loss function that predict probability distributions, allowing us to model statistical uncertainties in the galaxy–halo connection as well as its best-fitting trends. We extract a number of galaxy and halo variables from the Horizon-AGN and IllustrisTNG100-1 simulations and quantify the extent to which knowledge of some subset of one enables prediction of the other. This allows us to identify the key features of the galaxy–halo connection and investigate the origin of its scatter in various projections. We find that while halo properties beyond mass account for up to 50 per cent of the scatter in the halo-to-stellar mass relation, the prediction of stellar half-mass radius or total gas mass is not substantially improved by adding further halo properties. We also use these results to investigate semi-analytic models for galaxy size in the two simulations, finding that assumptions relating galaxy size to halo size or spin are not successful.

Understanding Public Perceptions of Measles from Twitter Using Multi-Task Convolutional Neural Networks.

Measles is a highly contagious cause of febrile illness typically seen in young children. Recent years have witnessed the resurgence of measles cases in the United States. Prompt understanding of public perceptions of measles will allow public health agencies to respond appropriately promptly. We proposed a multi-task Convolutional Neural Network (MT-CNN) model to classify measles-related tweets in terms of three characteristics: Type of Message (6 subclasses), Emotion Expressed (6 subclasses), and Attitude towards Vaccination (3 subclasses). A gold standard corpus that contains 2,997 tweets with annotation in these dimensions was manually curated. A variety of conventional machine learning and deep learning models were evaluated as baseline models. The MT-CNN model performed better than other baseline conventional machine learning and the signal-task CNN models, and was then applied to predict unlabeled measles-related Twitter discussions that were crawled from 2007 to 2019, and the trends of public perceptions were analyzed along three dimensions.

A Hybrid Ensemble Stacking Model for Gender Voice Recognition Approach

Gender recognition by voice is a vital research subject in speech processing and acoustics, as human voices have many remarkable characteristics. Voice recognition is beneficial in a variety of applications, including mobile health care systems, interactive systems, crime analysis, and recognition systems. Several algorithms for voice recognition have been developed, but there is still potential for development in terms of the system’s accuracy and efficiency. Recent research has focused on combining ensemble learning with a variety of machine learning models in order to create more accurate classifiers. In this paper, a stacked ensemble for gender voice recognition model is presented, using four classifiers, namely, k-nearest neighbor (KNN), support vector machine (SVM), stochastic gradient descent (SGD), and logistic regression (LR) as base classifiers and linear discriminant analysis (LDA) as meta classifier. The dataset used includes 3168 instances and 21 features, where 20 features are the predictors, and one feature is the target. Several prediction evaluation metrics, including precision, accuracy, recall, F1 score, and area under the receiver operating characteristic curve (AUC), were computed to verify the execution of the proposed model. The results obtained illustrated that the stacked model achieved better results compared to other conventional machine learning models. The stacked model achieved high accuracy with 99.64%.

Self-Supervised Graph Neural Networks for Accurate Prediction of Néel Temperature

Antiferromagnetic materials are exciting quantum materials with rich physics and great potential for applications. On the other hand, an accurate and efficient theoretical method is highly demanded for determining critical transition temperatures, Néel temperatures, of antiferromagnetic materials. The powerful graph neural networks (GNNs) that succeed in predicting material properties lose their advantage in predicting magnetic properties due to the small dataset of magnetic materials, while conventional machine learning models heavily depend on the quality of material descriptors. We propose a new strategy to extract high-level material representations by utilizing self-supervised training of GNNs on large-scale unlabeled datasets. According to the dimensional reduction analysis, we find that the learned knowledge about elements and magnetism transfers to the generated atomic vector representations. Compared with popular manually constructed descriptors and crystal graph convolutional neural networks, self-supervised material representations can help us to obtain a more accurate and efficient model for Néel temperatures, and the trained model can successfully predict high Néel temperature antiferromagnetic materials. Our self-supervised GNN may serve as a universal pre-training framework for various material properties.

Deep Learning-Based Diagnosis of Alzheimer's Disease.

Alzheimer’s disease (AD), the most familiar type of dementia, is a severe concern in modern healthcare. Around 5.5 million people aged 65 and above have AD, and it is the sixth leading cause of mortality in the US. AD is an irreversible, degenerative brain disorder characterized by a loss of cognitive function and has no proven cure. Deep learning techniques have gained popularity in recent years, particularly in the domains of natural language processing and computer vision. Since 2014, these techniques have begun to achieve substantial consideration in AD diagnosis research, and the number of papers published in this arena is rising drastically. Deep learning techniques have been reported to be more accurate for AD diagnosis in comparison to conventional machine learning models. Motivated to explore the potential of deep learning in AD diagnosis, this study reviews the current state-of-the-art in AD diagnosis using deep learning. We summarize the most recent trends and findings using a thorough literature review. The study also explores the different biomarkers and datasets for AD diagnosis. Even though deep learning has shown promise in AD diagnosis, there are still several challenges that need to be addressed.

DeepTESR: A Deep Learning Framework to Predict the Degree of Translational Elongation Short Ramp for Gene Expression Control.

Controlling translational elongation is essential for efficient protein synthesis. Ribosome profiling has revealed that the speed of ribosome movement is correlated with translational efficiency in the translational elongation ramp. In this work, we present a new deep learning model, called DeepTESR, to predict the degree of translational elongation short ramp (TESR) from mRNA sequence. The proposed deep learning model exhibited superior performance in predicting the TESR scores for 226 981 TESR sequences, resulting in the mean absolute error (MAE) of 0.285 and a coefficient of determination R2 of 0.627, superior to the conventional machine learning models (e.g., MAE of 0.335 and R2 of 0.571 for LightGBM). We experimentally validated that heterologous fluorescence expression of proteins with randomly selected TESR was moderately correlated with the predictions. Furthermore, a genome-wide analysis of TESR prediction in the 4305 coding sequences of Escherichia coli showed conserved TESRs over the clusters of orthologous groups. In this sense, DeepTESR can be used to predict the degree of TESR for gene expression control and to decipher the mechanism of translational control with ribosome profiling. DeepTESR is available at https://github.com/fmblab/DeepTESR.

Improved Classification Model using CNN for Detection of Alzheimer’s Disease

Alzheimer’s Disease (AD) is commonly called a neurodegenerative disorder and it is a common form of dementia. There is no permanent cure for this brain disease hence the early diagnosis of such disease using medical imaging system is highly significant. Machine learning models play a vital role in the detection of AD. Since most of the conventional machine learning models find it difficult to detect the essential features to classify the disease, an advanced deep learning framework called Convolutional Neural Network (CNN) is used in this study to detect essential features automatically and classify the disease. The building components of the proposed CNN-based classification method include convolution layer, batch normalization process, ReLU, and Max-pooling operation. The main objective of this CNN-based classification method is to predict whether the patient is suffering from Alzheimer's disease through the analysis of brain MRI. The proposed methodology implemented is identical to a classification-based system that undergoes training, evaluation, and testing process. Finally, the softmax layer is applied for classification, and the Adam optimization technique is applied for reducing the loss, and by applying Adam quicker convergence can be achieved. The proposed improved CNN classification method achieves an accuracy of 97.8%.

Predicting stress–strain curves using transfer learning: Knowledge transfer across polymer composites

The engineering stress–strain curve of a material allows one to determine mechanical properties such as elastic modulus, strength, and toughness. While machine learning has recently been used to predict stress–strain curves, large volumes of experimental data are required to achieve high prediction accuracy. More importantly, conventional machine learning models are not generalizable from one material to another. To address this issue, a novel transfer learning approach is introduced to predict stress–strain curves across different fiber reinforced polymer (FRP) composites fabricated via additive manufacturing. Optimal transport (OT) is integrated with the transfer learning approach by mapping the source label space to the target label space, which further improves knowledge transferability across different FRP composites. The stress–strain curves of the additively manufactured FRP composites are generated under flexural loading conditions. Experimental results show that the OT-integrated transfer learning approach can predict the complex stress–strain curves of FRP composites with small training data. The predictive model achieved a mean absolute percentage error (MAPE) of less than 10%. The extracted modulus and strength from the predicted stress–strain curves achieved MAPEs of less than 5% and 10%, respectively. This work demonstrates that transfer learning has the potential to transform how stress–strain curves are generated.

A multi-task Gaussian process self-attention neural network for real-time prediction of the need for mechanical ventilators in COVID-19 patients

ObjectiveThe Coronavirus Disease 2019 (COVID-19) pandemic has overwhelmed the capacity of healthcare resources and posed a challenge for worldwide hospitals. The ability to distinguish potentially deteriorating patients from the rest helps facilitate reasonable allocation of medical resources, such as ventilators, hospital beds, and human resources. The real-time accurate prediction of a patient's risk scores could also help physicians to provide earlier respiratory support for the patient and reduce the risk of mortality. MethodsWe propose a robust real-time prediction model for the in-hospital COVID-19 patients' probability of requiring mechanical ventilation (MV). The end-to-end neural network model incorporates the Multi-task Gaussian Process to handle the irregular sampling rate in observational data together with a self-attention neural network for the prediction task. ResultsWe evaluate our model on a large database with 9,532 nationwide in-hospital patients with COVID-19. The model demonstrates significant robustness and consistency improvements compared to conventional machine learning models. The proposed prediction model also shows performance improvements in terms of area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) compared to various deep learning models, especially at early times after a patient's hospital admission. ConclusionThe availability of large and real-time clinical data calls for new methods to make the best use of them for real-time patient risk prediction. It is not ideal for simplifying the data for traditional methods or for making unrealistic assumptions that deviate from observation's true dynamics. We demonstrate a pilot effort to harmonize cross-sectional and longitudinal information for mechanical ventilation needing prediction.

Use of 3D Coronal and Sagittal Images to Improve the Diagnosis of Brain Tumor

A brain tumour is a type of cancer which is very hard to detect by a doctor in the starting stages. Generally, the shape and size of the tumour are unidentified. The Brain tumour classification is performed by serologic analysis and is not usually conducted before conclusive brain surgery. Normally Brain tumour is predicted by Magnetic Resonance Imaging (MRI) images, however, it is time-consuming and high cost. Nowadays a lot of data sets are available for identifying the several stages of brain tumour such as Glioma, Meningioma and a Pituitary tumour to train the Machine Learning (ML) model. The conventional ML models logistic regression, support vector machine (SVM), Convolution Neural Network (CNN) and Residual Neural Network (RNN) are used to predict the location of tumour present in the brain and also able to create tumour pattern mask. However, their accuracy is very less. In this paper, an effective ML model 3D U-NET has been developed that can generate a tumour pattern mask for any type of tumour present in the brain. The proposed model provides better accuracy as compared to the conventional method. Simulation results shows that, 85% accuracy.

Comparison of multiple conventional and unconventional machine learning models for landslide susceptibility mapping of Northern part of Pakistan

Comparison of multiple conventional and unconventional machine learning models for landslide susceptibility mapping of Northern part of Pakistan

Mean–variance portfolio optimization with deep learning based-forecasts for cointegrated stocks

Most mean–variance (MV) models construct a portfolio based on nonstationary stocks. This study presents a new MV model constructed using stationary portfolios composed of cointegrated stocks. The expected return of this new model is predicted by using machine learning models, such as support vector machine, random forest, and attention-based long short-term memory (LSTM) network. The proposed model is evaluated using data on stocks in the CSI 300 and the S&P 500, with 42 features over 8 years from May 4, 2012 to August 4, 2020. The empirical results show that the portfolio constructed based on the stationary portfolios in both the Chinese and the US stock markets delivers significant profits. Further, the attention-based LSTM network can more accurately model the spread return using technical indicators, financial investment information, and lagged returns, and can successfully select pairs of cointegrated stocks for constructing a more profitable MV portfolio, than can conventional machine learning models. Using the attention-based LSTM to predict 20-day return movement results in model accuracy of up to 92.59% for the CSI 300 and 88.52% for the S&P 500, and a corresponding Sharpe ratio of 9.31 and 2.77, respectively.

Exploring Quantum Machine Learning and Feature Reduction Techniques for Wind Turbine Pitch Fault Detection

Driven by the development of machine learning (ML) and deep learning techniques, prognostics and health management (PHM) has become a key aspect of reliability engineering research. With the recent rise in popularity of quantum computing algorithms and public availability of first-generation quantum hardware, it is of interest to assess their potential for efficiently handling large quantities of operational data for PHM purposes. This paper addresses the application of quantum kernel classification models for fault detection in wind turbine systems (WTSs). The analyzed data correspond to low-frequency SCADA sensor measurements and recorded SCADA alarm logs, focused on the early detection of pitch fault failures. This work aims to explore potential advantages of quantum kernel methods, such as quantum support vector machines (Q-SVMs), over traditional ML approaches and compare principal component analysis (PCA) and autoencoders (AE) as feature reduction tools. Results show that the proposed quantum approach is comparable to conventional ML models in terms of performance and can outperform traditional models (random forest, k-nearest neighbors) for the selected reduced dimensionality of 19 features for both PCA and AE. The overall highest mean accuracies obtained are 0.945 for Gaussian SVM and 0.925 for Q-SVM models.

Machine learning for imbalanced datasets: Application in prediction of 3d-5d double perovskite structures

3d-5d double perovskite is one kind of the most promising materials due to its novel properties and excellent performance with strong spin-orbit coupling. We used machine learning (ML) to search new 3d-5d perovskites. Three new methods derived from rescaling strategy in training models are used to solve the imbalanced problem which is more effective and more accurate than conventional machine learning models. Out of 664 scanned candidates of A2BB'O6 double perovskite (A is nonmagnetic metal, B and B' are 3d and 5d transition metals, respectively), we find 166 compounds can be stable 3d-5d double-perovskite structures, where A cations have to be +3 or +2 changed. Our results may throw light on the development of perovskite-based devices with strong spin-orbital couplings.