Training Data Points Research Articles

Improving Covariance Matrices Derived from Tiny Training Datasets for the Classification of Event-Related Potentials with Linear Discriminant Analysis

Electroencephalogram data used in the domain of brain–computer interfaces typically has subpar signal-to-noise ratio and data acquisition is expensive. An effective and commonly used classifier to discriminate event-related potentials is the linear discriminant analysis which, however, requires an estimate of the feature distribution. While this information is provided by the feature covariance matrix its large number of free parameters calls for regularization approaches like Ledoit–Wolf shrinkage. Assuming that the noise of event-related potential recordings is not time-locked, we propose to decouple the time component from the covariance matrix of event-related potential data in order to further improve the estimates of the covariance matrix for linear discriminant analysis. We compare three regularized variants thereof and a feature representation based on Riemannian geometry against our proposed novel linear discriminant analysis with time-decoupled covariance estimates. Extensive evaluations on 14 electroencephalogram datasets reveal, that the novel approach increases the classification performance by up to four percentage points for small training datasets, and gracefully converges to the performance of standard shrinkage-regularized LDA for large training datasets. Given these results, practitioners in this field should consider using our proposed time-decoupled covariance estimation when they apply linear discriminant analysis to classify event-related potentials, especially when few training data points are available.

Lock-Free Parallelization for Variance-Reduced Stochastic Gradient Descent on Streaming Data

Stochastic Gradient Descent (SGD) is an iterative algorithm for fitting a model to the training dataset in machine learning problems. With low computation cost, SGD is especially suited for learning from large datasets. However, the variance of SGD tends to be high because it uses only a single data point to determine the update direction at each iteration of gradient descent, rather than all available training data points. Recent research has proposed variance-reduced variants of SGD by incorporating a correction term to approximate full-data gradients. However, it is difficult to parallelize such variants with high performance and accuracy, especially on streaming data. As parallelization is a crucial requirement for large-scale applications, this article focuses on the parallel setting in a multicore machine and presents LFS-STRSAGA, a lock-free approach to parallelizing variance-reduced SGD on streaming data. LFS-STRSAGA embraces a lock-free data structure to process the arrival of streaming data in parallel, and asynchronously maintains the essential information to approximate full-data gradients with low cost. Both our theoretical and empirical results show that LFS-STRSAGA matches the accuracy of the state-of-the-art variance-reduced SGD on streaming data under sparsity assumption (common in machine learning problems), and that LFS-STRSAGA reduces the model update time by over 98 percent.

Density-weighted support vector machines for binary class imbalance learning

In real-world binary classification problems, the entirety of samples belonging to each class varies. These types of problems where the majority class is notably bigger than the minority class can be called as class imbalance learning (CIL) problem. Due to the CIL problem, model performance may degrade. This paper presents a new support vector machine (SVM) model based on density weight for binary CIL (DSVM-CIL) problem. Additionally, an improved 2-norm-based density-weighted least squares SVM for binary CIL (IDLSSVM-CIL) is also proposed to increase the training speed of DSVM-CIL. In IDLSSVM-CIL, the least squares solution is obtained by considering 2-norm of slack variables and solving the primal problem of DSVM-CIL with equality constraints instead of inequality constraints. The basic ideas behind the algorithms are that the training datapoints are given weights during the training phase based on their class distributions. The weights are generated by using a density-weighted technique (Cha et al. in Expert Syst Appl 41(7):3343–3350, 2014) to reduce the effects of CIL. Experimental analyses are performed on some interesting imbalanced artificial and real-world datasets, and their performances are measured using the area under the curve and geometric mean (G-mean). The results are compared with SVM, least squares SVM, fuzzy SVM, improved fuzzy least squares SVM, affinity and class probability-based fuzzy SVM and entropy-based fuzzy least squares SVM. Similar or better generalization results indicate the efficacy and applicability of the proposed algorithms.

Fuzzy support vector machine for imbalanced data with borderline noise

This work is an extension of the Fuzzy Support Vector Machines for Class Imbalance Learning (FSVM-CIL) method proposed by Rukshan Batuwita and Vasile Palade. For FSVMs, a very important part is the fuzzy function transforming different distance measures to membership values between 0 and 1. The larger the membership value, the more important the corresponding training data point. Although various variants have been proposed recently, few have discussed proper fuzzy functions. This work first shows the limitations of fuzzy functions in original FSVM-CIL for imbalanced data with noise around the between-class borderline (noted as borderline noise in this paper), and then, a new fuzzy function, named the Gaussian fuzzy function, is proposed and explained in detail. Modifications are also made to the current distance measures. Experiments on several public imbalanced datasets show the effectiveness of the proposed methods through the comparison with FSVM-CIL and several other popular approaches for imbalanced data.

Least squares KNN-based weighted multiclass twin SVM

K-nearest neighbor (KNN) based weighted multi-class twin support vector machines (KWMTSVM) is a novel multi-class classification method. In this paper, we propose a novel least squares version of KWMTSVM called LS-KWMTSVM by replacing the inequality constraints with equality constraints and minimized the slack variables using squares of 2-norm instead of conventional 1-norm. This simple modification leads to a very fast algorithm with much better results. The modified primal problems in the proposed LS-KWMTSVM solves only two systems of linear equations whereas two quadratic programming problems (QPPs) need to solve in KWMTSVM. The proposed LS-KWMTSVM, same as KWMTSVM, employed the weight matrix in the objective function to exploit the local information of the training samples. To exploit the inter class information, we use weight vectors in the constraints of the proposed LS-KWMTSVM. If any component of vectors is zero then the corresponding constraint is redundant and thus we can avoid it. Elimination of redundant constraints and solving a system of linear equations instead of QPPs makes the proposed LS-KWMTSVM more robust and faster than KWMTSVM. The proposed LS-KWMTSVM, commensurate as the KWMTSVM, all the training data points into a “1-versus-1-versus-rest” structure, and thus our LS-KWMTSVM generate ternary output {-1,0,+1} which helps to deal with imbalance datasets. Numerical experiments on several UCI and KEEL imbalance datasets(with high imbalance ratio) clearly indicate that the proposed LS-KWMTSVM has better classification accuracy compared with other baseline methods but with remarkably less computational time.

A linear space adjustment by mapping data into an intermediate space and keeping low level data structures

ABSTRACT One of the most important assumptions in machine learning tasks is the fact that training data points and test data points are extracted from the same distribution. However, this paper assumes the situation in which this fact does no longer hold. Therefore, a task named space adjustment, through which the distribution of the data points in the training-data space and the distribution of the data points in the test-data space become identical, is inevitable. Hereby, authors propose a linear mapping for the space adjustment task in the paper. It considers four approaches for preserving localities among data samples during the space adjustment. Each approach is defined based on a different locality concept. Considering all locality concepts in an objective function, authors transform the space adjustment into an optimisation problem. The paper proposes to optimise the corresponding objective function by an iterative approach. Empirical study shows that the proposed method outperforms the baseline methods. To do experiments, authors employ a large number of real-world datasets.

Predicting hardness profile of steel specimens subjected to Jominy test using an artificial neural network and electromagnetic nondestructive techniques

ABSTRACT Based on relationships between electromagnetic properties and microstructural changes in steel, the current paper describes the development of an electromagnetic technique to determine steel hardenability, non-destructively. Performing Jominy end-quench test, magnetic hysteresis loop and eddy current methods have been applied to SAE 4140 standard steel samples to explore applicability of the non-destructive techniques for determining hardenability. Different hysteresis loop parameters (maximum magnetic flux density, coercivity, hysteresis loss and maximum differential permeability) as well as eddy current outputs (induced voltage, normalised impedance and phase angle) as a function of distance from the end-quenched surface were investigated and the results show a clear agreement with data obtained by traditional hardness testing. In addition, taking advantage of a generalised regression neural network, magnetic data obtained from hysteresis loop and eddy current methods could be used to accurately estimate results in the hardness profile of a given sample, using a few training data points. This paper shows that coupling the proposed non-destructive methods to Jominy end-quench test facilitates accurate non-destructive hardenability determination of steel samples while reducing time and cost of the test process.

Charge and Exciton Transfer Simulations Using Machine-Learned Hamiltonians.

Quantum-mechanical simulations of charge and exciton transfer in molecular organic materials are a key method to increase our understanding of organic semiconductors. Our goal is to build an efficient multiscale model to predict charge-transfer mobilities and exciton diffusion constants from nonadiabatic molecular dynamics simulations and Marcus-based Monte Carlo approaches. In this work, we apply machine learning models to simulate charge and exciton propagation in organic semiconductors. We show that kernel ridge regression models can be trained to predict electronic and excitonic couplings from semiempirical density functional tight binding (DFTB) reference data with very good accuracy. In simulations, the models could reproduce hole mobilities along the anthracene crystal axes to within 8.5% of the DFTB reference and 34% of the experimental results with only 1000 training data points. Using these models decreased the cost of exciton transfer simulations by one order of magnitude.

A generalized predictive model for TiO2–Catalyzed photo-degradation rate constants of water contaminants through artificial neural network

Titanium dioxide (TiO2) is a well-known photocatalyst in the applications of water contaminant treatment. Traditionally, the kinetics of photo-degradation rates are obtained from experiments, which consumes enormous labor and experimental investments. Here, a generalized predictive model was developed for prediction of the photo-degradation rate constants of organic contaminants in the presence of TiO2 nanoparticles and ultraviolet irradiation in aqueous solution. This model combines an artificial neural network (ANN) with a variety of factors that affect the photo-degradation performance, i.e., ultraviolet intensity, TiO2 dosage, organic contaminant type and initial concentration in water, and initial pH of the solution. The molecular fingerprints (MF) were used to interpret the organic contaminants as binary vectors, a format that is machine-readable in computational linguistics. A dataset of 446 data points for training and testing was collected from the literature. This predictive model shows a good accuracy with a root mean square error (RMSE) of 0.173.

Ensemble learning of coarse-grained molecular dynamics force fields with a kernel approach.

Gradient-domain machine learning (GDML) is an accurate and efficient approach to learn a molecular potential and associated force field based on the kernel ridge regression algorithm. Here, we demonstrate its application to learn an effective coarse-grained (CG) model from all-atom simulation data in a sample efficient manner. The CG force field is learned by following the thermodynamic consistency principle, here by minimizing the error between the predicted CG force and the all-atom mean force in the CG coordinates. Solving this problem by GDML directly is impossible because coarse-graining requires averaging over many training data points, resulting in impractical memory requirements for storing the kernel matrices. In this work, we propose a data-efficient and memory-saving alternative. Using ensemble learning and stratified sampling, we propose a 2-layer training scheme that enables GDML to learn an effective CG model. We illustrate our method on a simple biomolecular system, alanine dipeptide, by reconstructing the free energy landscape of a CG variant of this molecule. Our novel GDML training scheme yields a smaller free energy error than neural networks when the training set is small, and a comparably high accuracy when the training set is sufficiently large.

Machine-Learning Coupled Cluster Properties through a Density Tensor Representation.

The introduction of machine-learning (ML) algorithms to quantum mechanics enables rapid evaluation of otherwise intractable expressions at the cost of prior training on appropriate benchmarks. Many computational bottlenecks in the evaluation of accurate electronic structure theory could potentially benefit from the application of such models, from reducing the complexity of the underlying wave function parameter space to circumventing the complications of solving the electronic Schrödinger equation entirely. Applications of ML to electronic structure have thus far been focused on learning molecular properties (mainly the energy) from geometric representations. While this line of study has been quite successful, highly accurate models typically require a "big data" approach with thousands of training data points. Herein, we propose a general, systematically improvable scheme for wave function-based ML of arbitrary molecular properties, inspired by the underlying equations that govern the canonical approach to computing the properties. To this end, we combine the established ML machinery of the t-amplitude tensor representation with a new reduced density matrix representation. The resulting model provides quantitative accuracy in both the electronic energy and dipoles of small molecules using only a few dozen training points per system.

Real-Time Prediction of Rheological Properties of Invert Emulsion Mud Using Adaptive Neuro-Fuzzy Inference System.

Tracking the rheological properties of the drilling fluid is a key factor for the success of the drilling operation. The main objective of this paper is to relate the most frequent mud measurements (every 15 to 20 min) as mud weight (MWT) and Marsh funnel viscosity (MFV) to the less frequent mud rheological measurements (twice a day) as plastic viscosity (PV), yield point (YP), behavior index (n), and apparent viscosity (AV) for fully automating the process of retrieving rheological properties. The adaptive neuro-fuzzy inference system (ANFIS) was used to develop new models to determine the mud rheological properties using real field measurements of 741 data points. The data were collected from 99 different wells during drilling operations of 12 ¼ inches section. The ANFIS clustering technique was optimized by using training to a testing ratio of 80% to 20% as 591 data points for training and 150 points, cluster radius value of 0.1, and 200 epochs. The results of the prediction models showed a correlation coefficient (R) that exceeded 0.9 between the actual and predicted values with an average absolute percentage error (AAPE) below 5.7% for the training and testing data sets. ANFIS models will help to track in real-time the rheological properties for invert emulsion mud that allows better control for the drilling operation problems.

Point and interval forecasting of solar irradiance with an active Gaussian process

A Gaussian process regression (GPR) with active learning is proposed for developing the solar irradiance point and interval forecasting models, which consider the spatial-temporal information collected from a targeted site and a number of neighbouring sites. To enhance the performance of the GPR-based model an active learning process is developed for constructing an ad-hoc input feature set, selecting training data points, and optimising hyper-parameters of GPR models. To validate the advantages of the proposed method, a comprehensive computational study is conducted based on solar irradiance data collected from the northwest California area. In the point forecasting, the proposed method beats the state-of-the-art benchmarking methods including classical statistical models and data-driven models according to values of the normalised root mean squared error, normalised mean absolute error, normalised mean bias error, and coefficient of determination. In the interval forecasting, the proposed method outperforms the persistence model, autoregressive model with exogenous inputs, generic GPR, as well as two recently reported forecasting methods, the bootstrap-based extreme learning machine and quantile regression, in terms of the forecasting reliability. Computational results show that the proposed method is more effective than well-known existing benchmarks in the point and interval forecasting of the solar irradiance.

Scaling description of generalization with number of parameters in deep learning

Supervised deep learning involves the training of neural networks with a large number N of parameters. For large enough N, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as N grows past a certain threshold N*. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with N. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations of the neural net output function f N around its expectation . These affect the generalization error for classification: under natural assumptions, it decays to a plateau value in a power-law fashion ∼N−1/2. This description breaks down at a so-called jamming transition N = N*. At this threshold, we argue that diverges. This result leads to a plausible explanation for the cusp in test error known to occur at N*. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond N*, and averaging their outputs.

Application of remote sensing techniques and machine learning algorithms in dust source detection and dust source susceptibility mapping

The aim of this research was to develop a method to produce a Dust Source Susceptibility Map (DSSM). For this purpose, we applied remote sensing and statistical-based machine learning algorithms for experimental dust storm studies in the Khorasan Razavi Province, in north-eastern Iran. We identified dust sources in the study area using MODIS satellite images during the 2005–2016 period. For dust source identification, four indices encompassing BTD3132, BTD2931, NDDI, and D variable for 23 MODIS satellite images were calculated. As a result, 65 dust source points were identified, which were categorized into dust source data points for training and validation of the machine learning algorithms. Three statistical-based machine learning algorithms were used including Weights of Evidence (WOE), Frequency Ratio (FR), and Random Forest (RF) to produce DSSM for the study region. We used land use, lithology, slope, soil, geomorphology, NDVI (Normalized Difference Vegetation Index), and distance from river as conditioning variables in the modelling. To check the performance of the models, we applied the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC). As for the AUC success rate (training), the FR and WOE algorithms resulted in 82 and 83% accuracy, respectively, while the RF algorithm resulted in 91% accuracy. As for the AUC predictive rate (validation), the accuracy of all three models, FR, WOE, and RF, were 80, 81, and 88%, respectively. Although all three algorithms produced acceptable susceptibility maps of dust sources, the results indicated better performance of the RF algorithm.

Robust Multi-Linear Fuzzy SVR Designed With the Aid of Fuzzy C-Means Clustering Based on Insensitive Data Information

Multiple SVR based on ensemble learning could be enhanced from the viewpoint of the performance, but the performance of modeling closely depends on the initial condition of the partitioning method and they are easily affected by noise and outliers. In this study, a multi-linear fuzzy support vector regression (MFSVR) robust to noise is proposed with the aid of the composite kernel function and $\varepsilon $ -fuzzy c-means (FCM) clustering based on insensitive data information. Here insensitive data information stands for the interval data information of “$\varepsilon $ ” which stands for insensitive loss parameter used in the $\varepsilon $ - insensitive loss function. The objective of this study is to reduce the effect of noise and to alleviate the overfitting problem through the synergistic effect of the following methods: First, $\varepsilon $ -FCM clustering based on insensitive data information is used for considering more impact on decision boundary and reducing the effect of noise. Second, the composite kernel based on multiple linear kernel expression is proposed for implementing multi-linear decision boundary to alleviate overfitting problem. In more detail, each training data point is assigned with corresponding membership degrees in the $\varepsilon $ -FCM clustering. Some data which are potentially to be noise or outlier are assigned with lower membership degrees and given small contribution (compensation) considered in composite kernel function. Then, the composite kernel function for multiple local SVRs is constructed according to the distribution characteristics of $\varepsilon $ -FCM clustering. The proposed MFSVR is tested with both synthetic and UCI data sets in order to verify the effectiveness as well as efficient performance improvement. Experimental results demonstrate that the proposed method shows the better performance when compared to other some methods studied so far.

Modeling of Chromium (VI) adsorption on limonia acidissima hull powder using Artificial Neural Network (ANN) approach

Batch experiment was carried out for the removal of Chromium (VI) from aqueous solution to get experimental data to which an Artificial Neural Network (ANN) model was developed using 16 experimental data points (for testing) and 36 experimental data points (for training). A single layer feed forward back propagation was used to get minimum mean square error with eleven neurons in hidden layer. To develop ANN model, a tan sigmoid transfer function for input and purelin for output layers were used. The optimized process parameters viz., adsorbent dosage, pH and initial concentration of chromium (VI) were obtained along with the maximum percentage removal of Chromium (VI). The predicted ANN model data were in perfect match with the experimental data based on the regression coefficient R2 of 0.995. The optimized parameters along with maximum percentage removal of chromium (VI) were achieved by means of pattern search method in Genetic Algorithm.

Machine learning model for dynamical response of nano-composite pipe conveying fluid under seismic loading

Machine learning approaches including support vector regression (SVR) and multi-layer feedforward backpropagation neural network (FFBNN) were used in the present study along with classic theory for predicting maximum displacement of nanocomposite pipe conveying fluid under seismic load. The FFBNN consisted of three layers: 1) three neurons in input layer including length-to-radius ratio (L/R), fluid velocity (V) and volume percent of carbon nanotube; 2) hidden layer with 11 neurons obtained via trial and error; 3) maximum displacement-based seismic load. SVR model was obtained via three-input data with maximum likelihood estimator. Model predicted results were compared using three metrics, including Nash-Sutcliffe efficiency, root mean squared error and coefficient of correlation for 100 testing and 255 training data points. Results indicated that SVR achieved best predictions in the training phase, while FFBNN provided superior prediction in the testing phase. Increasing L/R, V and decreasing VCNT, increased maximum displacements under seismic load.

A new formulation for symbolic regression to identify physico-chemical laws from experimental data

A modification to the mixed-integer nonlinear programming (MINLP) formulation for symbolic regression was proposed with the aim of identification of physical models from noisy experimental data. In the proposed formulation, a binary tree in which equations are represented as directed, acyclic graphs, is fully constructed for a pre-defined number of layers. The introduced modification results in the reduction in the number of required binary variables and removal of redundancy due to possible symmetry of the tree formulation. The formulation was tested using numerical models and was found to be more efficient than the previous literature example with respect to the numbers of predictor variables and training data points. The globally optimal search was extended to identify physical models and to cope with noise in the experimental data predictor variable. The methodology was proven to be successful in identifying the correct physical models describing the relationship between shear stress and shear rate for both Newtonian and non-Newtonian fluids, and simple kinetic laws of chemical reactions. Future work will focus on addressing the limitations of the present formulation and solver to enable extension of target problems to larger, more complex physical models.

Regression Clustering for Improved Accuracy and Training Costs with Molecular-Orbital-Based Machine Learning.

Machine learning (ML) in the representation of molecular-orbital-based (MOB) features has been shown to be an accurate and transferable approach to the prediction of post-Hartree-Fock correlation energies. Previous applications of MOB-ML employed Gaussian Process Regression (GPR), which provides good prediction accuracy with small training sets; however, the cost of GPR training scales cubically with the amount of data and becomes a computational bottleneck for large training sets. In the current work, we address this problem by introducing a clustering/regression/classification implementation of MOB-ML. In the first step, regression clustering (RC) is used to partition the training data to best fit an ensemble of linear regression (LR) models; in the second step, each cluster is regressed independently, using either LR or GPR; and in the third step, a random forest classifier (RFC) is trained for the prediction of cluster assignments based on MOB feature values. Upon inspection, RC is found to recapitulate chemically intuitive groupings of the frontier molecular orbitals, and the combined RC/LR/RFC and RC/GPR/RFC implementations of MOB-ML are found to provide good prediction accuracy with greatly reduced wall-clock training times. For a data set of thermalized (350 K) geometries of 7211 organic molecules of up to seven heavy atoms (QM7b-T), both RC/LR/RFC and RC/GPR/RFC reach chemical accuracy (1 kcal/mol prediction error) with only 300 training molecules, while providing 35000-fold and 4500-fold reductions in the wall-clock training time, respectively, compared to MOB-ML without clustering. The resulting models are also demonstrated to retain transferability for the prediction of large-molecule energies with only small-molecule training data. Finally, it is shown that capping the number of training data points per cluster leads to further improvements in prediction accuracy with negligible increases in wall-clock training time.