Regression Datasets Research Articles

Deep stochastic configuration networks with different random sampling strategies

Stochastic configuration networks (SCNs) are a class of randomized learner models that ensure the universal approximation property, whereby random weights and biases are drawn from the uniform distribution and selected by a supervisory mechanism. This paper looks into the impact of the distribution of random weights on the performance of SCNs. In the light of a fundamental principle in machine learning, that is, a model with smaller parameters holds improved generalization, we recommend using symmetric zero-centered distributions in constructing SCNs to improve the generalization performance. Further, we introduce a scalar in the distributions to make the SCN model adaptively feasible to different datasets. Simulation results are reported for both regression and classification tasks over twenty-one benchmark datasets using SCN. Results are also presented on ten regression datasets using a deep implementation of SCN, known as deep stochastic configuration networks (DeepSCN).

Heterogeneous hybrid extreme learning machine for temperature sensor accuracy improvement

High-accuracy temperature sensors usually come at a high cost, making them cost-ineffective solutions for mass-produced commercial products. The overall product cost can be one of the decisive factors for its success (e.g., commercial freezers). On the other hand, low-cost temperature sensors are viable options, but they suffer from accuracy issues. This paper presents a system aimed at improving the accuracy of a low-cost temperature sensor using a hybrid machine learning algorithm. The system utilizes a low-cost single-board computer connected to a series of sensors to collect data. These data are then used as an input vector to the proposed linear regression heterogeneous hybrid extreme learning machine (LR-HE-HyELM), a supervised learning algorithm responsible for increasing the sensors’ accuracy. The proposed linear regression heterogeneous extreme learning machine (LR-HE-HyELM) combines linear regression with the self-adaptive heterogeneous hybrid extreme learning machine (SA-He-HyELM) algorithm. SA-He-HyELM is an extreme learning machine (ELM) based algorithm that utilizes a genetic algorithm (GA) for the creation of an optimized single-layer neural network (SLNN) having different combinations of custom-created neurons at its hidden layer. The proposed training algorithm has all the advantages of SA-He-HyELM. These advantages include utilizing low-order and higher-order neurons in the same network while retaining ELM’s simplicity by making all GA parameters self-adaptive. The utilization of low-order and higher-order unit types in the same network resulted in optimized neural networks with better generalization ability than those with the same neuron types in their hidden layer. The creation of SA-He-HyELM was nature-inspired since the human brain contains different neuron types. The proposed system is tested with eight existing methods, with most of them having different neuron types in the hidden layer. The experimental results involved using a custom-created regression dataset having measurements taken from a commercial fridge. The proposed method managed to get the lowest mean square error (MSE) compared to the other eight methods.

Transfer learning based on semantic approximation techniques in genetic programming

Transfer learning aims to reuse the knowledge taken from different but related source domains to enhance learning performance on target domains. In Genetic Programming (GP), transfer learning methods have helped GP systems expand their applicability in real applications. In this work, two transfer learning methods based on semantic approximation techniques (SAT) for GP are proposed. The proposed methods use the best individuals in the final generation of the source task as transfer material to the target task. Before transferring, SAT is applied on these individuals in order to simultaneously satisfy two objectives of reducing the size of the transferred individuals and preserving the knowledge of them. The methods are tested on ten symbolic regression datasets and compared with GP without transfer learning and two GP transfer learning methods. Experimental results showed that the proposed methods could effectively extract more useful knowledge from source domains to help to improve the performance of GP systems on the target domains.

Features Selection as a Nash-Bargaining Solution: Applications in Online Advertising and Information Systems

Feature selection is a fundamental problem in online advertising, as features usually need to be purchased from third parties, and they are costly. Although many feature selection techniques can be used in online advertising and the general information systems (IS) domain, their performance is often context specific. Therefore, the literature of IS is suffering from a lack of adequate and generic methods. In this study, we address this issue by proposing a novel approach that employs ideas from the field of cooperative game theory. We derive a (continuous) second-order cone program that any convex programming solver can solve for determining the best subset of features. We show the efficacy of our proposed method on a real-life online advertising case study. We demonstrate that our proposed approach performs better in accuracy, precision, recall, and F-1 score than the best of the other approaches with much fewer features. Also, to illustrate that our method’s benefits are not limited to the context of online advertising, we perform an extensive set of simulations and consider a well-established real-life data set drawn from the UCI Machine Learning Repository at the University of California, Irvine. Summary of Contribution: Selecting the best subset of features is an important problem in the context of online advertising and, more broadly, in the field of information systems because firms usually need to buy costly data to model and forecast economic outcomes. In this study, we propose a novel methodology for addressing this problem. The proposed method employs the concept of the Nash bargaining solution in cooperative game theory to create a good balance between maximizing the fit while minimizing the noise when selecting the best subset of features. We apply the method to a real-life online advertising case study, providing superior performance in predicting and interpreting the features. Moreover, we show that the proposed method applies to a broader range of feature selection problems. We conduct a comprehensive computational study on simulated regression data sets and other real-life classification data sets widely available in the machine learning domain. The result of these efforts indicates that our method is robust in terms of prediction accuracy by outperforming several state-of-the-art techniques.

Antenna Optimization Based on Auto-Context Broad Learning System

To enhance the efficiency of antenna optimization, surrogate model methods can usually be used to replace the full-wave electromagnetic simulation software. Broad learning system (BLS), as an emerging network with strong extraction ability and remarkable computational efficiency, has revolutionized the conventional artificial intelligence (AI) methods and overcome the shortcoming of excessive time-consuming training process in deep learning (DL). However, it is difficult to model the regression relationship between input and output variables in the electromagnetic field with the unsatisfactory fitting capability of the original BLS. In order to further improve the performance of the model and speed up the design of microwave components to achieve more accurate prediction of hard-to-measure quality variables through easy-to-measure parameter variables, the conception of auto-context (AC) for the regression scenario is proposed in this paper, using the current BLS training results as the prior knowledge, which are taken as the context information and combined with the original inputs as new inputs for further training. Based on the previous prediction results, AC learns an iterated low-level and context model and then iterates to approach the ground truth, which is very general and easy to implement. Three antenna examples, including rectangular microstrip antenna (RMSA), circular MSA (CMSA), and printed dipole antenna (PDA), and 10 UCI regression datasets are employed to verify the effectiveness of the proposed model.

Linear solvation energy relationships (LSERs) for robust prediction of partition coefficients between low density polyethylene and water. Part I: Experimental partition coefficients and model calibration

When equilibrium of leaching is reached within a product's duty cycle, partition coefficients polymer/solution dictate the maximum accumulation of a leachable and thus, patient exposure by leachables. Yet, in the pharmaceutical and food industry, exposure estimates based on predictive modeling typically rely on coarse estimations of the partition coefficient, with accurate and robust models lacking. This first part of the study aimed to investigate linear solvation energy relationships (LSERs) as high performing models for the prediction of partition coefficients polymer/water. For this, partition coefficients between low density polyethylene (LDPE) and aqueous buffers for 159 compounds spanning a wide range of chemical diversity, molecular weight, vapor pressure, aqueous solubility and polarity (hydrophobicity) were determined and complimentary data collected from the literature (n=159, MW: 32 to 722, logKi,O/W: -0.72 to 8.61 and logKi,LDPE/W: -3.35 up to 8.36). The chemical space represented by this compounds set is considered indicative for the universe of compounds potentially leaching from plastics. Based on the dataset for the LDPE material purified by solvent extraction, a LSER model for partitioning between LDPE and water was calibrated to give:logKi,LDPE/W=−0.529+1.098Ei−1.557Si−2.991Ai−4.617Bi+3.886Vi. The model was proven accurate and precise (n = 156, R2 = 0.991, RMSE = 0.264). Further, it was demonstrated superior over a log-linear model fitted to the same data. Nonetheless, it could be shown that log-linear correlations against logKi,O/W can be of value for the estimation of partition coefficients for nonpolar compounds exhibiting low hydrogen-bonding donor and/or acceptor propensity. For nonpolar compounds, the log - linear model was found as: logKi,LDPE/W=1.18logKi,O/W−1.33 (n = 115, R2=0.985, RMSE=0.313). In contrast, with mono-/bipolar compounds included into the regression data set, an only weak correlation was observed (n= 156, R2 = 0.930, RMSE = 0.742) rendering the log-linear model of more limited value for polar compounds. Notably, sorption of polar compounds into pristine (non-purified) LDPE was found to be up to 0.3 log units lower than into purified LDPE. To identify maximum (i. e. worst-case) levels of leaching in support of chemical safety risk assessments on systems attaining equilibrium before end of shelf-life, it appears adequate to utilize LSER - calculated partition coefficients (in combination with solubility data) by ignoring any kinetical information.

Mixture Correntropy-Based Kernel Extreme Learning Machines.

Kernel-based extreme learning machine (KELM), as a natural extension of ELM to kernel learning, has achieved outstanding performance in addressing various regression and classification problems. Compared with the basic ELM, KELM has a better generalization ability owing to no needs of the number of hidden nodes given beforehand and random projection mechanism. Since KELM is derived under the minimum mean square error (MMSE) criterion for the Gaussian assumption of noise, its performance may deteriorate under the non-Gaussian cases, seriously. To improve the robustness of KELM, this article proposes a mixture correntropy-based KELM (MC-KELM), which adopts the recently proposed maximum mixture correntropy criterion as the optimization criterion, instead of using the MMSE criterion. In addition, an online sequential version of MC-KELM (MCOS-KELM) is developed to deal with the case that the data arrive sequentially (one-by-one or chunk-by-chunk). Experimental results on regression and classification data sets are reported to validate the performance superiorities of the new methods.

Robust Variable Selection Based on Schwarz Information Criterion for Linear Regression Models

Schwarz information criterion (SIC) is a popular tool to select the best variables in regression data sets. However, SIC defined using an unbounded estimator (Least Squares (LS)) which is very sensitive to the presence of outlying observations, especially bad leverage points. Thus, robust variable selection based on SIC for linear regression models is in need. This paper study the robust properties of SIC derives its influence function and proposes robust SIC based on the MM-estimation scale, aim to produce criterion which is effective in selecting accurate models in the presence of vertical outliers and high leverage points. The advantages of the proposed robust SIC is demonstrated through simulation study and analysis of a real data set.

Evaluating regression algorithms at the instance level using item response theory

Algorithm evaluation is a very important task for different Machine Learning (ML) problems. In this work, we assume a different perspective in ML evaluation, in which algorithms are evaluated at the instance level. In this perspective, Item Response Theory (IRT) has recently been applied to algorithm evaluation in ML in order to identify which instances are more difficult and discriminating in a dataset, while also evaluating algorithms based on their predictions for instances with different difficulty values. In IRT, a strong algorithm returns accurate predictions for the most difficult instances, while maintaining a consistent behaviour in the easiest instances. The most common IRT models adopted in the literature only deal with dichotomous responses (i.e., a response has to be either correct or incorrect). This is suitable for evaluating classification algorithms, but not adequate in application contexts where responses are recorded in a continuous scale without an upper bound, such as regression. In this paper we propose the Γ-IRT model, particularly designed for dealing with positive unbounded responses, which we model using a Gamma distribution, parameterised according to respondent ability and item difficulty and discrimination parameters. The proposed parameterisation results in item characteristic curves with more flexible shapes compared to the traditional logistic curves adopted in IRT. We apply the proposed model to evaluate student responses (number of errors) in open-ended questions extracted from Statistics exams. Then, we use Γ-IRT to assess regression model abilities, where responses are the absolute errors in test instances. This novel application represents an alternative for evaluating regression performance and for identifying regions in a regression dataset that present different levels of difficulty and discrimination.

Aggregation as Unsupervised Learning and its Evaluation

Regression uses supervised machine learning to find a model that combines several independent variables to predict a dependent variable based on ground truth (labeled) data, i.e., tuples of independent and dependent variables (labels). Similarly, aggregation also combines several independent variables to a dependent variable. The dependent variable should preserve properties of the independent variables, e.g., the ranking or relative distance of the independent variable tuples, and/or represent a latent ground truth that is a function of these independent variables. However, ground truth data is not available for finding the aggregation model. Consequently, aggregation models are data agnostic or can only be derived with unsupervised machine learning approaches. We introduce a novel unsupervised aggregation approach based on intrinsic properties of unlabeled training data, such as the cumulative probability distributions of the single independent variables and their mutual dependencies. We present an empirical evaluation framework that allows assessing the proposed approach against other aggregation approaches from two perspectives: (i) how well the aggregation output represents properties of the input tuples, and (ii) how well can aggregated output predict a latent ground truth. To this end, we use data sets for assessing supervised regression approaches that contain explicit ground truth labels. However, the ground truth is not used for deriving the aggregation models, but it allows for the assessment from a perspective (ii). More specifically, we use regression data sets from the UCI machine learning repository and benchmark several data-agnostic and unsupervised approaches for aggregation against ours. The benchmark results indicate that our approach outperforms the other data-agnostic and unsupervised aggregation approaches. It is almost on par with linear regression.

Improving machine learning performance on small chemical reaction data with unsupervised contrastive pretraining.

Machine learning (ML) methods have great potential to transform chemical discovery by accelerating the exploration of chemical space and drawing scientific insights from data. However, modern chemical reaction ML models, such as those based on graph neural networks (GNNs), must be trained on a large amount of labelled data in order to avoid overfitting the data and thus possessing low accuracy and transferability. In this work, we propose a strategy to leverage unlabelled data to learn accurate ML models for small labelled chemical reaction data. We focus on an old and prominent problem—classifying reactions into distinct families—and build a GNN model for this task. We first pretrain the model on unlabelled reaction data using unsupervised contrastive learning and then fine-tune it on a small number of labelled reactions. The contrastive pretraining learns by making the representations of two augmented versions of a reaction similar to each other but distinct from other reactions. We propose chemically consistent reaction augmentation methods that protect the reaction center and find they are the key for the model to extract relevant information from unlabelled data to aid the reaction classification task. The transfer learned model outperforms a supervised model trained from scratch by a large margin. Further, it consistently performs better than models based on traditional rule-driven reaction fingerprints, which have long been the default choice for small datasets, as well as those based on reaction fingerprints derived from masked language modelling. In addition to reaction classification, the effectiveness of the strategy is tested on regression datasets; the learned GNN-based reaction fingerprints can also be used to navigate the chemical reaction space, which we demonstrate by querying for similar reactions. The strategy can be readily applied to other predictive reaction problems to uncover the power of unlabelled data for learning better models with a limited supply of labels.

BoostTree and BoostForest for Ensemble Learning.

Bootstrap aggregating (Bagging) and boosting are two popular ensemble learning approaches, which combine multiple base learners to generate a composite model for more accurate and more reliable performance. They have been widely used in biology, engineering, healthcare, etc. This article proposes BoostForest, which is an ensemble learning approach using BoostTree as base learners and can be used for both classification and regression. BoostTree constructs a tree model by gradient boosting. It increases the randomness (diversity) by drawing the cut-points randomly at node splitting. BoostForest further increases the randomness by bootstrapping the training data in constructing different BoostTrees. BoostForest generally outperformed four classical ensemble learning approaches (Random Forest, Extra-Trees, XGBoost and LightGBM) on 35 classification and regression datasets. Remarkably, BoostForest tunes its parameters by simply sampling them randomly from a parameter pool, which can be easily specified, and its ensemble learning framework can also be used to combine many other base learners.

Effective band selection of hyperspectral image by an attention mechanism-based convolutional network.

The selection of effective and representative spectral bands is extremely important in eliminating redundant information and reducing the computational burden for the potential real-time applications of hyperspectral imaging. However, current band selection methods act as a separate procedure before model training and are implemented merely based on extracted average spectra without incorporating spatial information. In this paper, an end-to-end trainable network framework that combines band selection, feature extraction, and model training was proposed based on a 3D CNN (convolutional neural network, CNN) with the attention mechanism embedded in its first layer. The learned band attention vector was adopted as the basis of a band importance indicator to select effective bands. The proposed network was evaluated by two datasets, a regression dataset for predicting the relative chlorophyll content (soil and plant analyzer development, SPAD) of basil leaves and a classification dataset for detecting the drought stress of pepper leaves. A number of calibration models, including SVM, 1D-CNN, 2B-CNN (two-branch CNN), 3D ResNet and the developed network were established for performance comparison. Results showed that the effective bands selected by the proposed attention-based model achieved higher regression R2 values and classification accuracies not only than the full-spectrum data, but also than the comparative band selection methods, including traditional SPA (successive projections algorithm) and GA (genetic algorithm) methods and the latest 2B-CNN algorithm. In addition, different from the traditional methods, the proposed band selection algorithm can effectively select bands while carrying out model training and can simultaneously take advantage of the original spectral–spatial information. The results confirmed the usefulness of the proposed attention mechanism-based convolutional network for selecting the most effective band combination of hyperspectral images.

Analytical Predictive Modeling: Impact of Financial and Economic Indicators on Stock

Stock price prediction is considered as an important task and is of great attention as predicting stock prices successfully may lead to attractive profits for investors. Information Technology Sector of S&P 500 is one of the most sought after business segments in S&P 500 and is one of the most attracting areas for many investors due to high percentage annual returns on investment over the years. We used Microsoft Corp. (MSFT), one of the leading companies of the Information Technology Sector Index of S&P 500 information to build a non-linear real data-driven analytical model which accurately predicts the Weekly Closing Price (WCP) of the stock with predictive accuracy of 99.3% using six financial, four economic indicators and their two way interactions as the attributable entities that drive the stock returns. We rank the statistically significant indicators and their interactions based on the percentage of contribution to the WCP of the stock that provides significant information for the beneficiary of the proposed predictive model. We present a unique way for feature selection when multicollinearity presents in the multiple regression dataset using L1-regularization based on supervised machine learning algorithm.

Permute Me Softly: Learning Soft Permutations for Graph Representations.

Graph neural networks (GNNs) have recently emerged as a dominant paradigm for machine learning with graphs. Research on GNNs has mainly focused on the family of message passing neural networks (MPNNs). Similar to the Weisfeiler-Leman (WL) test of isomorphism, these models follow an iterative neighborhood aggregation procedure to update vertex representations, and they next compute graph representations by aggregating the representations of the vertices. Although very successful, MPNNs have been studied intensively in the past few years. Thus, there is a need for novel architectures which will allow research in the field to break away from MPNNs. In this paper, we propose a new graph neural network model, so-called π-GNN which learns a "soft" permutation (i. e., doubly stochastic) matrix for each graph, and thus projects all graphs into a common vector space. The learned matrices impose a "soft" ordering on the vertices of the input graphs, and based on this ordering, the adjacency matrices are mapped into vectors. These vectors can be fed into fully-connected or convolutional layers to deal with supervised learning tasks. In case of large graphs, to make the model more efficient in terms of running time and memory, we further relax the doubly stochastic matrices to row stochastic matrices. We empirically evaluate the model on graph classification and graph regression datasets and show that it achieves performance competitive with state-of-the-art models.

AddGBoost: A gradient boosting-style algorithm based on strong learners

We present AddGBoost, a gradient boosting-style algorithm, wherein the decision tree is replaced by a succession of (possibly) stronger learners, which are optimized via a state-of-the-art hyperparameter optimizer. Through experiments over 90 regression datasets we show that AddGBoost emerges as the top performer for 33% (with 2 stages) up to 42% (with 5 stages) of the datasets, when compared with seven well-known machine-learning algorithms: KernelRidge, LassoLars, SGDRegressor, LinearSVR, DecisionTreeRegressor, HistGradientBoostingRegressor, and LGBMRegressor.

Stochastic configuration networks with fast implementations.

Stochastic configuration networks (SCNs) employ a supervisory mechanism to assignhidden-node parameters in the incremental construction process. SCNs offer the advantages of practical implementation, fast convergence, and better generalization performance. However, due to its high computational cost and the scalability of numerical algorithms for the least square technique, it is rather limited for dealing with enormous amounts of data. This paper proposes fast SCNs (F-SCNs), whose output weights are determined using orthogonal matrix Q and upper triangular matrix R decomposition. The network can iteratively update the output weights utilizing the output information from the predecessor node using this incremental technique. We investigated the computational complexity of SCNs and F-SCNs and demonstrated that F-SCNs are suitable for scenarios in which the hidden layer has a significant number of nodes. We evaluated the proposed method on four real-world regression datasets; experimental results show that our method has notable advantages in terms of speed and effectiveness of learning.

Using meta-learning for multi-target regression

Choosing the most suitable algorithm to perform a machine learning task for a new problem is a recurrent and complex task. In multi-target regression tasks, when problem transformation methods are applied, this choice is even harder. The reason is the need to simultaneously choose the problem transformation method and the base learning algorithm. This work investigates how to bridge the gap of method/base learner recommendation for problems with multiple outputs. In meta-learning experiments, we use a large number of multi-target regression datasets to investigate whether using meta-learning can provide good recommendations. To do this, we compared the meta-models induced by 3 different ML algorithms, including three variations for each of them, and selected 58 meta-features that we believe are relevant for extracting good dataset descriptions for the meta-learning process. In the experimental results, the meta-models outperformed the baselines (Majority and Random) by recommending the most suitable solution for multi-target regression (for the transformation method and base-learner) with high predictive performance, including real-world applications. The meta-features and the relation between the transformation method and base-learner provided important insights regarding the optimal problem transformation method. Furthermore, when comparing the application of algorithm adaptation and problem transformation methods, our meta-learning proposal was capable of statistically overcoming all competitors, which resulted in a predictive performance using the best choice per problem.

Evaluating High-Variance Leaves as Uncertainty Measure for Random Forest Regression.

Uncertainty measures estimate the reliability of a predictive model. Especially in the field of molecular property prediction as part of drug design, model reliability is crucial. Besides other techniques, Random Forests have a long tradition in machine learning related to chemoinformatics and are widely used. Random Forests consist of an ensemble of individual regression models, namely, decision trees and, therefore, provide an uncertainty measure already by construction. Regarding the disagreement of single-model predictions, a narrower distribution of predictions is interpreted as a higher reliability. The standard deviation of the decision tree ensemble predictions is the default uncertainty measure for Random Forests. Due to the increasing application of machine learning in drug design, there is a constant search for novel uncertainty measures that, ideally, outperform classical uncertainty criteria. When analyzing Random Forests, it appears obvious to consider the variance of the dependent variables within each terminal decision tree leaf to obtain predictive uncertainties. Hereby, predictions that arise from more leaves of high variance are considered less reliable. Expectedly, the number of such high-variance leaves yields a reasonable uncertainty measure. Depending on the dataset, it can also outperform ensemble uncertainties. However, small-scale comparisons, i.e., considering only a few datasets, are insufficient, since they are more prone to chance correlations. Therefore, large-scale estimations are required to make general claims about the performance of uncertainty measures. On several chemoinformatic regression datasets, high-variance leaves are compared to the standard deviation of ensemble predictions. It turns out that high-variance leaf uncertainty is meaningful, not superior to the default ensemble standard deviation. A brief possible explanation is offered.

Residual's influence index (RINFIN), bad leverage and unmasking in high dimensional L2‐regression

AbstractIn linear regression of Y on X(∈ Rp) with parameters β(∈ Rp+1), statistical inference is unreliable when observations are obtained from gross‐error model, Fϵ,G = (1 − ϵ)F + ϵG, instead of the assumed probability F;G is gross‐error probability, 0 < ϵ < 1. Residual's influence index (RINFIN) at (x, y) is introduced, with components measuring also the local influence of x in the residual and large value flagging a bad leverage case (from G), thus causing unmasking. Large sample properties of RINFIN are presented to confirm significance of the findings, but often the large difference in the RINFIN scores of the data is indicative. RINFIN is successful with microarray data, simulated, high dimensional data and classic regression data sets. RINFIN's performance improves as p increases and can be used in multiple response linear regression.