Regression Datasets Research Articles

Improved Crow Search Algorithm Optimized Extreme Learning Machine Based on Classification Algorithm and Application

In view of the problems of the connection weights and thresholds of the extreme learning machine are randomly generated before training and remain unchanged during the training process, the number of hidden layer nodes is pre-allocated, and the hidden layer parameters are randomly selected. Too many hidden layer nodes not only make the network more complex but also reduce the generalization ability of the algorithm. Aiming at this problem, an improved crow search algorithm is proposed to optimize the extreme learning machine. Based on the analysis of the limitations of the original crow search algorithm, a particle swarm algorithm search strategy is proposed to enhance the global search capability. In the latter part of the algorithm iteration, Gaussian function is added, and the penalty coefficient of the function is used for local disturbance, gradually reducing the amplitude of the search trajectory, and then adaptively adjusting the parameters to avoid being attracted by local extremum. Finally, the improved crow search algorithm is used to optimize the hidden layer neurons and connection weights of the extreme learning machine neural network, so as to obtain accurate prediction results. Through function fitting, regression data set fitting and classification data set for classification experiment verification, the proposed algorithm has higher training speed and efficiency. At the same time, this method is not only significantly higher than the traditional ELM method, but also obtains a more compact network structure, which is an effective neural network optimization algorithm.

Single-Index Model Tree

In this paper, a novel single-index model tree (SIMTree) is proposed. It adopts the recursive partitioning strategy and each data segment is modeled by a single-index model (SIM), which is a flexible extension of linear regression with non-parametric link functions. The proposed SIMTree has 2 major advantages: a) with only a few leaf nodes, it can achieve competitive predictive performance compared to complicated black-box models; b) SIMs fitted on each local data segment are intrinsically interpretable. However, using conventional techniques to build such a SIMTree can be extremely time-consuming. SIM estimation typically involves iterative optimization via Newton-type algorithms; such a resource-intensive estimation procedure is repeatedly used for fitting leaf node SIMs and the search of optimal splits. To make the computation burden affordable, an effective training algorithm is proposed as enabled by the efficient utilization of Stein's lemma and several accelerating strategies in the tree construction algorithm. Moreover, a new Python package simtree is developed with elegant visualization modules that can further facilitate the model interpretation. Numerical results on extensive regression datasets show that SIMTree is an accurate and interpretable machine learning model.

Incremental DC optimization algorithm for large-scale clusterwise linear regression

The objective function in the nonsmooth optimization model of the clusterwise linear regression (CLR) problem with the squared regression error is represented as a difference of two convex functions. Then using the difference of convex algorithm (DCA) approach the CLR problem is replaced by the sequence of smooth unconstrained optimization subproblems. A new algorithm based on the DCA and the incremental approach is designed to solve the CLR problem. We apply the Quasi-Newton method to solve the subproblems. The proposed algorithm is evaluated using several synthetic and real-world data sets for regression and compared with other algorithms for CLR. Results demonstrate that the DCA based algorithm is efficient for solving CLR problems with the large number of data points and in particular, outperforms other algorithms when the number of input variables is small.

A Memory-Efficient Encoding Method for Processing Mixed-Type Data on Machine Learning.

The most common machine-learning methods solve supervised and unsupervised problems based on datasets where the problem’s features belong to a numerical space. However, many problems often include data where numerical and categorical data coexist, which represents a challenge to manage them. To transform categorical data into a numeric form, preprocessing tasks are compulsory. Methods such as one-hot and feature-hashing have been the most widely used encoding approaches at the expense of a significant increase in the dimensionality of the dataset. This effect introduces unexpected challenges to deal with the overabundance of variables and/or noisy data. In this regard, in this paper we propose a novel encoding approach that maps mixed-type data into an information space using Shannon’s Theory to model the amount of information contained in the original data. We evaluated our proposal with ten mixed-type datasets from the UCI repository and two datasets representing real-world problems obtaining promising results. For demonstrating the performance of our proposal, this was applied for preparing these datasets for classification, regression, and clustering tasks. We demonstrate that our encoding proposal is remarkably superior to one-hot and feature-hashing encoding in terms of memory efficiency. Our proposal can preserve the information conveyed by the original data.

Do Randomized Algorithms Improve the Efficiency of Minimal Learning Machine?

Minimal Learning Machine (MLM) is a recently popularized supervised learning method, which is composed of distance-regression and multilateration steps. The computational complexity of MLM is dominated by the solution of an ordinary least-squares problem. Several different solvers can be applied to the resulting linear problem. In this paper, a thorough comparison of possible and recently proposed, especially randomized, algorithms is carried out for this problem with a representative set of regression datasets. In addition, we compare MLM with shallow and deep feedforward neural network models and study the effects of the number of observations and the number of features with a special dataset. To our knowledge, this is the first time that both scalability and accuracy of such a distance-regression model are being compared to this extent. We expect our results to be useful on shedding light on the capabilities of MLM and in assessing what solution algorithms can improve the efficiency of MLM. We conclude that (i) randomized solvers are an attractive option when the computing time or resources are limited and (ii) MLM can be used as an out-of-the-box tool especially for high-dimensional problems.

Non-parallel hyperplanes ordinal regression machine

This paper proposes a method to solve ordinal regression problems, namely the non-parallel hyperplanes ordinal regression machine (NPHORM). The goal of this approach is to find K different hyperplanes for the K classes with ordinal information, so that each class is as close as possible to the corresponding hyperplane while as far as possible from the adjacent to the left and right classes. The more flexible separate hyperplanes are preferred using the order information of the data. As a result, this approach only needs to solve K quadratic programming problems independently. Our approach NPHORM is validated on 2 artificial datasets, 16 discretized regression datasets and 17 real ordinal regression datasets and compared with 8 outstanding SVM-based ordinal regression approaches. The results show that our approach NPHORM is comparable with the other SVM-based approaches, especially in real ordinal regression datasets. In addition, our NPHORM is also carried out on the historical color image dataset to compare the performance of deep learning method. Experimental results demonstrate that the performance of our NPHORM outperforms the deep learning methods on MAE.

Assessing the distribution of air pollution health risks within cities: a neighborhood-scale analysis leveraging high resolution datasets in the Bay Area, California.

Background: Air pollution-attributable disease burdens are typically reported at country, state, or county levels, masking potential smaller scale geographic heterogeneity driven by non-uniform pollution levels and disease rates. Estimating the variation in air pollution health impacts within cities is now possible with high resolution pollutant concentrations derived from satellite remote sensing, modeling, and mobile monitoring. Objectives: We quantify neighborhood-level variation in air pollution health risks, using highly spatially resolved pollutant and disease rate datasets available for the Bay Area, California. Methods: We estimate mortality and morbidity attributable to nitrogen dioxide (NO2), black carbon (BC), and fine particulate matter (PM2.5) in the Bay Area using epidemiologically derived health impact functions. We compare geographic distributions of pollution-attributable risk estimates using pollutant concentrations from different sources: (1) mobile monitoring of NO2 and BC, covering roadways over a one-year period; and (2) models predicting annual NO2, BC and PM2.5 concentrations from land use variables and satellite observations, with full spatial coverage. We also compare the application of county versus census block group (CBG) level disease rates. Results: Across Bay Area counties, estimated pollution-attributable mortality rates (attributable deaths per 100,000 people) ranged by a factor 15, 13 and 8 for NO2, BC and PM2.5 (NO2 range=40-570, median=219; BC range=3-40, median=13; PM2.5 range=60-470, median=305). Applying concentrations from mobile monitoring and land use regression datasets resulted in similar spatial patterns of estimated NO2-attributable mortality rates, though the mobile monitoring concentrations yielded more spatial heterogeneity. Using CBG disease rates revealed spatial heterogeneity in estimated NO2 and PM2.5-attributable mortality rates that is masked when using county-level disease rates. Discussion: Estimated air pollutant attributable health burdens vary considerably between neighborhoods in the Bay Area, driven by spatial heterogeneity in both concentrations and disease rates. Future research should test the generalizability of these findings for other cities.

A hybrid fuzzy feature selection algorithm for high-dimensional regression problems: An mRMR-based framework

One of the most important factors affecting the interpretability of Fuzzy Rule-Based Systems (FRBSs) is the number of features used Indeed, the employment of a large number of features could be problematic for the components of an FRBS. In these cases,feature selection approaches can be employed to discover the most beneficial features. In this study, we present HFFS a Hybrid Fuzzy Feature Selection algorithm for high-dimensional regression problems. It benefits from both filter and wrapper methods. HFFS is composed of two main components: a filter-based Selector and a wrapper-based Modifier. Selector is an mRMR-based framework that sequentially selects the most informative features and adds them to a candidate subset. However, since these features may not result in an efficient FRBS, they are evaluated by Modifier, which fine-tunes them whenever their overall performance declines. This procedure is repeated until no further progress is possible within the candidate subset, acting as the stopping point for the algorithm. The effectiveness of HFFS was evaluated using twenty-eight real-world regression datasets and four different types of FRBSs. The experimental results and statistical tests confirmed that HFFS was able to improve the estimation accuracy compared to the filter methods and reduce the computation times compared to the wrapper methods.

Solving large-scale support vector ordinal regression with asynchronous parallel coordinate descent algorithms

Ordinal regression is one of the most influential tasks of supervised learning. Support vector ordinal regression (SVOR) is an appealing method to tackle ordinal regression problems. However, due to the complexity in the formulation of SVOR and the high cost of kernel computation, traditional SVOR solvers are inefficient for large-scale training. To address this problem, in this paper, we first highlight a special SVOR formulation whose thresholds are described implicitly, so that the dual formulation is concise to apply the state-of-the-art asynchronous parallel coordinate descent algorithm, such as AsyGCD. To further accelerate the training for SVOR, we propose two novel asynchronous parallel coordinate descent algorithms, called AsyACGD and AsyORGCD respectively. AsyACGD is an accelerated extension of AsyGCD using active set strategy. AsyORGCD is specifically designed for SVOR that it can keep the ordered thresholds when it is training so that it can obtain good performance with lower time. Experimental results on several large-scale ordinal regression datasets demonstrate the superiority of our proposed algorithms.

MINE: A framework for dynamic regressor selection

Dynamic Regressor Selection (DRS) techniques aim to select the most competent regressors from an ensemble per test pattern. So, for each test pattern, only a subset of the most competent regressors are used to estimate its target value. Hence, the central issue in DRS techniques is how to define the competence of the regressors which is usually defined using a single measure, such as the performance of the regressor in the local region of the feature space around the test pattern, called the region of competence. However, no single measure is the best for any task. In this work, we present the Meta INtEgration (MINE) framework that selects and combines the most competent regressors from an ensemble during the evaluation of a given test pattern. MINE uses different measures extracted from the region of competence as a criterion for the selection and combination of the regressors. In contrast to traditional combination schemes where all the regressors are weighted to produce the final answer, MINE selects the best regressors per pattern on-the-fly and combines them to predict the value of the test pattern. Comprehensive experiments on 20 regression datasets show that MINE compares favorably to literature techniques.

Broad Learning System Based on Maximum Correntropy Criterion.

As an effective and efficient discriminative learning method, broad learning system (BLS) has received increasing attention due to its outstanding performance in various regression and classification problems. However, the standard BLS is derived under the minimum mean square error (MMSE) criterion, which is, of course, not always a good choice due to its sensitivity to outliers. To enhance the robustness of BLS, we propose in this work to adopt the maximum correntropy criterion (MCC) to train the output weights, obtaining a correntropy-based BLS (C-BLS). Due to the inherent superiorities of MCC, the proposed C-BLS is expected to achieve excellent robustness to outliers while maintaining the original performance of the standard BLS in the Gaussian or noise-free environment. In addition, three alternative incremental learning algorithms, derived from a weighted regularized least-squares solution rather than pseudoinverse formula, for C-BLS are developed. With the incremental learning algorithms, the system can be updated quickly without the entire retraining process from the beginning when some new samples arrive or the network deems to be expanded. Experiments on various regression and classification data sets are reported to demonstrate the desirable performance of the new methods.

Fast Deep Stacked Networks based on Extreme Learning Machine applied to regression problems

Deep learning techniques are commonly used to process large amounts of data, and good results are obtained in many applications. Those methods, however, can lead to long training times. An alternative to simultaneously tune all parameters of a large network is to stack smaller modules, improving the model efficiency. However, methods such as Deep Stacked Network (DSN) have some problems that increase its training time and memory usage. To deal with these problems, Fast DSN (FDSN) was proposed, where the modules are trained using an Extreme Learning Machine (ELM) variant. Nonetheless, to speed-up the FDSN training, the ELM random feature mapping is shared among the modules, which can impact the network performance if the weights are not properly chosen. In this paper, we focus on the weight initialization of FDSN in order to improve its performance. We also propose FKDSN, a kernel-based variant of FDSN, besides discussing the theoretical complexity of the methods. We evaluate three different initialization approaches on ELM-trained neural networks over 50 public real-world regression datasets. Our experiments show that FDSN when combined with a more complex initialization method achieves similar results to ELM algorithms applied to large SLFNs, besides having a shorter training time and memory usage, implying that it can be suitable to be used on systems with restrict resources, such as Internet of Things devices. FKDSN also obtained similar results and training time to the large SLFNs, requiring less memory.

A Transferable and Effective Method for Monitoring Continuous Cover Forestry at the Individual Tree Level Using UAVs

Transformation to Continuous Cover Forestry (CCF) is a long and difficult process in which frequent management interventions rapidly alter forest structure and dynamics with long lasting impacts. Therefore, a critical component of transformation is the acquisition of up-to-date forest inventory data to direct future management decisions. Recently, the use of single tree detection methods derived from unmanned aerial vehicle (UAV) has been identified as being a cost effective method for inventorying forests. However, the rapidly changing structure of forest stands in transformation amplifies the difficultly in transferability of current individual tree detection (ITD) methods. This study presents a novel ITD Bayesian parameter optimisation approach that uses quantile regression and external biophysical tree data sets to provide a transferable and low cost ITD approach to monitoring stands in transformation. We applied this novel method to 5 stands in a variety of transformation stages in the UK and to a independent test study site in California, USA, to assess the accuracy and transferability of this method. Requiring small amounts of training data (15 reference trees) this approach had a mean test accuracy (F-score = 0.88) and provided mean tree diameter estimates (RMSE = 5.6 cm) with differences that were not significance to the ground data (p < 0.05). We conclude that this method can be used to monitor forests stands in transformation and thus can also be applied to a wide range of forest structures with limited manual parameterisation between sites.

Memory augmented hyper-heuristic framework to solve multi-disciplinary problems inspired by cognitive problem solving skills

This paper proposes a new framework, named Deja-Vu+, which is an extension of Deja Vu framework, a classic study on hyper-heuristic framework with 2R (Record and Recall) modules. Deja-Vu+ has the ability to handle two other domains, namely regression and unsupervised learning. The extension examines the strength of Deja-Vu+ for solving regression and unsupervised learning tasks. The regression problems are treated here as multiclass classification tasks, and unsupervised learning tasks are considered as clustering problems. The proposed framework is tested on a number of regression and unsupervised learning benchmark problems and has shown promising results to handle regression as classification. The framework attains an overall average accuracy of 70% for regression and clustering data sets. Deja-Vu+ is knowledge-rich hyper-heuristic framework, which is capable enough to transfer knowledge successfully. This knowledge transfer improves the performance of learning by avoiding the extensive heuristic search process. Our experimental results show that using previously attained knowledge to reduce the computational effort is beneficial in solving multi-disciplinary machine learning problems.

On robust randomized neural networks for regression: a comprehensive review and evaluation

Data from real-world regression problems are quite often contaminated with outliers. In order to efficiently handle such undesirable samples, robust parameter estimation methods have been incorporated into randomized neural network (RNN) models, usually replacing the ordinary least squares (OLS) method. Despite recent successful applications to outlier-contaminated scenarios, significant issues remain unaddressed in the design of reliable outlier-robust RNN models for regression tasks. For example, the number of hidden neurons impacts directly on the norm of the estimated output weights, since the OLS method will rely on an ill-conditioned hidden-layer output matrix. Another design concern involves the high sensitivity of RNNs to the randomization of the hidden layer weights, an issue that can be suitably handled, e.g., by intrinsic plasticity techniques. Bearing these concerns in mind, we describe several ideas introduced in previous works concerning the design of RNN models that are both robust to outliers and numerically stable. A comprehensive evaluation of their performances is carried out across several benchmarking regression datasets taking into account accuracy, weight norms, and training time as figures of merit.

Modeling the Spread of COVID-19 Infection Using a Multilayer Perceptron.

Coronavirus (COVID-19) is a highly infectious disease that has captured the attention of the worldwide public. Modeling of such diseases can be extremely important in the prediction of their impact. While classic, statistical, modeling can provide satisfactory models, it can also fail to comprehend the intricacies contained within the data. In this paper, authors use a publicly available dataset, containing information on infected, recovered, and deceased patients in 406 locations over 51 days (22nd January 2020 to 12th March 2020). This dataset, intended to be a time-series dataset, is transformed into a regression dataset and used in training a multilayer perceptron (MLP) artificial neural network (ANN). The aim of training is to achieve a worldwide model of the maximal number of patients across all locations in each time unit. Hyperparameters of the MLP are varied using a grid search algorithm, with a total of 5376 hyperparameter combinations. Using those combinations, a total of 48384 ANNs are trained (16128 for each patient group—deceased, recovered, and infected), and each model is evaluated using the coefficient of determination (R2). Cross-validation is performed using K-fold algorithm with 5-folds. Best models achieved consists of 4 hidden layers with 4 neurons in each of those layers, and use a ReLU activation function, with R2 scores of 0.98599 for confirmed, 0.99429 for deceased, and 0.97941 for recovered patient models. When cross-validation is performed, these scores drop to 0.94 for confirmed, 0.781 for recovered, and 0.986 for deceased patient models, showing high robustness of the deceased patient model, good robustness for confirmed, and low robustness for recovered patient model.

A robust graph-based semi-supervised sparse feature selection method

Feature selection is used for excluding redundant features and enhancing learning performance. Abundant unlabeled data are existed in many applications which can be used in semi-supervised feature selection. Semi-supervised sparse feature selection methods have been presented to apply labeled and unlabeled data and consider the correlation among features. Most of the sparse methods utilize square-norm based loss function which is sensitive to outliers. In this paper, we propose a robust Graph-based Semi-Supervised Sparse Feature Selection (GS3FS) method based on the mixed convex and non-convex l2,p-norm (0 < p ≤ 1) minimization on regularization and loss function. The l2,p-norm based regularization ensures that the method selects relevant and sparse features, and the l2,p-norm based loss function makes the method robust to outliers. Proposed method takes the advantages of manifold learning and l2,p-norm minimization on regularization and loss function. The method includes the mixed l2,p-norm which is difficult to solve. To solve the proposed method for the convex (p = 1) and non-convex (0 < p <1) cases, we propose an efficient unified algorithm and prove its convergence. Extensive experiments are carried out on various classification and regression datasets for evaluating the proposed method. The results indicate that the proposed method selects the relevant features and improves the performance of classification and regression models.

Extending the weightless WiSARD classifier for regression

This paper explores two new weightless neural network models, Regression WiSARD and ClusRegression WiSARD, in the challenging task of predicting the total palm oil production of a set of 28 (twenty eight) differently located sites under different climate and soil profiles. Both models were derived from Kolcz and Allinson’s n-Tuple Regression weightless neural model and obtained mean absolute error (MAE) rates of 0.09097 and 0.09173, respectively. Such results are very competitive with the state-of-the-art (0.07983), whilst being four orders of magnitude faster during the training phase. Additionally the models have been tested on three classic regression datasets, also presenting competitive performance with respect to other models often used in this type of task.

Uncovering tissue-specific binding features from differential deep learning.

Transcription factors (TFs) can bind DNA in a cooperative manner, enabling a mutual increase in occupancy. Through this type of interaction, alternative binding sites can be preferentially bound in different tissues to regulate tissue-specific expression programmes. Recently, deep learning models have become state-of-the-art in various pattern analysis tasks, including applications in the field of genomics. We therefore investigate the application of convolutional neural network (CNN) models to the discovery of sequence features determining cooperative and differential TF binding across tissues. We analyse ChIP-seq data from MEIS, TFs which are broadly expressed across mouse branchial arches, and HOXA2, which is expressed in the second and more posterior branchial arches. By developing models predictive of MEIS differential binding in all three tissues, we are able to accurately predict HOXA2 co-binding sites. We evaluate transfer-like and multitask approaches to regularizing the high-dimensional classification task with a larger regression dataset, allowing for the creation of deeper and more accurate models. We test the performance of perturbation and gradient-based attribution methods in identifying the HOXA2 sites from differential MEIS data. Our results show that deep regularized models significantly outperform shallow CNNs as well as k-mer methods in the discovery of tissue-specific sites bound in vivo.

Calibration of Machine Learning–Based Probabilistic Hail Predictions for Operational Forecasting

Abstract In this study, we use machine learning (ML) to improve hail prediction by postprocessing numerical weather prediction (NWP) data from the new High-Resolution Ensemble Forecast system, version 2 (HREFv2). Multiple operational models and ensembles currently predict hail, however ML models are more computationally efficient and do not require the physical assumptions associated with explicit predictions. Calibrating the ML-based predictions toward familiar forecaster output allows for a combination of higher skill associated with ML models and increased forecaster trust in the output. The observational dataset used to train and verify the random forest model is the Maximum Estimated Size of Hail (MESH), a Multi-Radar Multi-Sensor (MRMS) product. To build trust in the predictions, the ML-based hail predictions are calibrated using isotonic regression. The target datasets for isotonic regression include the local storm reports and Storm Prediction Center (SPC) practically perfect data. Verification of the ML predictions indicates that the probability magnitudes output from the calibrated models closely resemble the day-1 SPC outlook and practically perfect data. The ML model calibrated toward the local storm reports exhibited better or similar skill to the uncalibrated predictions, while decreasing model bias. Increases in reliability and skill after calibration may increase forecaster trust in the automated hail predictions.