Multi-target Regression Problems Research Articles

Outlier Robust Extreme Machine Learning for multi-target regression

The popularity of algorithms based on Extreme Learning Machine (ELM), which can be used to train Single Layer Feedforward Neural Networks (SLFN), has increased in the past years. They have been successfully applied to a wide range of classification and regression tasks. The most commonly used methods are the ones based on minimizing the ℓ2 norm of the error, which is not suitable to deal with outliers, essentially in regression tasks. The use of ℓ1 norm was proposed in Outlier Robust ELM (OR-ELM), which is defined to one-dimensional outputs. In this paper, we generalize OR-ELM to deal with multi-target regression problems, using the error ℓ2,1 norm and the Elastic Net theory, which can result in a more sparse network, resulting in our method, Generalized Outlier Robust ELM (GOR-ELM). We use Alternating Direction Method of Multipliers (ADMM) to solve the resulting optimization problem. An incremental version of GOR-ELM is also proposed. We chose 15 public real-world multi-target regression datasets to test our methods. Our conducted experiments show that they are statistically better than other ELM-based techniques, when considering data contaminated with outliers, and equivalent to them, otherwise.

Data-Driven Structuring of the Output Space Improves the Performance of Multi-Target Regressors

The task of multi-target regression (MTR) is concerned with learning predictive models capable of predicting multiple target variables simultaneously. MTR has attracted an increasing attention within research community in recent years, yielding a variety of methods. The methods can be divided into two main groups: problem transformation and problem adaptation. The former transform a MTR problem into simpler (typically single target) problems and apply known approaches, while the latter adapt the learning methods to directly handle the multiple target variables and learn better models which simultaneously predict all of the targets. Studies have identified the latter group of methods as having competitive advantage over the former, probably due to the fact that it exploits the interrelations of the multiple targets. In the related task of multi-label classification, it has been recently shown that organizing the multiple labels into a hierarchical structure can improve predictive performance. In this paper, we investigate whether organizing the targets into a hierarchical structure can improve the performance for MTR problems. More precisely, we propose to structure the multiple target variables into a hierarchy of variables, thus translating the task of MTR into a task of hierarchical multi-target regression (HMTR). We use four data-driven methods for devising the hierarchical structure that cluster the real values of the targets or the feature importance scores with respect to the targets. The evaluation of the proposed methodology on 16 benchmark MTR datasets reveals that structuring the multiple target variables into a hierarchy improves the predictive performance of the corresponding MTR models. The results also show that data-driven methods produce hierarchies that can improve the predictive performance even more than expert constructed hierarchies. Finally, the improvement in predictive performance is more pronounced for the datasets with very large numbers (more than hundred) of targets.

An ensemble-based method for the selection of instances in the multi-target regression problem

An ensemble-based method for the selection of instances in the multi-target regression problem

Safe semi supervised multi-target regression (MTR-SAFER) for new targets learning

Multi-target regression (MTR) is a challenging research problem which aims to predict more than one continuous variable as output in a pattern. In recent time, a number of novel applications have increased interest and research in this area. Applications include predicting brain activity from multimedia sensors, different values for stocks from continuous web data, condition of various attributes of the vegetation at a given site, etc. In contrast to conventional regression problems where each instance only belongs to single target from a set of disjoint targets, in multi-target regression, each instance may belong to more than one continuous variable as output. Multi-target regression problems are concerned with problems where there are more than one continuous variables to output. These output variables may or may not be related. A number of approaches have been proposed for this problem. However, for a dynamic multi-target learning system, a pre-trained multi-target system shall be revised as new targets emerge with very few instances. The objective of this paper is to investigate semi-supervised techniques on multi-target regression problems to predict new target using very limited amount of examples. Experiments are then conducted on real world multi-target regression (MTR) data sets. The proposed methodology is then compared with state of the art MTR methods. Promising results are obtained using proposed safe semi-supervised regressor with binary relevance.

A novel Boosted-neural network ensemble for modeling multi-target regression problems

In this paper, the concept of ensemble learning is adopted and applied to modeling multi-target regression problems with high-dimensional feature spaces and a small number of instances. A novel neural network ensemble (NNE) model is introduced, called Boosted-NNE based on notions from boosting, subspace projection methods and the negative correlation learning algorithm (NCL). Rather than using an entire feature space for training each component in the Boosted-NNE, a new cluster-based subspace projection method (CLSP) is proposed to automatically construct a low-dimensional input space with focus on the difficult instances in each step of the boosting approach. To enhance diversity in the Boosted-NNE, a new, sequential negative correlation learning algorithm (SNCL) is proposed to train negatively correlated components. Furthermore, the constrained least mean square error (CLMS) algorithm is employed to obtain the optimal weights of components in the combination module. The proposed Boosted-NNE model is compared with other ensemble and single models using four real cases of multi-target regression problems. The experimental results indicate that using the SNCL in combination with the CLSP method offers the capability to improve the diversity and accuracy of the Boosted-NNE. Thus, this model seems a promising alternative for modeling high-dimensional multi-target regression problems.

SPMoE: a novel subspace-projected mixture of experts model for multi-target regression problems

In this paper, we focus on modeling multi-target regression problems with high-dimensional feature spaces and a small number of instances that are common in many real-life problems of predictive modeling. With the aim of designing an accurate prediction tool, we present a novel mixture of experts (MoE) model called subspace-projected MoE (SPMoE). Training the experts of the SPMoE is done using a boosting-like manner by a combination of ideas from subspace projection method and the negative correlation learning algorithm (NCL). Instead of using whole original input space for training the experts, we develop a new cluster-based subspace projection method to obtain projected subspaces focused on the difficult instances at each step of the boosting approach for training the diverse experts. The experts of the SPMoE are trained on the obtained subspaces using a new NCL algorithm called sequential NCL. The SPMoE is compared with the other ensemble models using three real cases of high-dimensional multi-target regression problems; the electrical discharge machining, energy efficiency and an important problem in the field of operations strategy called the practice---performance problem. The experimental results show that the prediction accuracy of the SPMoE is significantly better than the other ensemble and single models and can be considered to be a promising alternative for modeling the high-dimensional multi-target regression problems.