Multi-response Linear Regression Research Articles

Multi-response Regression for Block-missing Multi-modal Data without Imputation.

Multi-modal data are prevalent in many scientific fields. In this study, we consider the parameter estimation and variable selection for a multi-response regression using block-missing multi-modal data. Our method allows the dimensions of both the responses and the predictors to be large, and the responses to be incomplete and correlated, a common practical problem in high-dimensional settings. Our proposed method uses two steps to make a prediction from a multi-response linear regression model with block-missing multi-modal predictors. In the first step, without imputing missing data, we use all available data to estimate the covariance matrix of the predictors and the cross-covariance matrix between the predictors and the responses. In the second step, we use these matrices and a penalized method to simultaneously estimate the precision matrix of the response vector, given the predictors, and the sparse regression parameter matrix. Lastly, we demonstrate the effectiveness of the proposed method using theoretical studies, simulated examples, and an analysis of a multi-modal imaging data set from the Alzheimer's Disease Neuroimaging Initiative.

Evaluation of Selected Stacked Ensemble Models for the Optimal Multi-class Cyber-Attacks Detection

The significant rise in the frequency and sophistication of cyber-attacks and their diversity necessitated various researchers to develop strong and effective approaches to address recurring cyber threat challenges. This study evaluated the performance of three selected meta-learning models for optimal multi-class detection of cyber-attacks using the University of New South Wales 2015 Network benchmark (UNSW-NB15) Intrusion Dataset. The results of this study show and confirm the ability of the three base models; Naive Bayes, C4.5 Decision Tree, and K-Nearest Neighbor for solving multi-class problems. It further affirms the knack of the duo of feature selection techniques and stacked ensemble learning to optimize ML models' performances. The stacking of the predictions of the information gain base models with Model Decision Tree meta-algorithm recorded the most improved and optimal cyber-attacks detection accuracy and Mattew's correlation Coefficient than the stacking with the Multiple Model Trees (MMT) and Multi Response Linear regression (MLR) Meta algorithms.

A cross-entropy based stacking method in ensemble learning

Stacking is one of the major types of ensemble learning techniques in which a set of base classifiers contributes their outputs to the meta-level classifier, and the meta-level classifier combines them so as to produce more accurate classifications. In this paper, we propose a new stacking algorithm that defines the cross-entropy as the loss function for the classification problem. The training process is conducted by using a neural network with the stochastic gradient descent technique. One major characteristic of our method is its treatment of each meta instance as a whole with one optimization model, which is different from some other stacking methods such as stacking with multi-response linear regression and stacking with multi-response model trees. In these methods each meta instance is divided into a set of sub-instances. Multiple models apply to those sub-instances and each for a class label. There is no connection between different models. It is very likely that our treatment is a better choice for finding suitable weights. Experiments with 22 data sets from the UCI machine learning repository show that the proposed stacking approach performs well. It outperforms all three base classifiers, several state-of-the-art stacking algorithms, and some other representative ensemble learning methods on average.

Data-driven disruption prediction in GOLEM Tokamak using ensemble classifiers

A robust disruption prediction system is mandatory in a Tokamak control system as the disruption can cause malfunctioning of the plasma-facing components and impair irrecoverable structural damage to the vessel. To mitigate the disruption, in this article, a data-driven based disruption predictor is developed using an ensemble technique. The ensemble algorithm classifies disruptive and non-disruptive discharges in the GOLEM Tokamak system. Ensemble classifiers combine the predictive capacity of several weak learners to produce a single predictive model and are utilized both in supervised and unsupervised learning. The resulting final model reduces the bias, minimizes variance and is unlikely to over-fit when compared to the individual model from a single algorithm. In this paper, popular ensemble techniques such as Bagging, Boosting, Voting, and Stacking are employed on the time-series Tokamak dataset, which consists of 117 normal and 70 disruptive shots. Stacking ensemble with REPTree (Reduced Error Pruning Tree) as a base learner and Multi-response Linear Regression as meta learner produced better results in comparison to other ensembles. A comparison with the widely employed stand-alone machine learning algorithms and ensemble algorithms are illustrated. The results show the excellent performance of the Stacking model with an F1 score of 0.973. The developed predictive model would be capable of warning the human operator with feedback about the feature(s) causing the disruption.

Variable selection in high-dimensional sparse multiresponse linear regression models

We consider variable selection in high-dimensional sparse multiresponse linear regression models, in which a q-dimensional response vector has a linear relationship with a p-dimensional covariate vector through a sparse coefficient matrix $$B\in R^{p\times q}$$. We propose a consistent procedure for the purpose of identifying the nonzeros in B. The procedure consists of two major steps, where the first step focuses on the detection of all the nonzero rows in B, the latter aims to further discover its individual nonzero cells. The first step is an extension of Orthogonal Matching Pursuit (OMP) and the second step adopts the bootstrap strategy. The theoretical property of our proposed procedure is established. Extensive numerical studies are presented to compare its performances with available representatives.

Emotion recognition using mobile phones

The availability of built-in sensors in mobile phones has enabled a host of innovative applications. One class of application deals with detecting a user's emotions. Previous applications have primarily relied on recording and displaying self-reported emotions. This paper presents an intelligent emotion detection system for mobile phones implemented as a smart keyboard that infers a user's emotional state using machine learning techniques. The system uses accelerometer readings and various aspect of typing behavior like speed and delay between letters to train a classifier to predict emotions. Naïve Bayes, J48, IBK, Multi-response linear regression and SVM were evaluated and J48 was found to be the best classifier with over 90% accuracy and precision. In addition to providing emotive feedback to individual users, the system also uses geo-tagged data to collect and display emotional states of regions or countries through a website.

Collective Support Recovery for Multi-Design Multi-Response Linear Regression

The multi-design multi-response linear regression problem is investigated, in which design matrices are Gaussian with covariance matrices Σ (1:K) = (Σ (1) , ... , 1 (K) ) for K linear regression tasks. Design matrices across tasks are assumed to be independent. The support union of K p-dimensional regression vectors (collected as columns of matrix B*) is recovered using l 1 /l 2 -regularized lasso. Sufficient and necessary conditions on sample complexity are characterized as a sharp threshold to guarantee successful recovery of the support union. This model has been previously studied via l 1 /l ∞ -regularized lassoand via l 1 /l 1 + l 1 /l ∞ -regularized lasso, in which sharp threshold on sample complexity is characterized only for K = 2 and under special conditions. In this paper, using l 1 /l 2 -regularized lasso, sharp threshold on sample complexity is characterized under standard regularization conditions. Namely, if n > c p1 ψ(B*, Σ (1:K) ) log(p - s) where c p1 is a constant, and s is the size of the support set, then l 1 /l 2 -regularized lasso correctly recovers the support union; and if n <; c p2 ψ(B*, Σ (1:K) )log(p - s) where c p2 is a constant, then l 1 /l 2 -regularized lasso fails to recover the support union. In particular, the function ψ(B*, Σ (1:K) ) captures the impact of the sparsity of K regression vectors and the statistical properties of the design matrices on the threshold on sample complexity. Therefore, such threshold function also demonstrates the advantages of joint support union recovery using multitask lasso over individual support recovery using single-task lasso.

Vehicle identification by improved stacking via kernel principal component regression

Purpose – Many applications in intelligent transportation demand accurate categorization of vehicles. The purpose of this paper is to propose a working image-based vehicle classification system. The first component vehicle detection is implemented by applying Dalal and Triggs's histograms of oriented gradients features and linear support vector machine (SVM) classifier. The second component vehicle classification, which is the emphasis of this paper, is accomplished by an improved stacked generalization. As an effective ensemble learning strategy, stacked generalization has been proposed to combine multiple models using the concept of a meta-learner. However, it was found that the well-known meta-learning scheme multi-response linear regression (MLR) for stacked generalization performs poorly on the vehicle classification. Design/methodology/approach – A new meta-learner is then proposed based on kernel principal component regression (KPCR). The stacked generalization scheme consists of a heterogeneous classifier ensemble with seven base classifiers, i.e. linear discriminant classifier, fuzzy k-nearest neighbor, logistic regression, Parzen classifier, Gaussian mixture model, multiple layer perceptron and SVM. Findings – Experimental results using more than 2,500 images from four types of vehicles (bus, light truck, car and van) demonstrated the effectiveness of the proposed approach. The improved stacked generalization produced consistently better results when compared to any of the single base classifier used and four other beta learning algorithms, including MLR, majority voting, logistic regression and decision template. Originality/value – With the seven base classifiers, the KPCR-based stacking offers a performance of 96 percent accuracy and 95 percent κ coefficient, thus exhibiting promising potentials for real-world applications.

Input selection and shrinkage in multiresponse linear regression

The regression problem of modeling several response variables using the same set of input variables is considered. The model is linearly parameterized and the parameters are estimated by minimizing the error sum of squares subject to a sparsity constraint. The constraint has the effect of eliminating useless inputs and constraining the parameters of the remaining inputs in the model. Two algorithms for solving the resulting convex cone programming problem are proposed. The first algorithm gives a pointwise solution, while the second one computes the entire path of solutions as a function of the constraint parameter. Based on experiments with real data sets, the proposed method has a similar performance to existing methods. In simulation experiments, the proposed method is competitive both in terms of prediction accuracy and correctness of input selection. The advantages become more apparent when many correlated inputs are available for model construction.

Is Combining Classifiers with Stacking Better than Selecting the Best One?

We empirically evaluate several state-of-the-art methods for constructing ensembles of heterogeneous classifiers with stacking and show that they perform (at best) comparably to selecting the best classifier from the ensemble by cross validation. Among state-of-the-art stacking methods, stacking with probability distributions and multi-response linear regression performs best. We propose two extensions of this method, one using an extended set of meta-level features and the other using multi-response model trees to learn at the meta-level. We show that the latter extension performs better than existing stacking approaches and better than selecting the best classifier by cross validation.

Flexible Discriminant Analysis by Optimal Scoring

Abstract Fisher's linear discriminant analysis is a valuable tool for multigroup classification. With a large number of predictors, one can find a reduced number of discriminant coordinate functions that are “optimal” for separating the groups. With two such functions, one can produce a classification map that partitions the reduced space into regions that are identified with group membership, and the decision boundaries are linear. This article is about richer nonlinear classification schemes. Linear discriminant analysis is equivalent to multiresponse linear regression using optimal scorings to represent the groups. In this paper, we obtain nonparametric versions of discriminant analysis by replacing linear regression by any nonparametric regression method. In this way, any multiresponse regression technique (such as MARS or neural networks) can be postprocessed to improve its classification performance.

Flexible Discriminant Analysis by Optimal Scoring

Abstract Fisher's linear discriminant analysis is a valuable tool for multigroup classification. With a large number of predictors, one can find a reduced number of discriminant coordinate functions that are “optimal” for separating the groups. With two such functions, one can produce a classification map that partitions the reduced space into regions that are identified with group membership, and the decision boundaries are linear. This article is about richer nonlinear classification schemes. Linear discriminant analysis is equivalent to multiresponse linear regression using optimal scorings to represent the groups. In this paper, we obtain nonparametric versions of discriminant analysis by replacing linear regression by any nonparametric regression method. In this way, any multiresponse regression technique (such as MARS or neural networks) can be postprocessed to improve its classification performance.