Multivariate Regression Problem Research Articles

AN ADAPTIVE METHOD FOR BUILDING A MULTIVARIATE REGRESSION

We propose an adaptive method for building a multivariate regression given by a weighted linear convolution of known scalar functions of deterministic input variables with unknown coefficients. As, for example, when multivariate regression is given by a multivariate polynomial. In contrast to the general procedure of the least squares method that minimizes only a single scalar quantitative measure, the adaptive method uses six different quantitative measures and represents a systemically connected set of different algorithms which allow each applied problem to be solved on their basis by an individual adaptive algorithm that, in the case of an active experiment, even for a relatively small volume of experimental data, implements a strategy of a statistically justified solving. The small amount of data of the active experiment we use in the sense that, for such an amount, the variances of estimates of unknown coefficients obtained by the general procedure of the least squares method do not allow to guarantee the accuracy acceptable for practice. We also proposed to significantly increase the efficiency of the proposed by O. A. Pavlov. and M. M. Holovchenko modified group method of data handling for building a multivariate regression which is linear with respect to unknown coefficients and given by a redundant representation. We improve it by including some criteria and algorithms of the adaptive method for building a multivariate regression. For the multivariate polynomial regression problem, the inclusion of a partial case of the new version of the modified group method of data handling in the synthetic method proposed by O. A. Pavlov, M. M. Golovchenko, and V. V. Drozd, for building a multivariate polynomial regression given by a redundant representation, also significantly increases its efficiency.

PLS for designed experiments

In many emerging new areas of research, for example in -omics sciences, complex data sets are generated implementing well-designed experiments with the aim to study the effects of factors on responses measured by high-through analytical platforms. The data sets are multivariate in their nature and require suitable data analysis techniques to be investigated. Partial Least Squares (PLS) regression has been developed and successfully applied to solve multivariate regression problems where data are correlated, redundant and noisy, but it is not able to take explicitly into account the experimental design used to generate the data. In this study, we adapt PLS to model data generated by crossed factorial designs. A new method based on constrained PLS2 regression and a type III sums of squares-like approach based on PLS2 are developed to treat orthogonal and non-orthogonal designs, respectively. Post-transformation and Procrustes analysis are applied to discover the 'latent’ structure useful to investigate the relationships between factors and responses. The mathematical properties of the data decomposition, the inference process to assess the reliability of the model and some basic tools for model interpretation are presented and discussed. One simulated and two experimental data sets, one generated implementing an orthogonal design and one obtained by a non-orthogonal design, are investigated to show how the methodology works in practice.

A Piecewise Linear Regression and Classification Algorithm With Application to Learning and Model Predictive Control of Hybrid Systems

This paper proposes an algorithm for solving multivariate regression and classification problems using piecewise linear predictors over a polyhedral partition of the feature space. The resulting algorithm that we call PARC (Piecewise Affine Regression and Classification) alternates between (<i>i</i>) solving ridge regression problems for numeric targets, softmax regression problems for categorical targets, and either softmax regression or cluster centroid computation for piecewise linear separation, and (<i>ii</i>) assigning the training points to different clusters on the basis of a criterion that balances prediction accuracy and piecewise-linear separability. We prove that PARC is a block-coordinate descent algorithm that minimizes a suitably constructed objective function and that it converges in a finite number of steps. The algorithm is used to learn hybrid numerical/categorical (HYNC) dynamical models from data that contain real and discrete labeled values. The resulting model has a piecewise linear structure that is particularly useful to formulate model predictive control problems and solve them by mixed-integer programming. A Python implementation of PARC is available at <uri>http://cse.lab.imtlucca.it/~bemporad/parc</uri>.

A similarity-based Bayesian mixture-of-experts model

We present a new nonparametric mixture-of-experts model for multivariate regression problems, inspired by the probabilistic k-nearest neighbors algorithm. Using a conditionally specified model, predictions for out-of-sample inputs are based on similarities to each observed data point, yielding predictive distributions represented by Gaussian mixtures. Posterior inference is performed on the parameters of the mixture components as well as the distance metric using a mean-field variational Bayes algorithm accompanied with a stochastic gradient-based optimization procedure. The proposed method is especially advantageous in settings where inputs are of relatively high dimension in comparison to the data size, where input–output relationships are complex, and where predictive distributions may be skewed or multimodal. Computational studies on five datasets, of which two are synthetically generated, illustrate clear advantages of our mixture-of-experts method for high-dimensional inputs, outperforming competitor models both in terms of validation metrics and visual inspection.

Signal Detection in GSM-Based In-Band Full-Duplex Communication Using DNN

An in-band full-duplex (IBFD) multiple-input multiple-output (MIMO) radio's self-interference (SI) cancellation strength usually determine its performance gains over conventional half-duplex ones. Accordingly, this letter explores an alternative to traditional optimization driven design (ODD) techniques available in literature for beamformer design in IBFD radios. In particular, we consider a generalized spatial modulation (GSM) based bi-directional IBFD system, that provides the flexibility of choosing active antennas among a set of antennas for the spatial symbol. We propose a run-time data-driven prediction approach to solve the multiple multivariate regression problem of detection of the transmitted signal in this GSM based IBFD system in the presence of residual SI arising from non-ideal SI cancellation. We compare the performance of the proposed detector with respect to conventional detectors under several communication parameters that are of practical interest. The proposed DNN-based detector learns the deviations from a standard model without SI and achieves performance that is superior to zero forcing and minimum mean squared error detectors and very close to maximum likelihood detector at faster computation time.

A modified partial envelope tensor response regression

The envelope model is a useful statistical technique that can be applied to multivariate linear regression problems. It aims to remove immaterial information via sufficient dimension reduction techniques while still gaining efficiency and providing accurate parameter estimates. Recently, envelope tensor versions have been developed to extend this technique to tensor data. In this work, a partial tensor envelope model is proposed that allows for a parsimonious version of tensor response regression when only certain predictors are of interest. The consistency and asymptotic normality of the regression coefficients estimator are also established theoretically, which provides a rigorous foundation for the proposed method. In numerical studies using both simulated and real‐world data, the partial tensor envelope model is shown to outperform several existing methods in terms of the efficiency of the regression coefficients associated with the selected predictors.

A new method for building single feedforward neural network models for multivariate static regression problems: a combined weight initialization and constructive algorithm

A new method for building single feedforward neural network models for multivariate static regression problems: a combined weight initialization and constructive algorithm

On extending Amdahl’s law to learn computer performance

The problem of learning parallel computer performance is investigated in the context of multicore processors. Given a fixed workload, the effect of varying system configuration on performance is sought. Conventionally, the performance speedup due to a single resource enhancement is formulated using Amdahl’s law. However, in case of multiple configurable resources the conventional formulation results in several disconnected speedup equations that cannot be combined together to determine the overall speedup. To solve this problem, we propose to (1) extend Amdahl’s law to accommodate multiple configurable resources into the overall speedup equation, and (2) transform the speedup equation into a multivariable regression problem suitable for machine learning. Using experimental data from fifty-eight tests spanning two benchmarks (SPECCPU 2017 and PCMark 10) and four hardware platforms (Intel Xeon 8180M, AMD EPYC 7702P, Intel CoffeeLake 8700K, and AMD Ryzen 3900X), analytical models are developed and cross-validated. Findings indicate that in most cases, the models result in an average cross-validated accuracy higher than 95%, thereby validating the proposed extension of Amdahl’s law. The proposed methodology enables rapid generation of multivariable analytical models to support future industrial development, optimization, and simulation needs.

A realizable and scale-consistent data-driven non-linear eddy viscosity modeling framework for arbitrary regression algorithms

A data-driven modeling framework for non-linear eddy viscosity models is presented. In contrast to the majority of similar approaches, it splits the multivariate regression problem into a set of univariate problems and is, furthermore, independent of a specific machine learning algorithm. Instead of inferring the closure equation from high-fidelity data as a whole, the coefficients of the tensor polynomial are learned individually. The target variables are obtained from an efficient field inversion procedure by virtue of successive tensor projections. The hypotheses for each closure coefficient are then fitted separately by employing an arbitrary regression technique. Ordinary curve fitting, neural networks and gene expression programming are considered as examples. The robustness of the model against an extrapolation and the stability of the solver are promoted by coefficient limiters and a full barycentric realizability correction. Additionally, turbulent scale consistency is guaranteed by a k-corrective frozen-RANS method, which is the subject of a companion paper (Mandler and Weigand, 2022). The proposed strategy leads to models which are well suited for the application to the same class of flows they were inferred from, namely separated channel flows. As proven by an extensive extrapolation study, the resulting neuralSST model is robust against geometry and Reynolds number modifications provided the type of flow does not drastically change. It consistently outperforms both the shear stress transport (SST) and a more complex elliptic-blending model and agrees well with the reference data.

CONSTRUCTION OF A MULTIVARIATE POLYNOMIAL GIVEN BY A REDUNDANT DESCRIPTION IN STOCHASTIC AND DETERMINISTIC FORMULATIONS USING AN ACTIVE EXPERIMENT

We present the methods for constructing a multivariate polynomial given by a redundant representation based on the results of a limited active experiment. We solve the problem in two formulations. The first is the problem of constructing a multivariate polynomial regression given by a redundant representation based on the results of a limited active experiment. The solution method is based on the previous results of Professor A. A. Pavlov and his students showing the fundamental possibility of reducing this problem to the sequential construction of univariate polynomial regressions and solving the corresponding nondegenerate systems of linear equations. There are two modifications of this method. The second modification is based on proving for an arbitrary limited active experiment the possibility of using only one set of normalized orthogonal polynomials of Forsythe. The second formulation refers to the solution of this problem for a particular but sufficient from the practical point of view case when an unknown implementation of a random variable is not added to the initial measurement results during an active experiment. This method is a modification of the solution method for the multivariate polynomial regression problem. Also, we used the main results of the general theory (which reduces the multivariate polynomial regression problem solving to the sequential construction of univariate polynomial regressions and solution of corresponding nondegenerate systems of linear equations) to consider and strictly substantiate fairly wide from the practical point of view particular cases leading to estimating the coefficients at nonlinear terms of the multivariate polynomial regression as a solution of linear equations with a single variable.

A novel wide & deep transfer learning stacked GRU framework for network intrusion detection

With the increasing frequency, severity and complexity of recent cyber attacks around the world, network intrusion detection has become mandatory and highly sophisticated task. Achieving high performance in network intrusion detection by applying benchmark machine learning classifiers (including deep learning techniques) has become a major challenge in recent times. One of the biggest challenges is improving the memorization capacity and generalization ability of NIDS (Network Intrusion Detection Systems). In this paper, we propose a highly scalable novel wide & deep transfer learning (TL) based stacked GRU (Gated Recurrent Unit) model to deal with multi-dimensional point data and multi-variate time series regression and classification problems in network intrusion detection. The proposed model has the memorization capacity of linear regression model and the generalization ability of deep GRU model. The deep component consists of a transfer learning framework that pretrains a source model and then fine-tunes the whole source model on the same dataset multiple times until it gives peak performance. This method gives a multi-class evaluation accuracy of 99.92% on KDDCup 99(10%) dataset and 94.22% on UNSW-NB15 dataset respectively. Extensive experimentations and evaluations have been carried out by comparing it with other machine learning (including deep learning) network intrusion detection techniques. The proposed method outperforms most of the existing intrusion detection approaches.

ANNs Going Beyond Time Series Forecasting: An Urban Network Perspective

5G is expected to bring forth disruptive indus-trial-societal transformation by enabling a broad catalog of (radically new, highly heterogeneous) applications and services. This scenario has called for zero-touch network and service management (ZSM). With the recent advancements in artificial intelligence, key ZSM capabilities such as the runtime prediction of user demands can be facilitated by data-driven and machine learning methods. In this respect, the article proposes a runtime prediction approach that transforms time series forecasting into a simpler multivariate regression problem with artificial neural networks (ANNs), structurally optimized with a genetic algorithm (GA) metaheuristic. Leveraging on a novel set of input features that capture seasonality and calendar effects, the proposed approach removes the prediction accuracy's dependence on the temporal succession of input data and the forecast horizon. Evaluation results based on real telecommunications data show that the GA-optimized ANN regressor has better prediction performance. It achieved average improvements of ~59 percent and ~86 percent compared to 1-day and 1-hour ahead forecasts obtained with state-of-the-art multi-seasonal time series and long short-term memory forecasting models, respectively. Furthermore, despite its longer training times compared to the baseline models, the proposed ANN regressor relaxes the monitoring requirements in 5G dynamic management systems by allowing less frequent retraining offline.

Multiple haplotype reconstruction from allele frequency data.

Because haplotype information is of widespread interest in biomedical applications, effort has been put into their reconstruction. Here, we propose an efficient method, called haploSep, that is able to accurately infer major haplotypes and their frequencies just from multiple samples of allele frequency data. Even the accuracy of experimentally obtained allele frequencies can be improved by re-estimating them from our reconstructed haplotypes. From a methodological point of view, we model our problem as a multivariate regression problem where both the design matrix and the coefficient matrix are unknown. Compared to other methods, haploSep is very fast, with linear computational complexity in the haplotype length. We illustrate our method on simulated and real data focusing on experimental evolution and microbial data.

Feature and Language Selection in Temporal Symbolic Regression for Interpretable Air Quality Modelling

Air quality modelling that relates meteorological, car traffic, and pollution data is a fundamental problem, approached in several different ways in the recent literature. In particular, a set of such data sampled at a specific location and during a specific period of time can be seen as a multivariate time series, and modelling the values of the pollutant concentrations can be seen as a multivariate temporal regression problem. In this paper, we propose a new method for symbolic multivariate temporal regression, and we apply it to several data sets that contain real air quality data from the city of Wrocław (Poland). Our experiments show that our approach is superior to classical, especially symbolic, ones, both in statistical performances and the interpretability of the results.

On the convergence of rank-one multi-target linear regression

This paper presents a new method for solving rank-one multivariate regression problems, providing a solution that maximizes the sum of squared correlations of the one-dimensional fitted pattern with the target variates. The suitability of the method and the consistency of the estimator are formally proved and experimentally tested. In particular, it is shown that the estimate converges not only as a function of the number of items, but also as a function of the number of target variates. An equivalent conventional reduced-rank regression case is identified, and it inherits the convergence properties of the new approach. Application programs in Matlab/Octave code are provided and numerical examples using artificial data as well as real data are presented.

Efron–Petrosian integrals for doubly truncated data with covariates: An asymptotic analysis

In survival analysis, epidemiology and related fields there exists an increasing interest in statistical methods for doubly truncated data. Double truncation appears with interval sampling and other sampling schemes, and refers to situations in which the target variable is subject to two (left and right) random observation limits. Doubly truncated data require specific corrections for the observational bias, and this affects a variety of settings including the estimation of marginal and multivariate distributions, regression problems, and multi-state models. In this work multivariate Efron–Petrosian integrals for doubly truncated data are introduced. These integrals naturally arise when the goal is the estimation of the mean of a general transformation which involves the doubly truncated variable and covariates. An asymptotic representation of the Efron–Petrosian integrals as a sum of i.i.d. terms is derived and, from this, consistency and distributional convergence are established. As a by-product, uniform i.i.d. representations for the marginal nonparametric maximum likelihood estimator and its corresponding weighting process are provided. Applications to correlation analysis, regression, and competing risks models are presented. A simulation study is reported too.

A nonparametric empirical Bayes approach to large-scale multivariate regression

Multivariate regression has many applications, ranging from time series prediction to genomics. Borrowing information across the outcomes can improve prediction error, even when outcomes are statistically independent. Many methods exist to implement this strategy, for example the multiresponse lasso, but choosing the optimal method for a given dataset is difficult. These issues are addressed by establishing a connection between multivariate linear regression and compound decision problems. A nonparametric empirical Bayes procedure that can learn the optimal regression method from the data itself is proposed. Furthermore, the proposed procedure is free of tuning parameters and performs well in simulations and in a multiple stock price prediction problem.

High-dimensional dynamic systems identification with additional constraints

This note presents a unified analysis of the identification of dynamical systems with low-rank constraints under high-dimensional scaling. This identification problem for dynamic systems is challenging due to the intrinsic dependency of the data. To alleviate this problem, we first formulate this identification problem into a multivariate linear regression problem with a row-sub-Gaussian measurement matrix using the more general input designs and the independently repeated sampling schemes. We then propose a nuclear norm heuristic method that estimates the parameter matrix of a dynamic system from a few input-state data samples. Based on this, we can extend the existing results. In this article, we consider two scenarios. (i) In the noiseless scenario, nuclear-norm minimization is introduced for promoting low-rank. We define the notion of weak restricted isometry property, which is weaker than the ordinary restricted isometry property, and show it holds with high probability for the row-sub-Gaussian measurement matrix. Thereby, the rank-minimization matrix can be exactly recovered from a finite number of data samples. (ii) In the noisy scenario, a regularized framework involving the nuclear norm penalty is established. We give the notion of operator norm curvature condition for the loss function and show it holds for the row-sub-Gaussian measurement matrix with high probability. Consequently, when specifying the suitable choice of the regularization parameter, the operator norm error of the optimal solution of this program has a sharp bound given a finite amount of data samples. This operator norm error bound is always smaller than the ordinary Frobenius norm error bound obtained in the existing work.

Beam data modeling of linear accelerators (linacs) through machine learning and its potential applications in fast and robust linac commissioning and quality assurance

Background and purposeTo propose a novel machine learning-based method for reliable and accurate modeling of linac beam data applicable to the processes of linac commissioning and QA. Materials and methodsWe hypothesize that the beam data is a function of inherent linac features and percentage depth doses (PDDs) and profiles of different field sizes are correlated with each other. The correlation is formulated as a multivariable regression problem using a machine learning framework. Varian TrueBeam beam data sets (n = 43) acquired from multiple institutions were used to evaluate the framework. The data sets included PDDs and profiles across different energies and field sizes. A multivariate regression model was trained for prediction of beam specific PDDs and profiles of different field sizes using a 10 × 10 cm2 field as input. ResultsPredictions of PDDs were achieved with a mean absolute percent relative error (%RE) of 0.19–0.35% across the different beam energies investigated. The maximum mean absolute %RE was 0.93%. For profile prediction, the mean absolute %RE was 0.66–0.93% with a maximum absolute %RE of 3.76%. The largest uncertainties in the PDD and profile predictions were found at the build-up region and at the field penumbra, respectively. The prediction accuracy increased with the number of training sets up to around 20 training sets. ConclusionsThrough this novel machine learning-based method we have shown accurate and reproducible generation of beam data for linac commissioning for routine radiation therapy. This method has the potential to simplify the linac commissioning procedure, save time and manpower while increasing the accuracy of the commissioning process.

DNA methylation-based age prediction using cell separation algorithm

The age of each individual can be predicted based on the alteration rule of DNA methylation with age. In this paper, an age prediction method is developed in order to solve multivariate regression problems from DNA methylation data, by optimizing the artificial neural network (ANN) model using a new proposed algorithm named the Cell Separation Algorithm (CSA). The CSA imitates cell separation action by using a differential centrifugation process involving multiple centrifugation steps and increasing the rotor speed in each step. The CSA performs similar to the centrifugal force in separating the solutions based on their objective function in different steps, with velocity increasing in each step. Firstly, 25 test functions are used to test the CSA. Secondly, the CSA is examined on three forms of age prediction problems from two body fluids (blood and saliva). The healthy blood samples, diseased blood samples and saliva samples are used to test the method's capability. The results of the CSA are compared not only with other methods proposed in previous studies, but also with the results from stochastic gradient descent (SGD), ADAM, and genetic algorithm (GA). The model results of CSA are extremely better than the four methods proposed in previous works that have not used ANN training process. The CSA also outperformed SGD, ADAM that employ the ANN model without ANN optimization by meta-heuristics. The CSA results are comparable (even superior) to the GA model which takes the advantages of both ANN and meta-heuristics.