Multiple-output Regression Research Articles

Development of a learner model tool for predicting strength and embodied carbon for lightweight concrete production

The demand for sustainable concrete in meeting the net zero carbon target places a burden in optimizing concrete response to structural strength that satisfy acceptable embodied carbon. In most cases, a low carbon concrete is deficient in structural requirement and vice versa. This dilemma informs the need for a tool that can predict compressive strength as well as embodied carbon using the same input data. Since the use of alternative materials as cement replacement to enhance sustainability is emerging in the quest for a sustainable concrete, an optimal material that satisfy both conditions of structural integrity and sustainability is still lacking. Paucity of data in the emerging lightweight low carbon concrete using alternative materials, portends an upheave in the bias for prediction of the behaviour of lightweight low carbon concrete. This study therefore uses concrete data of lightweight low carbon concrete from laboratory experiment for the prediction of compressive strength and embodied carbon with their performance evaluated using eight machine leaning regression models. The results obtained indicates that the XG boost regression model exhibited excellent performance with a low Mean Squared Error (MSE) of 50.15, Mean absolute error(MAE) = 5.26, Mean absolute percentage error(MAPE) = 11.76 %, Explained variance score = 0.97, Root mean square error(RMSE) = 7.08 and a high R squared value of 0.96. The tool predicted compressive strength and embodied carbon for lightweight carbon concrete using multiple output regression such that the output can be limited to the yearly structural embodied carbon threshold to achieving 2050 net zero target. The developed tool when compared with concrete of similar mix ingredients performed more than 95 % in predicting concrete compressive strength and associated embodied carbon. In line with the inclusion of embodied carbon in carbon regulations of buildings in the UK as suggested by the professionals in the construction industry, the developed learner model and prediction tool has integrated concrete strength and embodied carbon to initiate a holistic approach to design and construction, balancing performance, cost, and environmental impact.

Optimum design of a novel Ku-band rasorber for RADAR warfare systems using ML neural network

The design, fabrication, and testing of a conformal Frequency Selective Rasorber (FSR) operating at A-T-A mode is designed for RADAR warfare systems. In the design of a miniaturized single-layer FSR element, multiple parameters influencing the absorption and transmission characteristics need to be optimized. This simulation using conventional methods consumes more simulation time. Thus, the geometrical parameters are optimized and predicted using supervised machine learning (ML) techniques to expedite the process. The multiple output regression neural network (MORNN) is used to generate multiple input and output features from the dataset. The ML algorithm is trained using the datasets generated from the electromagnetic solver using which a scalable FSR is synthesized. The reflection coefficient, (|S11|), and transmission coefficient (|S21|) are used as input data, and the dimension of the FSR unit cell for the user input frequency requirements are derived as output data. The extracted dimensions of the FSR offered a small mean square error (MSE) of 0.02 between the desired and observed results. The designed FSR offers absorption at 11.5 GHz and 18.9 GHz while the transmission window extends from 15.02 GHz to 16.09 GHz. The neural network results are endorsed using the EM simulation tool and validated by experimental measurements.

Prediction of Wave Spectral Parameters Using Multiple-Output Regression Models to Support the Execution of Marine Operations

Abstract Execution of a marine operation (MO) requires coordinated actions of several vessels conducting simultaneous and sequential offshore activities. These activities have their operational limits given in terms of environmental parameters. Wave parameters are important because of their high energetic level. During the execution of a MO, forecast wave spectral parameters, i.e., significant wave height (Hs), peak period (Tp), and peak direction, are used to make an on-board decision. For critical operations, the use of forecasts can be complemented with buoy measurements. This paper proposes to use synthetic statistics of vessel dynamic responses to predict “real-time” wave spectral parameters using multi-output machine learning (ML) regression algorithms. For a case study of a vessel with no forward speed, it is observed that the random forest model predicts accurate Hs and Tp parameters. The prediction of wave direction is not very accurate but it can be corrected with on-board observations. The random forest model has good performance; it is efficient, useful for practical purposes, and comparable with other deep learning models reported in the scientific literature. Findings from this research can be valuable for real-time assessment of wave spectral parameters, which are necessary to support decision-making during the execution of MOs.

Multi‐output regression for imbalanced data stream

AbstractIn this article, we describe an imbalanced regression method for making predictions over imbalanced data streams. We present MORSTS (Multiple Output Regression for Streaming Time Series), an online ensemble regressors devoted to non‐stationary and imbalanced data streams. MORSTS relies on several multiple output regressor submodels, adopts a cost sensitive weighting technique for dealing with imbalanced datasets, and handles overfitting by means of the K‐fold cross validation. For assessment purposes, experiments have been conducted on known real datasets and compared with known base regression techniques.

Data Depth and Multiple Output Regression, the Distorted M-Quantiles Approach

For a univariate distribution, its M-quantiles are obtained as solutions to asymmetric minimization problems dealing with the distance of a random variable to a fixed point. The asymmetry refers to the different weights awarded to the values of the random variable at either side of the fixed point. We focus on M-quantiles whose associated losses are given in terms of a power. In this setting, the classical quantiles are obtained for the first power, while the expectiles correspond to quadratic losses. The M-quantiles considered here are computed over distorted distributions, which allows to tune the weight awarded to the more central or peripheral parts of the distribution. These distorted M-quantiles are used in the multivariate setting to introduce novel families of central regions and their associated depth functions, which are further extended to the multiple output regression setting in the form of conditional and regression regions and conditional depths.

A Bayesian Approach to Multiple-Output Quantile Regression

This article presents a Bayesian approach to multiple-output quantile regression. The prior can be elicited as ex-ante knowledge of the distance of the τ-Tukey depth contour to the Tukey median, the first prior of its kind. The parametric model is proven to be consistent and a procedure to obtain confidence intervals is proposed. A proposal for nonparametric multiple-output regression is also presented. These results add to the literature of misspecified Bayesian modeling, consistency, and prior elicitation for nonparametric multivariate modeling. The model is applied to the Tennessee Project Steps to Achieving Resilience (STAR) experiment and finds a joint increase in τ-quantile subpopulations for mathematics and reading scores given a decrease in the number of students per teacher. Supplementary materials for this article are available online.

Efficient Fully Distribution-Free Center-Outward Rank Tests for Multiple-Output Regression and MANOVA

Extending rank-based inference to a multivariate setting such as multiple-output regression or MANOVA with unspecified d-dimensional error density has remained an open problem for more than half a century. None of the many solutions proposed so far is enjoying the combination of distribution-freeness and efficiency that makes rank-based inference a successful tool in the univariate setting. A concept of center-outward multivariate ranks and signs based on measure transportation ideas has been introduced recently. Center-outward ranks and signs are not only distribution-free but achieve in dimension d > 1 the (essential) maximal ancillarity property of traditional univariate ranks. In the present case, we show that fully distribution-free testing procedures based on center-outward ranks can achieve parametric efficiency. We establish the Hájek representation and asymptotic normality results required in the construction of such tests in multiple-output regression and MANOVA models. Simulations and an empirical study demonstrate the excellent performance of the proposed procedures.

Big data-oriented wheel position and geometry calculation for cutting tool groove manufacturing based on AI algorithms

Groove is a key structure of high-performance integral cutting tools. It has to be manufactured by 5-axis grinding machine due to its complex spatial geometry and hard materials. The crucial manufacturing parameters (CMP) are grinding wheel positions and geometries. However, it is a challenging problem to solve the CMP for the designed groove. The traditional trial-and-error or analytical methods have defects such as time-consuming, limited-applying, and low accuracy. In this study, the problem is translated into a multiple output regression model of groove manufacture (MORGM) based on the big data technology and AI algorithms. The inputs are 34 groove geometry features, and the outputs are 5 CMP. Firstly, two groove machining big data sets with different range are established, each of which is includes 46,656 records. They are used as data resource for MORGM. Secondly, 7 AI algorithms, including linear regression, k nearest-neighbor regression, decision trees, random forest regression, support vector regression, and ANN algorithms, are discussed to build the model. Then, 28 experiments are carried out to test the big data set and algorithms. Finally, the best MORGM is built by ANN algorithm and the big data set with a larger range. The results show that CMP can be calculated accurately and conveniently by the built MORGM.

Measure Transportation and Statistical Decision Theory

Unlike the real line, the real space, in dimension d ≥ 2, is not canonically ordered. As a consequence, extending to a multivariate context fundamental univariate statistical tools such as quantiles, signs, and ranks is anything but obvious. Tentative definitions have been proposed in the literature but do not enjoy the basic properties (e.g., distribution-freeness of ranks, their independence with respect to the order statistic, their independence with respect to signs) they are expected to satisfy. Based on measure transportation ideas, new concepts of distribution and quantile functions, ranks, and signs have been proposed recently that, unlike previous attempts, do satisfy these properties. These ranks, signs, and quantiles have been used, quite successfully, in several inference problems and have triggered, in a short span of time, a number of applications: fully distribution-free testing for multiple-output regression, MANOVA, and VAR models; R-estimation for VARMA parameters; distribution-free testing for vector independence; multiple-output quantile regression; nonlinear independent component analysis; and so on.

Pig Weight and Body Size Estimation Using a Multiple Output Regression Convolutional Neural Network: A Fast and Fully Automatic Method.

Pig weight and body size are important indicators for producers. Due to the increasing scale of pig farms, it is increasingly difficult for farmers to quickly and automatically obtain pig weight and body size. Due to this problem, we focused on a multiple output regression convolutional neural network (CNN) to estimate pig weight and body size. DenseNet201, ResNet152 V2, Xception and MobileNet V2 were modified into multiple output regression CNNs and trained on modeling data. By comparing the estimated performance of each model on test data, modified Xception was selected as the optimal estimation model. Based on pig height, body shape, and contour, the mean absolute error (MAE) of the model to estimate body weight (BW), shoulder width (SW), shoulder height (SH), hip width (HW), hip width (HH), and body length (BL) were 1.16 kg, 0.33 cm, 1.23 cm, 0.38 cm, 0.66 cm, and 0.75 cm, respectively. The coefficient of determination (R2) value between the estimated and measured results was in the range of 0.9879–0.9973. Combined with the LabVIEW software development platform, this method can estimate pig weight and body size accurately, quickly, and automatically. This work contributes to the automatic management of pig farms.

Some futures for cognitive modeling and architectures: design patterns for including better interaction with the world, moderators, and improved model to data fits (and so can you)

We note some future areas for work with cognitive models and agents that as Colbert (I am America (and so can you!), 2007) notes, “so can you”. We present three approaches as something like design patterns, so they can be applied to other architectures and tasks. These areas are: (a) Interacting directly with the screen-as-world. It is now possible for models to interact with uninstrumented interfaces both on the machine that the model is running on as well as remote machines. Improved interaction can not only support a broader range of behavior but also make the interaction more accurately model human behavior on tasks that include interaction. Just one implication is that this will force models to have more knowledge about interaction, an area that has been little modeled but essential for all tasks. (b) Providing the cognitive architecture with more representation of the body. In our example, we provide a physiological substrate to implement behavioral moderators’ effects on cognition. Cognitive architectures can now be broader in the measurements they predict and correspond to. This approach provides a more complete and theoretically appropriate way to include new aspects of behavior including stressor effects and emotions in models. And (c) using machine learning techniques, particularly genetic algorithms (GAs), to fit models to data. Because of the model complexity, this is equivalent to performing a multi-variable non-linear stochastic multiple-output regression. Doing this by hand is completely inadequate. While there is a danger of overfitting using a GA, these fits can help provide a better understanding of the model and architecture, including how the architecture changes under moderators such stress. This paper also includes some notes on model maintenance and reporting.

A COMPACT OPTIMAL LEARNING MACHINE

Artificial neural networks (ANNs) have been developed and applied to a variety of problems, such as pattern recognition, clustering, function approximation, forecasting, optimization, etc. However, existing ANNs have a high computational cost, since their learning methods are mostly based on a parameter tuning approach. Extreme learning machine (ELM) is a state-of-the-art method that generally dramatically reduces the computational cost. An analysis of the ELM method reveals that there are unsolved key factors, including inefficient hidden node construction, redundant hidden nodes, and unstable results. Therefore, we describe a new learning machine based on analytical incremental learning (AIL) in conjunction with principal component analysis (PCA). This learning machine, PCA-AIL, inherited the advantages from the original one and solved the unsolved key factors of ELM, and also extended AIL capability to serve a multiple-output structure. PCA-AIL was implemented with a single-layer feed-forward neural network architecture, used an adaptive weight determination technique to achieve a compact optimal structure and also used objective relations to support multiple output regression tasks. PCA-AIL has two steps: objective relation estimation and multiple optimal hidden node constructions. In the first step, PCA estimated the objective relations from multiple-output residual errors. In the second step, the multiple optimal nodes were obtained from objective relations and added to the model. PCA-AIL was tested with 16 multiple-objective regression datasets. PCA-AIL mostly outperformed other methods (ELM, EM-ELM, CP-ELM, DP-ELM, PCA-ELM, EI-ELM) in terms of fast testing speed – 0.0017 second, a compact model – 19.9 nodes, an accurate performance – RMSE 0.11261, and a stable result – S.D. of RMSE 0.00911: reported in averaged.

A penalized regression model for the joint estimation of eQTL associations and gene network structure

In this work, we present a new approach for jointly performing eQTL mapping and gene network inference while encouraging a transfer of information between the two tasks. We address this problem by formulating it as a multiple-output regression task in which we aim to learn the regression coefficients while simultaneously estimating the conditional independence relationships among the set of response variables. The approach we develop uses structured sparsity penalties to encourage the sharing of information between the regression coefficients and the output network in a mutually beneficial way. Our model, inverse-covariance-fused lasso, is formulated as a biconvex optimization problem that we solve via alternating minimization. We derive new, efficient optimization routines to solve each convex sub-problem that are based on extensions of state-of-the-art methods. Experiments on both simulated data and a yeast eQTL dataset demonstrate that our approach outperforms a large number of existing methods on the recovery of the true sparse structure of both the eQTL associations and the gene network. We also apply our method to a human Alzheimer’s disease dataset and highlight some results that support previous discoveries about the disease.

Dynamic Structure Embedded Online Multiple-Output Regression for Streaming Data.

Online multiple-output regression is an important machine learning technique for modeling, predicting, and compressing multi-dimensional correlated data streams. In this paper, we propose a novel online multiple-output regression method, called MORES, for streaming data. MORES can dynamically learn the structure of the regression coefficients to facilitate the model's continuous refinement. Considering that limited expressive ability of regression models often leading to residual errors being dependent, MORES intends to dynamically learn and leverage the structure of the residual errors to improve the prediction accuracy. Moreover, we introduce three modified covariance matrices to extract necessary information from all the seen data for training, and set different weights on samples so as to track the data streams' evolving characteristics. Furthermore, an efficient algorithm is designed to optimize the proposed objective function, and an efficient online eigenvalue decomposition algorithm is developed for the modified covariance matrix. Finally, we analyze the convergence of MORES in certain ideal condition. Experiments on two synthetic datasets and three real-world datasets validate the effectiveness and efficiency of MORES. In addition, MORES can process at least 2,000 instances per second (including training and testing) on the three real-world datasets, more than 12 times faster than the state-of-the-art online learning algorithm.

Directional quantile regression in R

Recently, the eminently popular standard quantile regression has been generalized to the multiple-output regression setup by means of directional regression quantiles in two rather interrelated ways. Unfortunately, they lead to complicated optimization problems involving parametric programming, and this may be the main obstacle standing in the way of their wide dissemination. The presented R package modQR is intended to address this issue. It originates as a quite faithful translation of the authors' moQuantile toolbox for Octave and MATLAB, and provides all the necessary computational support for both the directional multiple-output quantile regression methods to the wide statistical public. The article offers a concise summary of the statistical theory behind modQR, overviews the package in brief, points out its departures from moQuantile, comments on its use and performance, and demonstrates its application.

Multiple output regression with latent noise

In high-dimensional data, structured noise caused by observed and unobserved factors affecting multiple target variables simultaneously, imposes a serious challenge for modeling, by masking the often weak signal. Therefore, (1) explaining away the structured noise in multiple-output regression is of paramount importance. Additionally, (2) assumptions about the correlation structure of the regression weights are needed. We note that both can be formulated in a natural way in a latent variable model, in which both the interesting signal and the noise are mediated through the same latent factors. Under this assumption, the signal model then borrows strength from the noise model by encouraging similar effects on correlated targets. We introduce a hyperparameter for the latent signal-to-noise ratio which turns out to be important for modelling weak signals, and an ordered infinite-dimensional shrinkage prior that resolves the rotational unidentifiability in reduced-rank regression models. Simulations and prediction experiments with metabolite, gene expression, FMRI measurement, and macroeconomic time series data show that our model equals or exceeds the state-of-the-art performance and, in particular, outperforms the standard approach of assuming independent noise and signal models.

Novel applications of multitask learning and multiple output regression to multiple genetic trait prediction.

Given a set of biallelic molecular markers, such as SNPs, with genotype values encoded numerically on a collection of plant, animal or human samples, the goal of genetic trait prediction is to predict the quantitative trait values by simultaneously modeling all marker effects. Genetic trait prediction is usually represented as linear regression models. In many cases, for the same set of samples and markers, multiple traits are observed. Some of these traits might be correlated with each other. Therefore, modeling all the multiple traits together may improve the prediction accuracy. In this work, we view the multitrait prediction problem from a machine learning angle: as either a multitask learning problem or a multiple output regression problem, depending on whether different traits share the same genotype matrix or not. We then adapted multitask learning algorithms and multiple output regression algorithms to solve the multitrait prediction problem. We proposed a few strategies to improve the least square error of the prediction from these algorithms. Our experiments show that modeling multiple traits together could improve the prediction accuracy for correlated traits. Availability and implementation: The programs we used are either public or directly from the referred authors, such as MALSAR (http://www.public.asu.edu/~jye02/Software/MALSAR/) package. The Avocado data set has not been published yet and is available upon request.Contact: dhe@us.ibm.com

Sparse conditional copula models for structured output regression

We deal with the multiple output regression task where the central theme is to capture the sparse output correlation among the output variables. Sparse inverse covariance learning of linear Gaussian conditional models has been recently studied, shown to achieve superb prediction performance. However, it can fail when the underlying true input–output process is non-Gaussian and/or non-linear. We introduce a novel sparse conditional copula model to represent the joint density of the output variables. By incorporating a Gaussian copula function, yet modeling univariate marginal densities by (non-Gaussian) mixtures of experts, we achieve high flexibility in representation that admits non-linear and non-Gaussian densities. We then propose a sparse learning method for this copula-based model that effectively imposes sparsity in the conditional dependency among output variables. The learning optimization is efficient as it can be decomposed into gradient-descent marginal density estimation and the sparse inverse covariance learning for the copula function. Improved performance of the proposed approach is demonstrated on several interesting image/vision tasks with high dimensions.

Elliptical multiple-output quantile regression and convex optimization

This article extends linear quantile regression to an elliptical multiple-output regression setup. The definition of the proposed concept leads to a convex optimization problem. Its elementary properties, and the consistency of its sample counterpart, are investigated. An empirical application is provided.

Sparse inverse covariance learning of conditional Gaussian mixtures for multiple-output regression

We consider the task of multiple-output regression where both input and output are high-dimensional. Due to the limited amount of training samples compared to data dimensions, properly imposing loose statistical dependency in learning a regression model is crucial for reliable prediction accuracy. The sparse inverse covariance learning of conditional Gaussian random fields has been recently emerging to achieve this goal, shown to exhibit superior performance to non-sparse approaches. However, one of its main drawbacks is the strong assumption of linear Gaussianity in modeling the input-output relationship. For certain application domains, the assumption might be too restricted and less powerful in representation, and consequently, prediction based on the wrong models can result in suboptimal performance. In this paper, we extend the idea of sparse learning to a non-Gaussian model, especially the powerful conditional Gaussian mixture. For this latent-variable model, we propose a novel sparse inverse covariance learning algorithm based on the expectation-maximization lower-bound optimization technique. It is shown that each M-step reduces to solving the regular sparse inverse covariance estimation of linear Gaussian models, in conjunction with estimating sparse logistic regression. We demonstrate the improved prediction performance of the proposed algorithm over exisitng methods on several datasets.