Multi-target Regression Problems Research Articles

Image-based prediction and optimization of hysteresis properties of nanocrystalline permanent magnets using deep learning

We demonstrate the use of model order reduction and neural networks for estimating the hysteresis properties of nanocrystalline permanent magnets from microstructure. With a data-driven approach, we learn the demagnetization curve from data-sets created by grain growth and micromagnetic simulations. We show that the granular structure of a magnet can be encoded within a low-dimensional latent space. Latent codes are constructed using a variational autoencoder. The mapping of structure code to hysteresis properties is a multi-target regression problem. We apply deep neural network and use parameter sharing, in order to predict anchor points along the demagnetization curves from the magnet’s structure code. We present a proof of concept for microstructure design of nanocrystalline Nd2Fe14B permanent magnets using image-based machine learning. We show how new grain structures can be generated by interpolation between two points in the latent space and demonstrate inverse design. We show how to compute the granular microstructure that will give a demagnetization curve with a given remanence, squareness, and coercivity through optimization in latent space.

Filter method-based feature selection process for unattributed-identity multi-target regression problem

Unattributed-identity multi-target regression (UIMTR) is defined as a multi-target regression problem in which the identity of the target and predictor variables is not predefined. It is a problem that can be found in several real-world applications. For example, when historical data is available from a set of devices, but real-time data can only be requested from a subset of them (so called sentinels). For estimating real-time status of non-sentinels, it will be necessary to generate multi-target regression models. Therefore, attributing the identity of the real-time communicators (sentinels), i.e., the predictor variables, is a critical aspect. Moreover, unlike classical feature selection problems, the set of target variables is determined after applying the selection methods and not before, thus, some adaptations are necessary. We introduce three novel methods to solve the UIMTR and, after extensive evaluation, we demonstrate: (i) the feasibility of the methods, (ii) the usefulness of the approach, and (iii) the improvement over other classical techniques. The results have been evaluated from three perspectives: (i) the quality of the predictions, (ii) the stability of the methods and (iii) the execution time.

A systematic analysis of design choices in short-term taxi demand prediction models

Accurate prediction of short-term taxi demand allows taxi services to strategically position idle vehicles in areas with insufficient supply, thus reducing idle times for taxis and waiting times for passengers. Towards this end, state-of-the-art approaches use Machine Learning (ML) models based on historic demand data, such as Long Short-Term Memory (LSTM), Fully Connected Feed Forward Neural Network (FCNN), Convolutional Neural Network (CNN), Graph Convolutional Network (GCN), or a combination of these. The majority of approaches use a grid representation of the environment to discretize the pickup locations of trips and time bins to order the temporal dimension of the large multi-target regression problem. While these models show good accuracy, many \ extit{design choices}, such as the size of the grid cells or the number of previous time steps inserted into a model, are made without justification or without proper tuning. This paper systematically investigates these design decisions. Our hypothesis is that considering them jointly rather than in isolation can improve accuracy. We investigate the influence of (1) the selected grid cell size, (2) the number of input maps, (3) the usage of time shift to enlarge the training data, and (4) the potential of transfer learning on the accuracy of short-term taxi demand prediction. We selected different network types and evaluated them on two large real-world datasets. The evaluation shows that our model outperforms state-of-the-art short-term taxi demand prediction models.

Multi-target HSIC-Lasso via exploiting target correlations

Multi-target regression (MTR) has been widely applied due to its compact structure and better computational efficiency in learning multiple related tasks, and has received extensive attention in machine learning, while great challenge arises from handling complex nonlinear relationships in high-dimensional data. In this study, we propose a novel approach to nonlinear dimensionality reduction in MTR. Specifically, the Hilbert–Schmidt Independence Criterion Lasso (HSIC-Lasso) is introduced to captures nonlinear dependencies between input–output and inter-target relationships in a unified framework. Additionally, sparse learning is incorporated to effectively eliminate redundant and irrelevant features. The proposed method is guaranteed to find an optimal subset of maximally predictive features with minimal redundancy, and explores target correlations by treating the targets as input features, which achieves a better learning of the target dependencies and thus improves the effectiveness of the feature selection. Extensive experiments on twelve public data sets have shown that the proposed methods outperform the compared methods in feature selection for the MTR problem.

An ensemble-adaptive tree-based chain framework for multi-target regression problems

Multi-target regression has always been a challenging task in engineering applications. Nevertheless, it is easy to encounter problems such as low accuracy and inadequate robustness in some scenarios. To address these issues, an ensemble strategy considering correlations is proposed, named Ensemble-Adaptive Tree-based Correlation Chains. Specifically, a Follow-up Correlation Chaining strategy that quantifies the relationships among targets by arranging the L1 norms of correlations is suggested. Compared with other related strategies, it allows for the representation of these relationships through a single regressor chain. Under the proposed framework, the ensemble strategy integrates ten chains, wherein each chain adaptively updates the sample weights during training. This process involves employing the out-of-sample observations with new convergence criteria. Furthermore, the eXtreme Gradient Boosting is introduced as the base regressor to enhance the overall accuracy of the entire method. Finally, the proposed method is validated based on 25 multi-target datasets and a lightweight design of a high-speed rail bogie. The results demonstrate the superior accuracy and robustness compared to other state-of-the-art methods. In general, this study provides reliable predictions for specific scenarios and delivers practical significance in addressing relevant problems.

Deep learning-based multi-target regression for traffic-related air pollution forecasting

Traffic-related air pollution (TRAP) remains one of the main contributors to urban pollution and its impact on climate change cannot be overemphasised. Experts in developed countries strive to make optimal use of traffic and air quality data to gain valuable insights into its effect on public health. Over the years, the research community has developed advanced methods of forecasting traffic-related pollution using several machine learning methods albeit with persistent accuracy and insufficient data challenges. Despite the potentials of emerging techniques such as multi-target deep neural network to achieve optimal solutions, they are yet to be fully exploited in the air quality space due to their complexity and unavailability of the right training data. It is to this end that this study investigates the impact of integrating an updated data set including road elevation, vehicle emissions factor and background maps with traffic flow, weather and pollution data on TRAP forecasting. To explore the robustness and adaptability of our methodology, the study was carried out in one major city (London), one smaller city (Newport) and one large town (Chepstow) in the United Kingdom. The forecasting task was modelled as a multi-target regression problem and experiments were carried out to predict NO2,PM2.5 and PM10 concentrations over multiple timesteps. Fastai’s tabular model was used alongside prophet’s time-series model and scikit-learn’s multioutputregressor for experimentation with fastai recording the overall best performance. Statistical tests run using Friedman and Wilcoxon test also revealed the significance of the fastai model with a p-values < 0.05. Finally, a model explanation tool was then used to reveal the most and least influential features from the newly curated data set. Results showed traffic count and speed were part of the most contributing features. This result demonstrates the impact of these and other introduced features on TRAP forecasting and will serve as a foundation for related studies.

Multitarget Stochastic Configuration Network and Applications

Existing stochastic configuration network (SCN)-based modeling methods are underperformed in handling multitarget regression problems. An important reason is that they ignore the intertarget correlations, which have an important effect on improving the modeling accuracy. To enhance the performance of these SCN-based models, in this article, a novel multitarget SCN (MTSCN) modeling approach is presented. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> norm of a structure matrix can be utilized to explicitly reveal the correlations between multiple targets and an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> term are attached to the cost function of SCN. Considering the nonsmoothness of the constructed cost function, an alternating optimization algorithm is adopted to compute the structure matrix and the output weights of MTSCN. Then, a new supervisory mechanism is proposed to ensure the convergence of MTSCN. Finally, experimental results using the synthetic data and several real-world datasets show that the developed MTSCN is more superior to other modeling methods in resolving multitarget modeling problems.

IMon: Network Function Virtualisation Monitoring Based on a Unique Agent

As a key technology of 5G, NFV has attracted much attention. In addition, monitoring plays an important role, and can be widely used for virtual network function placement and resource optimisation. The existing monitoring methods focus on the monitoring load without considering they own resources needed. This raises a unique challenge: jointly optimising the NFV monitoring systems and minimising their monitoring load at runtime. The objective is to enhance the gain in real-time monitoring metrics at minimum monitoring costs. In this context, we propose a novel NFV monitoring solution, namely, iMon (Monitoring by inferring), that jointly optimises the monitoring process and reduces resource consumption. We formalise the monitoring process into a multitarget regression problem and propose three regression models. These models are implemented by a deep neural network, and an experimental platform is built to prove their availability and effectiveness. Finally, experiments also show that monitoring resource requirements are reduced, and the monitoring load is just 0.6% of that of the monitoring tool cAdvisor on our dataset.

Food Frequency Questionnaire Personalisation Using Multi-Target Regression.

Fondazione Bruno Kessler is developing a mobile app prototype for empowering citizens to improve their health conditions through different lifestyle interventions that will be incorporated into a mobile application for lifestyle promotion of the Province of Trento in the context of the Trentino Salute 4.0 Competence Center. The envisioned interventions are based on promoting behaviour change in various domains such as physical activity, mental health and nutrition. In particular, the nutrition component is a self-monitoring module that collects dietary habits to analyse them and recommend healthier eating behaviours. Dietary assessment is completed using a Food Frequency Questionnaire on the Mediterranean diet that is presented to the user as a grid of images. The questionnaire returns feedback on 11 aspects of nutrition. Although the questionnaire used in the application only consists of 24 questions, it still could be a bit overwhelming and a bit crowded when shown on the screen. In this paper, we tried to find a machine-learning-based solution to reduce the number of questions in the questionnaire. We proposed a method that uses the user’s previous answers as additional information to find the goals that need more attention. We compared this method with a case where the subset of questions is randomly selected and with a case where the subset is chosen using feature selection. We also explored how large the subset should be to obtain good predictions. All the experiments are conducted as a multi-target regression problem, which means several goals are predicted simultaneously. The proposed method adjusts well to the user in question and has the slightest error when predicting the goals.

Exploring the Hysteresis Properties of Nanocrystalline Permanent Magnets Using Deep Learning

We demonstrate the use of model order reduction and neural networks for estimating the hysteresis properties of nanocrystalline permanent magnets from microstructure. With a data-driven approach, we learn the demagnetization curve from data-sets created by grain growth and micromagnetic simulations. We show that the granular structure of a magnet can be encoded within a low-dimensional latent space. Latent codes are constructed using a variational autoencoder. The mapping of structure code to hysteresis properties is a multi-target regression problem. We apply deep neural network and use parameter sharing, in order to predict anchor points along the demagnetization curves from the magnet's structure code. The method is applied to study the magnetic properties of nanocrystalline permanent magnets. We show how new grain structures can be generated by interpolation between two points in the latent space and how the magnetic properties of the resulting magnets can be predicted.

Multi-target regression via non-linear output structure learning

The problem of simultaneously predicting multiple real-valued outputs using a shared set of input variables is known as multi-target regression and has attracted considerable interest in the past couple of years. The dominant approach in the literature for multi-target regression is to capture the dependencies between the outputs through a linear model and express it as an output mixing matrix. This modelling formalism, however, is too simplistic in real-world problems where the output variables are related to one another in a more complex and non-linear fashion. To address this problem, in this study, we propose a structural modelling approach where the correlations between output variables are modelled using a non-linear approach. In particular, we pose the multi-target regression problem as one of vector-valued composition function learning in the reproducing kernel Hilbert space and propose a non-linear structure learning approach to capture the relationship between the outputs via an output kernel. By virtue of using a non-linear output kernel function, the proposed approach can better discover non-linear dependencies among targets for improved prediction performance. An extensive evaluation conducted on different databases reveals the benefits of the proposed multi-target regression technique against the baseline and the state-of-the-art methods.

Copula-based conformal prediction for multi-target regression

There are relatively few works dealing with conformal prediction for multi-task learning issues, and this is particularly true for multi-target regression. This paper focuses on the problem of providing valid (i.e., frequency calibrated) multi-variate predictions. To do so, we propose to use copula functions for inductive conformal prediction, and illustrate our proposal by applying it to deep neural networks and random forests. We show that the proposed method ensures efficiency and validity for multi-target regression problems on various data sets.

VMFS: A VIKOR-based multi-target feature selection

This paper proposed a Multi-Criteria Decision-Making (MCDM) modeling to deal with multi-target regression problem. This model offered a feature ranking approach for multi-target regression by one of the famous MCDM algorithms called VIKOR. In fact, in this approach, the multi-target feature selection has become an MCDM model in which features are evaluated based on relations with targets. Our proposed method, VIKOR-based Multi-target Feature Selection (VMFS), first uses cosine similarity to construct the decision matrix for the MCDM process. Then VIKOR method is applied to the decision matrix to rank the features in the multi-target space. To illustrate the optimality and efficiency of the proposed method, we have compared our approach with some multi-output feature selection methods. The results show that our method in terms of average RRMSE is superior to other similar methods and performs in a short time, and it is more efficient than the other methods.

An ensemble-based semi-supervised feature ranking for multi-target regression problems

This study focuses on semi-supervised feature ranking (FR) applications for multi-target regression problems (MTR). As MTRs require prediction of several targets, we use a learning model that includes target interrelations via multi-objective trees. In processing the features for a semi-supervised learning model, transformation or scaling operations are usually required. To resolve this issue, we create a dissimilarity matrix via totally randomized trees to process the unsupervised information. Besides, we treat the split score function as a vector to make it suitable for considering each criterion regardless of their scales. We propose a semi-supervised FR scheme embedded to multi-objective trees that takes into account target and feature contributions simultaneously. Proposed FR score is compared with the state-of-the-art multi-target FR strategies via statistical analyses. Experimental studies show that proposed score significantly improves the performance of a recent tree-based and competitive multi-target learning model, i.e. predictive clustering trees. In addition, proposed approach outperforms its benchmarks when the available labelled data increase.

Rotation Forest for multi-target regression

The prediction of multiple numeric outputs at the same time is called multi-target regression (MTR), and it has gained attention during the last decades. This task is a challenging research topic in supervised learning because it poses additional difficulties to traditional single-target regression (STR), and many real-world problems involve the prediction of multiple targets at once. One of the most successful approaches to deal with MTR, although not the only one, consists in transforming the problem in several STR problems, whose outputs will be combined building up the MTR output. In this paper, the Rotation Forest ensemble method, previously proposed for single-label classification and single-target regression, is adapted to MTR tasks and tested with several regressors and data sets. Our proposal rotates the input space in an efficient and novel fashion, avoiding extra rotations forced by MTR problem decomposition. Four approaches for MTR are used: single-target (ST), stacked-single target (SST), Ensembles of Regressor Chains (ERC), and Multi-target Regression via Quantization (MRQ). For assessing the benefits of the proposal, a thorough experimentation with 28 MTR data sets and statistical tests are used, concluding that Rotation Forest, adapted by means of these approaches, outperforms other popular ensembles, such as Bagging and Random Forest.

Feature selection for semi-supervised multi-target regression using genetic algorithm

Multi-target regression (MTR) is an exciting area of machine learning where the challenge is to predict the values of more than one target variables which can take on continuous values. These variables may or may not be correlated. Such problems commonly occur in real life scenarios, and therefore, interest and research in this area has increased in recent times. Some examples of applications include analyzing brain-activity data gathered using multimedia sensors, stock information from continuous web data, data related to characteristics of the vegetation at a certain site, etc. For a real-world multi-target learning system, the problem can be further complicated when new issues emerge with very little data available. In such cases, a semi-supervised approach can be adopted. This paper proposes a Genetic Algorithm (GA) based semi-supervised technique on multi-target regression problems to predict new targets, using very small number of labelled examples by incorporating GA with MTR-SAFER. Experiments are carried out on real world MTR data sets. The proposed method isexplored with different variations and also compared with the state of the art MTR methods. Results have indicated a significantly better performance with the further benefit of having a reduced feature set.

Performing multi-target regression via gene expression programming-based ensemble models

Multi-Target Regression problem comprises the prediction of multiple continuous variables given a common set of input features, unlike traditional regression tasks, where just one output target is available. There are two major challenges when addressing this problem, namely the exploration of the inter-target dependencies and the modeling of complex input–output relationships. This work proposes a Symbolic Regression method following the basis of Gene Expression Programming paradigm to solve the multi-target regression problem, and called GEPMTR. It evolves a population of individuals, where each one represents a complete solution to the problem by using a multi-genic chromosome, and encodes a mathematical function for each target variable involving the input attributes. The proposed model can estimate the inter-target dependencies by applying some genetic operators. Furthermore, three ensemble-based methods are developed to better exploit the inter-target and input–output relationships. The effectiveness of the proposals is analyzed through an extensive experimental study on 18 datasets. The codification schema and the process followed to ensure a diverse population in GEPMTR lead to obtain an effective proposal to solve the MTR problem. Furthermore, the EGEPMTR-B ensemble method obtained the best performance across all proposed models, being the best in 8 out of 11 cases, demonstrating that more sophisticated mechanisms were not needed for ensuring that GEPMTR method would properly model the existing inter-target dependencies. Finally, the experimental results also showed that the proposed approach attains competitive results compared to state-of-the-art, showing the possibilities that can bring this research line for effectively solving the MTR problem.

URDNet: A Unified Regression Network for GGO Detection in Lung CT Images

We present a 3D deep neural network known as URDNet for detecting ground-glass opacity (GGO) nodules in 3D CT images. Prior work on GGO detection repurposes classifiers on a large number of windows to perform detection or fine-tuning by box regression based on a previous window classification step. Instead, we consider GGO detection as a multitarget regression problem to focus on the location of GGO. Furthermore, to capture multiscale information, we introduce a backbone network which is a contracting-expanding structure similar to 2D U-net, but we inject the source CT inputs into each layer in the contracting pathway to prevent source information loss at different scales. At last, we propose a two-stage training method for URDNet. In the first stage, the backbone of the network for feature extraction is trained, and in the second, the overall URDNet is fine-tuned based on the previous pretrained weights. By using this training method in conjunction with data augmentation and hard negative mining techniques, our URDNet can be effectively trained even on a small amount of annotated CT images. We evaluate the proposed method on the LIDC-IDRI dataset. It achieves the sensitivity of 90.8% with only 1 false positive per scan. Experimental results show that our detection method achieves the superior detection performance over the state-of-the-art methods. Due to its simplicity and effective, URDNet can be easier to apply to medical IoT systems for improving the efficiency of overall health systems.

A machine-learning framework for predicting multiple air pollutants' concentrations via multi-target regression and feature selection.

Air pollution is considered one of the biggest threats for the ecological system and human existence. Therefore, air quality monitoring has become a necessity in urban and industrial areas. Recently, the emergence of Machine Learning techniques justifies the application of statistical approaches for environmental modeling, especially in air quality forecasting. In this context, we propose a novel feature ranking method, termed as Ensemble of Regressor Chains-guided Feature Ranking (ERCFR) to forecast multiple air pollutants simultaneously over two cities. This approach is based on a combination of one of the most powerful ensemble methods for Multi-Target Regression problems (Ensemble of Regressor Chains) and the Random Forest permutation importance measure. Thus, feature selection allowed the model to obtain the best results with a restricted subset of features. The experimental results reveal the superiority of the proposed approach compared to other state-of-the-art methods, although some cautions have to be considered to improve the runtime performance and to decrease its sensitivity over extreme and outlier values.

Performing Multi-Target Regression via a Parameter Sharing-Based Deep Network.

Multi-target regression (MTR) comprises the prediction of multiple continuous target variables from a common set of input variables. There are two major challenges when addressing the MTR problem: the exploration of the inter-target dependencies and the modeling of complex input-output relationships. This paper proposes a neural network model that is able to simultaneously address these two challenges in a flexible way. A deep architecture well suited for learning multiple continuous outputs is designed, providing some flexibility to model the inter-target relationships by sharing network parameters as well as the possibility to exploit target-specific patterns by learning a set of nonshared parameters for each target. The effectiveness of the proposal is analyzed through an extensive experimental study on 18 datasets, demonstrating the benefits of using a shared representation that exploits the commonalities between target variables. According to the experimental results, the proposed model is competitive with respect to the state-of-the-art in MTR.