Active Learning For Regression Research Articles

Active learning for regression by inverse distance weighting

This paper proposes an active learning (AL) algorithm to solve regression problems based on inverse-distance weighting functions for selecting the feature vectors to query. The algorithm has the following features: (i) supports both pool-based and population-based sampling; (ii) is not tailored to a particular class of predictors; (iii) can handle known and unknown constraints on the queryable feature vectors; and (iv) can run either sequentially, or in batch mode, depending on how often the predictor is retrained. The potentials of the method are shown in numerical tests on illustrative synthetic problems and real-world datasets. An implementation of the algorithm, that we call IDEAL (Inverse-Distance based Exploration for Active Learning), is available athttp://cse.lab.imtlucca.it/bemporad/ideal.

Active Learning Query Strategies for Linear Regression Based on Efficient Global Optimization

Active learning, a subfield of machine learning, can train a good model by selecting a minimum number of labeled samples. In many machine learning scenarios, needed information (such as the best value in unlabeled datasets) is acquired by prediction. When there is too little data in the training model, the prediction accuracy would obviously affect the accuracy of the results. To establish a high-performance regression model for a small dataset while accelerating the search for the best sample, a new active learning query strategy, EGO-ALR, that combines efficient global optimization (EGO) and active learning for regression (ALR) was proposed. It was found that the performance of EGO-ALR was significantly better than the original ALR query strategies in terms of the root mean square error (RMSE), correlation coefficient (CC), and opportunity cost (Oppo Cost). Specifically, EGO-ALR increased the Oppo Cost by an average of 25.27% when the RMSE and CC values were not more than 1.07% different from the original ALR. This study validated the efficiency and robustness of EGO-ALR approaches using 19 datasets from various domains and three distinct linear regression models (Ridge regression, Lasso, and Elastic network).

Adaptive Batch Size Selection in Active Learning for Regression

Training supervised machine learning models requires labeled examples. A judicious choice of examples is helpful when there is a significant cost associated with assigning labels. This article improves upon a promising extant method – Batch-mode Expected Model Change Maximization (B-EMCM) method – for selecting examples to be labeled for regression problems. Specifically, it develops and evaluates alternate strategies for adaptively selecting batch size in B-EMCM.<br/> By determining the cumulative error that occurs from the estimation of the stochastic gradient descent, a stop criteria for each iteration of the batch can be specified to ensure that selected candidates are the most beneficial to model learning. This new methodology is compared to B-EMCM via mean absolute error and root mean square error over ten iterations benchmarked against machine learning data sets.<br/> Using multiple data sets and metrics across all methods, one variation of AB-EMCM, the max bound of the accumulated error (AB-EMCM Max), showed the best results for an adaptive batch approach. It achieved better root mean squared error (RMSE) and mean absolute error (MAE) than the other adaptive and nonadaptive batch methods while reaching the result in nearly the same number of iterations as the non-adaptive batch methods.

Pool-based unsupervised active learning for regression using iterative representativeness-diversity maximization (iRDM)

Active learning (AL) selects the most beneficial unlabeled samples to label, and hence a better machine learning model can be trained from the same number of labeled samples. Most existing active learning for regression (ALR) approaches are supervised, which means the sampling process must use some label information, or an existing regression model. This paper considers completely unsupervised ALR, i.e., how to select the samples to label without knowing any true label information. We propose a novel unsupervised ALR approach, iterative representativeness-diversity maximization (iRDM), to optimally balance the representativeness and the diversity of the selected samples. Experiments on 60 datasets from various domains demonstrated its effectiveness. Our iRDM can be applied to both linear regression and kernel regression, and it even significantly outperforms supervised ALR when the number of labeled samples is small.

Optimizing Field Data Collection for Individual Tree Attribute Predictions Using Active Learning Methods

Light detection and ranging (lidar) data are nowadays a standard data source in studies related to forest ecology and environmental mapping. Medium/high point density lidar data allow to automatically detect individual tree crowns (ITCs), and they provide useful information to predict stem diameter and aboveground biomass of each tree represented by a detected ITC. However, acquisition of field data is necessary for the construction of prediction models that relate field data to lidar data and for validation of such models. When working at ITC level, field data collection is often expensive and time-consuming as accurate tree positions are needed. Active learning (AL) can be very useful in this context as it helps to select the optimal field trees to be measured, reducing the field data collection cost. In this study, we propose a new method of AL for regression based on the minimization of the field data collection cost in terms of distance to navigate between field sample trees, and accuracy in terms of root mean square error of the predictions. The developed method is applied to the prediction of diameter at breast heights (DBH) and aboveground biomass (AGB) of individual trees by using their height and crown diameter as independent variables and support vector regression. The proposed method was tested on two boreal forest datasets, and the obtained results show the effectiveness of the proposed selecting strategy to provide substantial improvements over the different iterations compared to a random selection. The obtained RMSE of DBH/AGB for the first dataset was 5.09 cm/95.5 kg with a cost equal to 8256/6173 m by using the proposed multi-objective method of selection. However, by using a random selection, the RMSE was 5.20 cm/102.1 kg with a cost equal to 28,391/30,086 m. The proposed approach can be efficient in order to get more accurate predictions with smaller costs, especially when a large forest area with no previous field data is subject to inventory and analysis.

Pool-Based Sequential Active Learning for Regression.

Active learning (AL) is a machine-learning approach for reducing the data labeling effort. Given a pool of unlabeled samples, it tries to select the most useful ones to label so that a model built from them can achieve the best possible performance. This paper focuses on pool-based sequential AL for regression (ALR). We first propose three essential criteria that an ALR approach should consider in selecting the most useful unlabeled samples: informativeness, representativeness, and diversity, and compare four existing ALR approaches against them. We then propose a new ALR approach using passive sampling, which considers both the representativeness and the diversity in both the initialization and subsequent iterations. Remarkably, this approach can also be integrated with other existing ALR approaches in the literature to further improve the performance. Extensive experiments on 11 University of California, Irvine, Carnegie Mellon University StatLib, and University of Florida Media Core data sets from various domains verified the effectiveness of our proposed ALR approaches.

Active learning for regression using greedy sampling

Regression problems are pervasive in real-world applications. Generally a substantial amount of labeled samples are needed to build a regression model with good generalization ability. However, many times it is relatively easy to collect a large number of unlabeled samples, but time-consuming or expensive to label them. Active learning for regression (ALR) is a methodology to reduce the number of labeled samples, by selecting the most beneficial ones to label, instead of random selection. This paper proposes two new ALR approaches based on greedy sampling (GS). The first approach (GSy) selects new samples to increase the diversity in the output space, and the second (iGS) selects new samples to increase the diversity in both input and output spaces. Extensive experiments on 10 UCI and CMU StatLib datasets from various domains, and on 15 subjects on EEG-based driver drowsiness estimation, verified their effectiveness and robustness.

Active Learning for Regression With Correlation Matching and Labeling Error Suppression

In this letter, we develop an active learning algorithm to optimize the selection of training data for robust linear regression. This algorithm selects training data based on the principle of correlation matching between the training dataset and the overall data pool. Considering the inevitable and potentially heavy human labeling errors, we model the probability of labeling errors based on the item response theory (IRT) and develop data screening techniques to control the error sparsity. Compressive sensing theory is then exploited for human labeling error suppression. This algorithm is robust even in the case of short training dataset with nonsparse labeling errors. Its performance is verified by simulations with both artificial data and real benchmark data. Experiments are also conducted to demonstrate the validity of the IRT-based human labeling error model and the superior performance of the algorithm in practical applications.

Batch Mode Active Learning for Regression With Expected Model Change.

While active learning (AL) has been widely studied for classification problems, limited efforts have been done on AL for regression. In this paper, we introduce a new AL framework for regression, expected model change maximization (EMCM), which aims at choosing the unlabeled data instances that result in the maximum change of the current model once labeled. The model change is quantified as the difference between the current model parameters and the updated parameters after the inclusion of the newly selected examples. In light of the stochastic gradient descent learning rule, we approximate the change as the gradient of the loss function with respect to each single candidate instance. Under the EMCM framework, we propose novel AL algorithms for the linear and nonlinear regression models. In addition, by simulating the behavior of the sequential AL policy when applied for k iterations, we further extend the algorithms to batch mode AL to simultaneously choose a set of k most informative instances at each query time. Extensive experimental results on both UCI and StatLib benchmark data sets have demonstrated that the proposed algorithms are highly effective and efficient.

Density Power Divergenceを用いたロバスト能動回帰学習

Density Power Divergenceを用いたロバスト能動回帰学習