Cost-sensitive Active Learning Research Articles

CSAL: Cost sensitive active learning for multi-source drifting stream

Multi-source stream classification is a prominent real-world problem challenged by the limited real labels and non-stationary environment. Despite growing research achievements in this field, most existing works solved this problem by requiring all real labels of source or target streams to conduct domain adaption or transfer learning mechanisms, which brings high labeling costs. However, in real-world applications, there are usually insufficient labeled data in both source and target streams, and the annotation cost of the source and target streams are generally unequal. Thus, we propose a Cost-Sensitive Active Learning (CSAL) method for multi-source drifting streams. Specifically, a multi-source ensemble framework with an asymmetry weighting mechanism is presented to ensure beneficial knowledge transfer and avoid the negative transfer. Then, a multi-perspective similarity estimation method is proposed to evaluate the similarity of source and target streams. On this basis, a novel cost-sensitive hybrid labeling strategy that combines volatility strategy and uncertainty strategy with a cost-sensitive budget control mechanism is proposed, which adaptively selects representative samples at the appropriate time. At last, a parallel multiple hypothesis drift detection method is proposed, which can efficiently utilize real labels to detect concept drift. Experimental results on real-world and synthetic data streams show that our CSAL outperforms the state-of-the-art methods with even fewer labels.

Cost-Sensitive Active Learning for Incomplete Data

Practical data often suffer from missing attribute values and lack of class labels. A reasonable machine learning scenario involves obtaining certain values and labels at cost on request. In this article, we propose the cost-sensitive active learning through unified evaluation and dynamic selection (CALS) algorithm to handle the learning task in this new scenario. For data representation, we consider misclassification cost, label query cost, and attribute query cost. For the cost/benefit estimation, we design a unified assessment of attribute values and labels with softmax regression. For the selection of attribute value and label, we propose an optimal acquisition scheme with permutation and greedy strategies. We perform experiments with synthetic, benchmark, and domain datasets. The results of the significance test verify the effectiveness of CALS and its superiority over cost-sensitive active learning and missing data imputation algorithms.

A cost-sensitive active learning algorithm: toward imbalanced time series forecasting

A cost-sensitive active learning algorithm: toward imbalanced time series forecasting

Cost sensitive active learning using bidirectional gated recurrent neural networks for imbalanced fault diagnosis

Most existing fault diagnosis methods may fail in the following three scenarios: (1) serial correlations exist in the process data; (2) fault data are much less than normal data; and (3) it is impractical to obtain enough labeled data. In this paper, a novel form of the bidirectional gated recurrent unit (BGRU) is developed to underpin effective and efficient fault diagnosis using cost sensitive active learning. Specifically, BGRU is devised to consider the dynamic behavior of a complex process. In the training phase of BGRU, the idea of weighting each training example is proposed to reduce the effect of class imbalance. Besides, in order to explore the unlabeled data, cost sensitive active learning is utilized to select the candidate instances. The effectiveness of the proposed method is evaluated on the Tennessee Eastman (TE) dataset and a real plasma etching process dataset. The experiment results show that the proposed cost senstive active learning bidirectional gated recurrent unit (CSALBGRU) method achieves better performance in both binary fault diagnosis and multi-class fault diagnosis.

Cost-sensitive active learning through statistical methods

Clustering-based active learning splits the data into a number of blocks and queries the labels of most the representative instances. When the cost of labeling and misclassification are considered, we also face a key issue: How many labels should be queried for a given block. In this paper, we present theoretical and practical statistical methods to handle this issue. The theoretical statistical method calculates the optimal number of query labels for a predefined label distribution. Considering label distributions for different clustering qualities, we obtain three hypothetical models, namely Gaussian, Uniform, and V models. The practical statistical method calculates empirical label distribution of the cluster blocks. Considering four popular clustering algorithms, we use symmetry and curve fitting techniques on 30 datasets to obtain empirical distributions. Inspired by three-way decision, we design an algorithm called the cost-sensitive active learning through statistical methods (CATS). Experiments were performed on 12 binary-class datasets for both the distribution evaluation and learning task. The results of significance tests verify the effectiveness of CATS and its superior performance with respect to state-of-the-art cost-sensitive active learning algorithms.

Annotation cost-sensitive active learning by tree sampling

Active learning is an important machine learning setup for reducing the labelling effort of humans. Although most existing works are based on a simple assumption that each labelling query has the same annotation cost, the assumption may not be realistic. That is, the annotation costs may actually vary between data instances. In addition, the costs may be unknown before making the query. Traditional active learning algorithms cannot deal with such a realistic scenario. In this work, we study annotation cost-sensitive active learning algorithms, which need to estimate the utility and cost of each query simultaneously. We propose a novel algorithm, the cost-sensitive tree sampling algorithm, that conducts the two estimation tasks together and solve it with a tree-structured model motivated from hierarchical sampling, a famous algorithm for traditional active learning. Extensive experimental results using datasets with simulated and true annotation costs validate that the proposed method is generally superior to other annotation cost-sensitive algorithms.

Cost-sensitive active learning with a label uniform distribution model

Active learning is a man-machine interaction scenario in which the machine acquires information actively from the expert. Cost-sensitive active learning balances the misclassification cost with the teacher cost paid for label queries. Inspired by granular computing (GrC) and three-way decision (3WD), this paper presents a new algorithm called cost-sensitive active learning through density clustering under a label uniform distribution model (CADU). CADU iteratively divides the universe, queries labels, and classifies instances until each label is queried or predicted. The density clustering technique is used to divide the universe into blocks. A label uniform distribution model is built to calculate the expected label distribution of each block. According to the teacher and misclassification cost settings, an optimization function is designed to determine the number of labels to be queried. Comparison study with 10 state-of-the-art algorithms are undertaken on 12 public datasets. Results show that CADU outperforms others in terms of average cost.

Cost-Sensitive Multitask Active Learning for Characterization of Urban Environments With Remote Sensing

We propose a novel cost-sensitive multitask active learning (CSMTAL) approach. Cost-sensitive active learning (CSAL) methods were recently introduced to specifically minimize labeling efforts emerging from ground surveys. Here, we build upon a CSAL method but compile a set of unlabeled samples from a learning set which can be considered relevant with respect to multiple target variables. To this purpose, a multitask meta-protocol based on alternating selection is implemented. It comprises a so-called one-sided selection (i.e., single-task AL selection for a reference target variable with simultaneous labeling of the residual target variables) with a changing leading variable in an iterative selection process. Experimental results are obtained for the city of Cologne, Germany. The target variables to be predicted, using features from remote sensing and a support vector machine framework, are “building type” and “roof type.” Comparative model accuracy evaluations underline the capability of the CSMTAL method to provide beneficial solutions with respect to a random sampling strategy and noncost-sensitive multitask active sampling.

Tri-partition cost-sensitive active learning through kNN

Active learning differs from the training–testing scenario in that class labels can be obtained upon request. It is widely employed in applications where the labeling of instances incurs a heavy manual cost. In this paper, we propose a new algorithm called tri-partition active learning through k-nearest neighbors (TALK). The optimization objective is to minimize the total teacher and misclassification costs. First, a k-nearest neighbors classifier is employed to divide unlabeled instances into three disjoint regions. Region I contains instances for which the expected misclassification cost is lower than the teacher cost, Region II contains instances to be labeled by human experts, and Region III contains the remaining instances. Various strategies are designed to determine which instances are in Region II. Second, instances in Regions I and II are labeled and added to the training set, and the tri-partition process is repeated until all instances have been labeled. Experiments are undertaken on eight University of California, Irvine, datasets using different cost settings. Compared with the state-of-the-art cost-sensitive classification and active learning algorithms, our new algorithm generally exhibits a lower total cost.

Optimised probabilistic active learning (OPAL)

In contrast to ever increasing volumes of automatically generated data, human annotation capacities remain limited. Thus, fast active learning approaches that allow the efficient allocation of annotation efforts gain in importance. Furthermore, cost-sensitive applications such as fraud detection pose the additional challenge of differing misclassification costs between classes. Unfortunately, the few existing cost-sensitive active learning approaches rely on time-consuming steps, such as performing self-labelling or tedious evaluations over samples. We propose a fast, non-myopic, and cost-sensitive probabilistic active learning approach for binary classification. Our approach computes the expected reduction in misclassification loss in a labelling candidate's neighbourhood. We derive and use a closed-form solution for this expectation, which considers the possible values of the true posterior of the positive class at the candidate's position, its possible label realisations, and the given labelling budget. The resulting myopic algorithm runs in the same linear asymptotic time as uncertainty sampling, while its non-myopic counterpart requires an additional factor of $$O(m \cdot \log m)$$O(m·logm) in the budget size. The experimental evaluation on several synthetic and real-world data sets shows competitive or better classification performance and runtime, compared to several uncertainty sampling- and error-reduction-based active learning strategies, both in cost-sensitive and cost-insensitive settings.

Cost-Sensitive Active Learning With Lookahead: Optimizing Field Surveys for Remote Sensing Data Classification

Active learning typically aims at minimizing the number of labeled samples to be included in the training set to reach a certain level of classification accuracy. Standard methods do not usually take into account the real annotation procedures and implicitly assume that all samples require the same effort to be labeled. Here, we consider the case where the cost associated with the annotation of a given sample depends on the previously labeled samples. In general, this is the case when annotating a queried sample is an action that changes the state of a dynamic system, and the cost is a function of the state of the system. In order to minimize the total annotation cost, the active sample selection problem is addressed in the framework of a Markov decision process, which allows one to plan the next labeling action on the basis of an expected long-term cumulative reward. This framework allows us to address the problem of optimizing the collection of labeled samples by field surveys for the classification of remote sensing data. The proposed method is applied to the ground sample collection for tree species classification using airborne hyperspectral images. Experiments carried out in the context of a real case study on forest inventory show the effectiveness of the proposed method.

Definition of Effective Training Sets for Supervised Classification of Remote Sensing Images by a Novel Cost-Sensitive Active Learning Method

This paper proposes a novel cost-sensitive active learning (CSAL) method to the definition of reliable training sets for the classification of remote sensing images with support vector machines. Unlike standard active learning (AL) methods, the proposed CSAL method redefines AL by assuming that the labeling cost of samples during ground survey is not identical, but depends on both the samples accessibility and the traveling time to the considered locations. The proposed CSAL method selects the most informative samples on the basis of three criteria: 1) uncertainty; 2) diversity; and 3) labeling cost. The labeling cost of the samples is modeled by a novel cost function that exploits ancillary data such as the road network map and the digital elevation model of the considered area. In the proposed method, the three criteria are applied in two consecutive steps. In the first step, the most uncertain samples are selected, whereas in the second step the uncertain samples that are diverse and have low labeling cost are chosen. In order to select the uncertain samples that optimize the diversity and cost criteria, we propose two different optimization algorithms. The first algorithm is defined on the basis of a sequential forward selection optimization strategy, whereas the second one relies on a genetic algorithm. Experimental results show the effectiveness of the proposed CSAL method compared to standard AL methods that neglect the labeling cost.

Cost-sensitive active learning for computer-assisted translation

Machine translation technology is not perfect. To be successfully embedded in real-world applications, it must compensate for its imperfections by interacting intelligently with the user within a computer-assisted translation framework. The interactive–predictive paradigm, where both a statistical translation model and a human expert collaborate to generate the translation, has been shown to be an effective computer-assisted translation approach. However, the exhaustive supervision of all translations and the use of non-incremental translation models penalizes the productivity of conventional interactive–predictive systems.We propose a cost-sensitive active learning framework for computer-assisted translation whose goal is to make the translation process as painless as possible. In contrast to conventional active learning scenarios, the proposed active learning framework is designed to minimize not only how many translations the user must supervise but also how difficult each translation is to supervise. To do that, we address the two potential drawbacks of the interactive–predictive translation paradigm. On the one hand, user effort is focused to those translations whose user supervision is considered more “informative”, thus, maximizing the utility of each user interaction. On the other hand, we use a dynamic machine translation model that is continually updated with user feedback after deployment. We empirically validated each of the technical components in simulation and quantify the user effort saved. We conclude that both selective translation supervision and translation model updating lead to important user-effort reductions, and consequently to improved translation productivity.

Cost-Sensitive Active Visual Category Learning

We present an active learning framework that predicts the tradeoff between the effort and information gain associated with a candidate image annotation, thereby ranking unlabeled and partially labeled images according to their expected "net worth" to an object recognition system. We develop a multi-label multiple-instance approach that accommodates realistic images containing multiple objects and allows the category-learner to strategically choose what annotations it receives from a mixture of strong and weak labels. Since the annotation cost can vary depending on an image's complexity, we show how to improve the active selection by directly predicting the time required to segment an unlabeled image. Our approach accounts for the fact that the optimal use of manual effort may call for a combination of labels at multiple levels of granularity, as well as accurate prediction of manual effort. As a result, it is possible to learn more accurate category models with a lower total expenditure of annotation effort. Given a small initial pool of labeled data, the proposed method actively improves the category models with minimal manual intervention.

Pessimistic cost-sensitive active learning of decision trees for profit maximizing targeting campaigns

In business applications such as direct marketing, decision-makers are required to choose the action which best maximizes a utility function. Cost-sensitive learning methods can help them achieve this goal. In this paper, we introduce Pessimistic Active Learning (PAL). PAL employs a novel pessimistic measure, which relies on confidence intervals and is used to balance the exploration/exploitation trade-off. In order to acquire an initial sample of labeled data, PAL applies orthogonal arrays of fractional factorial design. PAL was tested on ten datasets using a decision tree inducer. A comparison of these results to those of other methods indicates PAL's superiority.