Labeling Budget Research Articles

Robust online active learning with cluster-based local drift detection for unbalanced imperfect data

With the rapid development of data-driven technologies, a massive amount of actual data emerges from industrial systems, forming data stream. Their data distribution may change over time and outliers may be generated as unbalanced imperfect data due to time-varying working condition, aging equipment, etc. Previous methods struggle with the dual challenges of concept drift and unbalance, however, fail to efficiently distinguishing outliers from a drift under the limited labeling budget, causing the performance degradation. To address the issue, robust online active learning with cluster-based local drift detection is proposed to classify unbalanced imperfect data stream with the above characteristics. The cluster-based local drift detection is first designed to capture a new concept and recognize the corresponding drifted regions. Following that, an improved active learning mechanism is presented to distinguish outliers from a drift, and select most valuable instances for labeling and updating ensemble classifier. Experimental results for eight synthetic and four real-world data streams show that the proposed method outperforms seven comparative methods on classification accuracy and robustness.

Learning evolving prototypes for imbalanced data stream classification with limited labels

Real-world data streams often exhibit long-tailed distributions with heavy class imbalance, posing great challenges for data stream classification, especially in the case of label scarcity and concept drift. Several active learning methods have been proposed to address this problem by selecting the most valuable instances for labeling. However, existing methods often struggle to dynamically identify the most valuable instances that truly represent the current concept while still requiring a large label budget. In this work, we propose a new algorithm, LEPID, to combine dynamic micro-cluster concept modeling and local entropy modeling to select current important concepts and prototypes. Specifically, we give greater weight to concept drift prototypes and minority prototypes to focus more on those regions that represent current concepts. We use a local entropy strategy based on micro-clusters to select the most valuable instances for labeling and reduce the label budget. Extensive experiments on real-world and synthetic imbalanced datasets show that, compared to state-of-the-art algorithms, our method can naturally adapt to concept drift and dynamically capture the current and most valuable prototypes to achieve better results even in the case of label scarcity.

Composite Active Learning: Towards Multi-Domain Active Learning with Theoretical Guarantees

Active learning (AL) aims to improve model performance within a fixed labeling budget by choosing the most informative data points to label. Existing AL focuses on the single-domain setting, where all data come from the same domain (e.g., the same dataset). However, many real-world tasks often involve multiple domains. For example, in visual recognition, it is often desirable to train an image classifier that works across different environments (e.g., different backgrounds), where images from each environment constitute one domain. Such a multi-domain AL setting is challenging for prior methods because they (1) ignore the similarity among different domains when assigning labeling budget and (2) fail to handle distribution shift of data across different domains. In this paper, we propose the first general method, dubbed composite active learning (CAL), for multi-domain AL. Our approach explicitly considers the domain-level and instance-level information in the problem; CAL first assigns domain-level budgets according to domain-level importance, which is estimated by optimizing an upper error bound that we develop; with the domain-level budgets, CAL then leverages a certain instance-level query strategy to select samples to label from each domain. Our theoretical analysis shows that our method achieves a better error bound compared to current AL methods. Our empirical results demonstrate that our approach significantly outperforms the state-of-the-art AL methods on both synthetic and real-world multi-domain datasets. Code is available at https://github.com/Wang-ML-Lab/multi-domain-active-learning.

The impacts of active and self-supervised learning on efficient annotation of single-cell expression data

A crucial step in the analysis of single-cell data is annotating cells to cell types and states. While a myriad of approaches has been proposed, manual labeling of cells to create training datasets remains tedious and time-consuming. In the field of machine learning, active and self-supervised learning methods have been proposed to improve the performance of a classifier while reducing both annotation time and label budget. However, the benefits of such strategies for single-cell annotation have yet to be evaluated in realistic settings. Here, we perform a comprehensive benchmarking of active and self-supervised labeling strategies across a range of single-cell technologies and cell type annotation algorithms. We quantify the benefits of active learning and self-supervised strategies in the presence of cell type imbalance and variable similarity. We introduce adaptive reweighting, a heuristic procedure tailored to single-cell data—including a marker-aware version—that shows competitive performance with existing approaches. In addition, we demonstrate that having prior knowledge of cell type markers improves annotation accuracy. Finally, we summarize our findings into a set of recommendations for those implementing cell type annotation procedures or platforms. An R package implementing the heuristic approaches introduced in this work may be found at https://github.com/camlab-bioml/leader.

Sparsely-Supervised Object Tracking.

Recent years have witnessed the incredible performance boost of data-driven deep visual object trackers. Despite the success, these trackers require millions of sequential manual labels on videos for supervised training, implying the heavy burden of human annotating. This raises a crucial question: how to train a powerful tracker from abundant videos using limited manual annotations? In this paper, we challenge the conventional belief that frame-by-frame labeling is indispensable, and show that providing a small number of annotated bounding boxes in each video is sufficient for training a strong tracker. To facilitate that, we design a novel SParsely-supervised Object Tracking (SPOT) framework. It regards the sparsely annotated boxes as anchors and progressively explores in the temporal span to discover unlabeled target snapshots. Under the teacher-student paradigm, SPOT leverages the unique transitive consistency inherent in the tracking task as supervision, extracting knowledge from both anchor snapshots and unlabeled target snapshots. We also utilize several effective training strategies, i.e., IoU filtering, asymmetric augmentation, and temporal calibration to further improve the training robustness of SPOT. The experimental results demonstrate that, given less than 5 labels for each video, trackers trained via SPOT perform on par with their fully-supervised counterparts. Moreover, our SPOT exhibits two desirable properties: 1) SPOT enables us to fully exploit large-scale video datasets by efficiently allocating sparse labels to more videos even under a limited labeling budget; 2) when equipped with a target discovery module, SPOT can even learn from purely unlabeled videos for performance gain. We hope this work could inspire the community to rethink the current annotation principles and make a step towards practical label-efficient deep tracking.

Dynamic budget allocation for sparsely labeled drifting data streams

Learning from non-stationary data streams is inherently challenging due to their evolving nature and concept drift. Furthermore, the assumption that all instances come labeled is often impractical in real-world applications. Many strategies have been proposed to tackle learning from sparsely labeled data streams. However, they typically rely on fixed labeling budgets, which can be a limitation in the context of drifting data streams. In this study, we introduce a novel active learning strategy that dynamically manages the labeling budget to optimize its utilization and adapt promptly to concept drift. Our approach continuously monitors the data stream for concept drift, and upon detecting such drift, it dynamically increases the maximum labeling budget for a predefined time window. This adjustment provides the classifier with more flexibility to adapt to the new concept. We conducted experiments using 7 synthetic data generators encompassing various drifting scenarios and 7 real-world data streams with different labeling budgets. Our results demonstrate that offering a flexible budget to the classifier can significantly enhance performance compared to merely increasing a fixed budget. Notably, our strategy outperformed state-of-the-art active learning strategies, all while maintaining a comparable or lower number of labeled instances. Experiments are available at https://github.com/gabrieljaguiar/DBAL.

Active learning with fairness-aware clustering for fair classification considering multiple sensitive attributes

Fairness concerns have recently been gaining increasing attention in machine learning (ML) research and applications. ML models typically require massive data, which can be costly and challenging for collecting and labeling. To address these concerns, active learning has been proposed as a promising approach to build accurate and fair ML models by interactively querying an oracle within a labeling budget. However, current research on fair active learning only focuses on single sensitive attribute, where the model often show unfairness on subgroups defined by the intersections of different protected attribute values. In this article, a novel active learning approach with a fairness-aware clustering technique is proposed for fair classification that can simultaneously ensure the equality of fairness metrics across multiple sensitive attributes. The fair clustering method enables to control the trade-off levels between the clustering and fairness objectives with multiple sensitive attributes, which mitigates bias and protects the unprivileged classes by balancing their representativeness in all clusters. A tight upper bound is theoretically derived for our fairness objective based on its concave-convex decomposition, which can be jointly optimized with the clustering objective through Majorize-Minimize technique and enable parallel computing. The active learning sampler aims to select highly informative and representative samples from fair clusters to enhance the model classification accuracy and fairness. The effectiveness of our proposed approach is demonstrated by comparing with state-of-the-art methods on real-world datasets. The experimental results are accordant with theoretic analysis, and show that (i) our approach significantly improves the fairness while maintaining model classification accuracy; (ii) our approach outperforms state-of-the-art fair active learning methods across fairness metrics on both single and multiple sensitive attributes for fair classification tasks.

Active broad learning with multi-objective evolution for data stream classification

In a streaming environment, the characteristics and labels of instances may change over time, forming concept drifts. Previous studies on data stream learning generally assume that the true label of each instance is available or easily obtained, which is impractical in many real-world applications due to expensive time and labor costs for labeling. To address the issue, an active broad learning based on multi-objective evolutionary optimization is presented to classify non-stationary data stream. The instance newly arrived at each time step is stored to a chunk in turn. Once the chunk is full, its data distribution is compared with previous ones by fast local drift detection to seek potential concept drift. Taking diversity of instances and their relevance to new concept into account, multi-objective evolutionary algorithm is introduced to find the most valuable candidate instances. Among them, representative ones are randomly selected to query their ground-truth labels, and then update broad learning model for drift adaption. More especially, the number of representative is determined by the stability of adjacent historical chunks. Experimental results for 7 synthetic and 5 real-world datasets show that the proposed method outperforms five state-of-the-art ones on classification accuracy and labeling cost due to drift regions accurately identified and the labeling budget adaptively adjusted.

Reducing Spatial Labeling Redundancy for Active Semi-Supervised Crowd Counting.

Labeling is onerous for crowd counting as it should annotate each individual in crowd images. Recently, several methods have been proposed for semi-supervised crowd counting to reduce the labeling efforts. Given a limited labeling budget, they typically select a few crowd images and densely label all individuals in each of them. Despite the promising results, we argue the None-or-All labeling strategy is suboptimal as the densely labeled individuals in each crowd image usually appear similar while the massive unlabeled crowd images may contain entirely diverse individuals. To this end, we propose to break the labeling chain of previous methods and make the first attempt to reduce spatial labeling redundancy for semi-supervised crowd counting. First, instead of annotating all the regions in each crowd image, we propose to annotate the representative ones only. We analyze the region representativeness from both vertical and horizontal directions of initially estimated density maps, and formulate them as cluster centers of Gaussian Mixture Models. Additionally, to leverage the rich unlabeled regions, we exploit the similarities among individuals in each crowd image to directly supervise the unlabeled regions via feature propagation instead of the error-prone label propagation employed in the previous methods. In this way, we can transfer the original spatial labeling redundancy caused by individual similarities to effective supervision signals on the unlabeled regions. Extensive experiments on the widely-used benchmarks demonstrate that our method can outperform previous best approaches by a large margin.

The theoretical and practical foundations for the implementation the "green" budgeting principles

The subject of the study is the conceptual foundations of the implementation of the principles of "green" budgeting. The purpose of the article is to study the peculiarities of the formation of "green" (ecological) budgeting in the developed countries of the world and countries with a transformational economy, to provide suggestions on the possibilities, advantages and risks of introducing the principles of "green" budgeting in the budget process in Ukraine. Method or methodology of research. In the article, a set of methods of scientific knowledge was used to solve the tasks, which made it possible to realize the conceptual unity of scientific research. System method, method of generalization, methods of analysis and synthesis, comparison, scientific abstraction are applied. Results of the article.. The article discusses approaches to defining the concept of "green" budgeting, environmental labeling of budget expenditures. The factors that determine the success of achieving the Sustainable Development Goals and the role of "green" budgeting in this process are systematized. The need to revise state policy measures in the sphere of solving issues of social, economic and environmental direction, increasing the level of people's well-being and reducing social inequality in the form of leveling environmental risks and ensuring conditions for the harmonious development of ecosystems as a component of achieving the Sustainable Development Goals by 2023 has been identified. The peculiarities of the introduction of "green" budgeting in the countries of the European Union and the member countries of the Organization for Economic Cooperation and Development were studied, and the level of "greening" of the budget system and budget policy was determined. Field of application of results. The results of the study can be used in the process of reforming the public finance management system in terms of improving the budget process through the gradual introduction of environmental labeling of budget expenditures. Conclusions. Green budgeting can have a broader scope related to a range of environmental goals and starts by labeling budget items related to a narrower range of environmental goals, then these methods provide a useful basis for creating markers related to other environmental goals . The primary tasks of the introduction of environmental labeling are to understand the purpose of such budgetary policy instruments, in particular, to understand the area of ​​problems that the government is trying to solve with such an innovation, to assess whether green labeling is a valid solution to environmental problems, as well as the stage of the budget process at which the introduction of green labeling should begin.

A novel anomaly detection approach based on ensemble semi-supervised active learning (ADESSA)

As an industrial infrastructure, the safety and reliability of the Cyber-Physical System requires the effective anomaly detection. However, the existing detection methods have bottleneck in the face of insufficient training datasets. This work proposed and a novel anomaly detection approach based on ensemble semi-supervised active learning, which can effectively detect anomalous traffic when there is few labeled samples and the dataset is unbalanced. Specifically, this work proposed balanced sampling strategy, which combines the margin sampling and the democratic co-learning techniques, to construct a balanced training set that consists of manually labeled high-information samples and automatically labeled high-confidence samples, to effectively train the detection model on a limited budget. We also found adding correctly labeled high-confidence samples into training set improves the performance of detection model when the training samples are few and the label budget is limited. This work achieves a good balance between the effectiveness of model training and the cost of sample querying when the traffic data in CPS is rare labeled and imbalanced. In addition, we designed five pairs of experiments with NSL-KDD and SWaT dataset, and the results demonstrate the capability and advancement of proposed approach.

An innovative deep active learning approach for improving unlabeled audio classification by selectively querying informative instance

Audio classification tasks like speech recognition and acoustic scene analysis require substantial labeled data, which is expensive. This work explores active learning to reduce annotation costs for a sound classification problem with rare target classes where existing datasets are insufficient. A deep convolutional recurrent neural network extracts spectro-temporal features and makes predictions. An uncertainty sampling strategy queries the most uncertain samples for manual labeling by experts and non-experts. A new alternating confidence sampling strategy and two other certainty-based strategies are proposed and evaluated. Experiments show significantly higher accuracy than passive learning baselines with the same labeling budget. Active learning generalizes well in a qualitative analysis of 20,000 unlabeled recordings. Overall, active learning with a novel sampling strategy minimizes the need for expensive labeled data in audio classification, successfully leveraging unlabeled data to improve accuracy with minimal supervision.

Cloud Detection in Optical Remote Sensing Images With Deep Semi-Supervised and Active Learning

Clouds hinder the surface observation by optical remote sensing sensors. It is of great significance to detect clouds and non-clouds in remote sensing images. Compared with traditional cloud detection methods, deep learning methods usually achieve promising detection results. Moreover, large-scale, high-quality labeled datasets can effectively improve the accuracy and generalization of deep learning models. However, this incurs a great deal of label effort and cost. In this paper, we proposes a cloud detection method based on deep semi-supervised learning(SSL) and active learning(AL) in optical remote sensing images named SSAL-CD. SSL part of SSAL-CD is implemented by the cross-validation between two-way neural networks, which can effectively train in the case of insufficient labeled data and reduce the cost of labeling. AL part proposes a suitable query strategy based on the uncertainty, degree of divergence and diversity information provided by the training model. The query strategy can screen out high-value samples for SSL, which further improves the detection effect. Repeat the process of SSL and AL until the accuracy meets requirements or the labeling budget is exhausted. Extensive experimental results in Landsat-8 Biome dataset demonstrate that SSAL-CD can achieve state-of-the-art segmentation performance with a small number of labels.

Automatic SDG budget tagging: Building public financial management capacity through natural language processing

Abstract The “budgeting for SDGs”–B4SDGs–paradigm seeks to coordinate the budgeting process of the fiscal cycle with the sustainable development goals (SDGs) set by the United Nations. Integrating the goals into public financial management systems is crucial for an effective alignment of national development priorities with the objectives set in the 2030 Agenda. Within the dynamic process defined in the B4SDGs framework, the step of SDG budget tagging represents a precondition for subsequent budget diagnostics. However, developing a national SDG taxonomy requires substantial investment in terms of time, human, and administrative resources. Such costs are exacerbated in least-developed countries, which are often characterized by a constrained institutional capacity. The automation of SDG budget tagging could represent a cost-effective solution. We use well-established text analysis and machine learning techniques to explore the scope and scalability of automatic labeling budget programs within the B4SDGs framework. The results show that, while our classifiers can achieve great accuracy, they face limitations when trained with data that is not representative of the institutional setting considered. These findings imply that a national government trying to integrate SDGs into its planning and budgeting practices cannot just rely solely on artificial intelligence (AI) tools and off-the-shelf coding schemes. Our results are relevant to academics and the broader policymaker community, contributing to the debate around the strengths and weaknesses of adopting computer algorithms to assist decision-making processes.

Active Sampling for Text Classification with Subinstance Level Queries

Active learning algorithms are effective in identifying the salient and exemplar samples from large amounts of unlabeled data. This tremendously reduces the human annotation effort in inducing a machine learning model as only a few samples, which are identified by the algorithm, need to be labeled manually. In problem domains like text mining and video classification, human oracles peruse the data instances incrementally to derive an opinion about their class labels (such as reading a movie review progressively to assess its sentiment). In such applications, it is not necessary for the human oracles to review an unlabeled sample end-to-end in order to provide a label; it may be more efficient to identify an optimal subinstance size (percentage of the sample from the start) for each unlabeled sample, and request the human annotator to label the sample by analyzing only the subinstance, instead of the whole data sample. In this paper, we propose a novel framework to address this challenging problem, in an effort to further reduce the labeling burden on the human oracles and utilize the available labeling budget more efficiently. We pose the sample and subinstance size selection as a constrained optimization problem and derive a linear programming relaxation to select a batch of exemplar samples, together with the optimal subinstance size of each, which can potentially augment maximal information to the underlying classification model. Our extensive empirical studies on six challenging datasets from the text mining domain corroborate the practical usefulness of our framework over competing baselines.

RareGAN: Generating Samples for Rare Classes

We study the problem of learning generative adversarial networks (GANs) for a rare class of an unlabeled dataset subject to a labeling budget. This problem is motivated from practical applications in domains including security (e.g., synthesizing packets for DNS amplification attacks), systems and networking (e.g., synthesizing workloads that trigger high resource usage), and machine learning (e.g., generating images from a rare class). Existing approaches are unsuitable, either requiring fully-labeled datasets or sacrificing the fidelity of the rare class for that of the common classes. We propose RareGAN, a novel synthesis of three key ideas: (1) extending conditional GANs to use labelled and unlabelled data for better generalization; (2) an active learning approach that requests the most useful labels; and (3) a weighted loss function to favor learning the rare class. We show that RareGAN achieves a better fidelity-diversity tradeoff on the rare class than prior work across different applications, budgets, rare class fractions, GAN losses, and architectures.

Fair active learning

Machine learning (ML) is increasingly being used in high-stakes applications impacting society. Therefore, it is of critical importance that ML models do not propagate discrimination. Collecting accurate labeled data in societal applications is challenging and costly. Active learning is a promising approach to build an accurate classifier by interactively querying an oracle within a labeling budget. We introduce the fair active learning framework to carefully select data points to be labeled so as to balance model accuracy and fairness. To incorporate the notion of fairness in the active learning sampling core, it is required to measure the fairness of the model after adding each unlabeled sample. Since their labels are unknown in advance, we propose an expected fairness metric to probabilistically measure the impact of each sample if added for each possible class label. Next, we propose multiple optimizations to balance the trade-off between accuracy and fairness. Our first optimization linearly aggregate the expected fairness with entropy using a control parameter. To avoid erroneous estimation of the expected fairness, we propose a nested approach to maintain the accuracy of the model, limiting the search space to the top bucket of sample points with large entropy. Finally, to ensure the unfairness reduction of the model after labeling, we propose to replicate the points that truly reduce the unfairness after labeling. We demonstrate the effectiveness and efficiency of our proposed algorithms over widely used benchmark datasets using demographic parity and equalized odds notions of fairness.

Active Classification With Uncertainty Comparison Queries.

Noisy pairwise comparison feedback has been incorporated to improve the overall query complexity of interactively learning binary classifiers. The positivity comparison oracle is extensively used to provide feedback on which is more likely to be positive in a pair of data points. Because it is impossible to determine accurate labels using this oracle alone without knowing the classification threshold, existing methods still rely on the traditional explicit labeling oracle, which explicitly answers the label given a data point. The current method conducts sorting on all data points and uses explicit labeling oracle to find the classification threshold. However, it has two drawbacks: (1) it needs unnecessary sorting for label inference and (2) it naively adapts quick sort to noisy feedback. In order to avoid these inefficiencies and acquire information of the classification threshold at the same time, we propose a new pairwise comparison oracle concerning uncertainties. This oracle answers which one has higher uncertainty given a pair of data points. We then propose an efficient adaptive labeling algorithm to take advantage of the proposed oracle. In addition, we address the situation where the labeling budget is insufficient compared to the data set size. Furthermore, we confirm the feasibility of the proposed oracle and the performance of the proposed algorithm theoretically and empirically.

Semi-Active Convolutional Neural Networks for Hyperspectral Image Classification

Owing to the powerful data representation ability of deep learning (DL) techniques, tremendous progress has been recently made in hyperspectral image (HSI) classification. Convolutional neural network (CNN), as a main part of the DL family, has been proven to be considerably effective to extract spatial-spectral features for HSIs. Nevertheless, its classification performance, to a great extent, depends on the quality and quantity of samples in the network training process. To select those samples, either labeled or unlabeled, that can be used to enhance the generalization ability of CNNs and further improve the classification accuracy, we propose an iterative semi-supervised CNNs framework by means of active learning and superpixel segmentation techniques, dubbed as semi-active CNNs (SA-CNNs), for HSI classification. More specifically, we start to pre-train a CNNs-based model on a small-scale unbiased labeled set and infer unlabeled data using the trained model, i.e., generating pseudo-labels. Then, the reliable samples, which consist of two parts: high label-homogeneity and most informativeness, are actively selected from superpixel segments. These selected labeled and unlabeled samples with their labels and pseudo-labels are re-fed into the next-round network training. Moreover, three different schedules, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">log</i> -, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">exp</i> -, and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">linear</i> -schedules, are progressively adopted to fully explore their potentials in sample selection, until a labeling budget is finally reached. Extensive experiments are conducted on three benchmark HSI datasets, demonstrating substantial performance improvements of the proposed SA-CNNs over other similar competitors.

Semi-supervised Active Salient Object Detection

In this paper, we propose a novel semi-supervised active salient object detection (SOD) method that actively acquires a small subset of the most discriminative and representative samples for labeling. Two main contributions have been made to prevent the method from being overwhelmed by labeling similar distributed samples. First, we design a saliency encoder-decoder with adversarial discriminator to generate a confidence map, representing the network uncertainty on the current prediction. Then, we select the least confident (discriminative) samples from the unlabeled pool to form the “candidate labeled pool”. Second, we train a Variational Auto-Encoder (VAE) to select and add the most representative data from the “candidate labeled pool” into the labeled pool by comparing their corresponding features in the latent space. Within our framework, these two networks are optimized conditioned on the states of each other progressively. Experimental results on six benchmarking SOD datasets demonstrate that our annotation-efficient learning based salient object detection method, reaching to 14% labeling budget, can be on par with the state-of-the-art fully-supervised deep SOD models. The source code is publicly available via our project page: https://github.com/JingZhang617/Semi-sup-active-self-sup-Learning.