Uncertainty-based Active Learning Research Articles

A study of the role of data and model uncertainty in active learning

Uncertainty-based active learning strategies have demonstrated significant superiority in small data research of materials domain. This study explores the effects of model uncertainty and data uncertainty separately on the performance of active learning strategies, specifically focusing on the number of iterations required to identify the optimal samples. For model uncertainty, three kinds of acquisition functions are compared, including predicted value strategy (PV), ranking of predicted value strategy (PR) and expected improvement strategy (EI). Among these, the active learning model utilizing PR requires the fewest average iterations (1.75). For data uncertainty, we evaluate the iterations of active learning by Gaussian process models that incorporate the uncertainty of the observations and noise samples that takes account into the uncertainty of the input features respectively. The results indicate that the active learning iterations of the three strategies converge to similar at the optimal weighting when the uncertainty of the observations is considered in the model (EI for 1.75, PV for 1.21 and PR for 1.18). In contrast, incorporating noise samples into the augmented dataset after the original samples would severely deteriorate the efficiency of active learning recommendations. Our findings aim to offer guidance for exploring more favorable acquisition functions and methods for active learning strategies.

Performance of uncertainty-based active learning for efficient approximation of black-box functions in materials science.

Obtaining a fine approximation of a black-box function is important for understanding and evaluating innovative materials. Active learning aims to improve the approximation of black-box functions with fewer training data. In this study, we investigate whether active learning based on uncertainty sampling enables the efficient approximation of black-box functions in regression tasks using various material databases. In cases where the inputs are provided uniformly and defined in a relatively low-dimensional space, the liquidus surfaces of the ternary systems are the focus. The results show that uncertainty-based active learning can produce a better black-box function with higher prediction accuracy than that by random sampling. Furthermore, in cases in which the inputs are distributed discretely and unbalanced in a high-dimensional feature space, datasets extracted from materials databases for inorganic materials, small molecules, and polymers are addressed, and uncertainty-based active learning is occasionally inefficient. Based on the dependency on the material descriptors, active learning tends to produce a better black-box functions than random sampling when the dimensions of the descriptor are small. The results indicate that active learning is occasionally inefficient in obtaining a better black-box function in materials science.

Exploring the potential of machine learning to understand the occurrence and health risks of haloacetic acids in a drinking water distribution system

Determining the occurrence of disinfection byproducts (DBPs) in drinking water distribution system (DWDS) remains challenging. Predicting DBPs using readily available water quality parameters can help to understand DBPs associated risks and capture the complex interrelationships between water quality and DBP occurrence. In this study, we collected drinking water samples from a distribution network throughout a year and measured the related water quality parameters (WQPs) and haloacetic acids (HAAs). 12 machine learning (ML) algorithms were evaluated. Random Forest (RF) achieved the best performance (i.e., R2 of 0.78 and RMSE of 7.74) for predicting HAAs concentration. Instead of using cytotoxicity or genotoxicity separately as the surrogate for evaluating toxicity associated with HAAs, we created a health risk index (HRI) that was calculated as the sum of cytotoxicity and genotoxicity of HAAs following the widely used Tic-Tox approach. Similarly, ML models were developed to predict the HRI, and RF model was found to perform the best, obtaining R2 of 0.69 and RMSE of 0.38. To further explore advanced ML approaches, we developed 3 models using uncertainty-based active learning. Our findings revealed that Categorical Boosting Regression (CAT) model developed through active learning substantially outperformed other models, achieving R2 of 0.87 and 0.82 for predicting concentration and the HRI, respectively. Feature importance analysis with the CAT model revealed that temperature, ions (e.g., chloride and nitrate), and DOC concentration in the distribution network had a significant impact on the occurrence of HAAs. Meanwhile, chloride ion, pH, ORP, and free chlorine were found as the most important features for HRI prediction. This study demonstrates that ML has the potential in the prediction of HAA occurrence and toxicity. By identifying key WQPs impacting HAA occurrence and toxicity, this research offers valuable insights for targeted DBP mitigation strategies.

Low-resource entity resolution with domain generalization and active learning

Entity Resolution (ER), a fundamental task in data cleaning and integration, is critical in various fields such as healthcare, e-commerce, and social networks. Traditional ER methods are constrained by the need for substantial labeled samples and the challenge of generalization to unseen domains. To address the low-resource challenge in ER, a novel two-phase framework is proposed. Initially, we introduce the Domain Generalization Entity Resolution (DGER) framework, combining domain adversarial learning and simulated target learning to improve the generalization performance on unseen target domains. Subsequently, to further adapt to the target dataset, we present a novel active learning approach called Domain-Aware Uncertainty Active Learning (DUAL), for fine-tuning the DGER model with minimal annotation cost. DUAL manually annotates target domain samples that are highly uncertain and exhibit high divergence from the source, while assigning pseudo-labels to high-confidence samples. Experimental results on multiple real-world datasets demonstrate that our framework outperforms traditional ER methods in generalizing to unseen domains. Specifically, our DGER method outperforms the best-performing ER baseline in each task, achieving an average F1 score improvement of 9.02% across eight different test tasks. Moreover, within a limited annotation budget during the active learning phase, our DUAL fine-tuning strategy for the ER model outperforms uncertainty-based active learning techniques.

Innovative Strategies for Early Autism Diagnosis: Active Learning and Domain Adaptation Optimization.

The early diagnosis of autism spectrum disorder (ASD) encounters challenges stemming from domain variations in facial image datasets. This study investigates the potential of active learning, particularly uncertainty-based sampling, for domain adaptation in early ASD diagnosis. Our focus is on improving model performance across diverse data sources. Utilizing the Kaggle ASD and YTUIA datasets, we meticulously analyze domain variations and assess transfer learning and active learning methodologies. Two state-of-the-art convolutional neural networks, Xception and ResNet50V2, pretrained on distinct datasets, demonstrate noteworthy accuracies of 95% on Kaggle ASD and 96% on YTUIA, respectively. However, combining datasets results in a modest decline in average accuracy, underscoring the necessity for effective domain adaptation techniques. We employ uncertainty-based active learning to address this, which significantly mitigates the accuracy drop. Xception and ResNet50V2 achieve 80% and 79% accuracy when pretrained on Kaggle ASD and applying active learning on YTUIA, respectively. Our findings highlight the efficacy of uncertainty-based active learning for domain adaptation, showcasing its potential to enhance accuracy and reduce annotation needs in early ASD diagnosis. This study contributes to the growing body of literature on ASD diagnosis methodologies. Future research should delve deeper into refining active learning strategies, ultimately paving the way for more robust and efficient ASD detection tools across diverse datasets.

Uncertainty-based Active Learning by Bayesian U-Net for Multi-label Cone-beam CT Segmentation

IntroductionTraining of Artificial Intelligence (AI) for biomedical image analysis depends on large annotated datasets. This study assessed the efficacy of Active Learning (AL) strategies training AI models for accurate multilabel segmentation and detection of periapical lesions in cone-beam CTs (CBCTs) using a limited dataset. MethodsLimited field-of-view CBCT volumes (n = 20) were segmented by clinicians (clinician segmentation [CS]) and Bayesian U-Net-based AL strategies. Two AL functions, Bayesian Active Learning by Disagreement [BALD] and Max_Entropy [ME], were used for multilabel segmentation (“Lesion”-“Tooth Structure”-“Bone”-“Restorative Materials”-“Background”), and compared to a non-AL benchmark Bayesian U-Net function. The training-to-testing set ratio was 4:1. Comparisons between the AL and Bayesian U-Net functions versus CS were made by evaluating the segmentation accuracy with the Dice indices and lesion detection accuracy. The Kruskal-Wallis test was used to assess statistically significant differences. ResultsThe final training set contained 26 images. After 8 AL iterations, lesion detection sensitivity was 84.0% for BALD, 76.0% for ME, and 32.0% for Bayesian U-Net, which was significantly different (P < .0001; H = 16.989). The mean Dice index for all labels was 0.680 ± 0.155 for Bayesian U-Net and 0.703 ± 0.166 for ME after eight AL iterations, compared to 0.601 ± 0.267 for Bayesian U-Net over the mean of all iterations. The Dice index for “Lesion” was 0.504 for BALD and 0.501 for ME after 8 AL iterations, and at a maximum 0.288 for Bayesian U-Net. ConclusionsBoth AL strategies based on uncertainty quantification from Bayesian U-Net BALD, and ME, provided improved segmentation and lesion detection accuracy for CBCTs. AL may contribute to reducing extensive labeling needs for training AI algorithms for biomedical image analysis in dentistry.

Exploiting redundancy in large materials datasets for efficient machine learning with less data

Extensive efforts to gather materials data have largely overlooked potential data redundancy. In this study, we present evidence of a significant degree of redundancy across multiple large datasets for various material properties, by revealing that up to 95% of data can be safely removed from machine learning training with little impact on in-distribution prediction performance. The redundant data is related to over-represented material types and does not mitigate the severe performance degradation on out-of-distribution samples. In addition, we show that uncertainty-based active learning algorithms can construct much smaller but equally informative datasets. We discuss the effectiveness of informative data in improving prediction performance and robustness and provide insights into efficient data acquisition and machine learning training. This work challenges the “bigger is better” mentality and calls for attention to the information richness of materials data rather than a narrow emphasis on data volume.

Evaluating uncertainty-based active learning for accelerating the generalization of molecular property prediction

Deep learning models have proven to be a powerful tool for the prediction of molecular properties for applications including drug design and the development of energy storage materials. However, in order to learn accurate and robust structure–property mappings, these models require large amounts of data which can be a challenge to collect given the time and resource-intensive nature of experimental material characterization efforts. Additionally, such models fail to generalize to new types of molecular structures that were not included in the model training data. The acceleration of material development through uncertainty-guided experimental design has the promise to significantly reduce the data requirements and enable faster generalization to new types of materials. To evaluate the potential of such approaches for electrolyte design applications, we perform comprehensive evaluation of existing uncertainty quantification methods on the prediction of two relevant molecular properties - aqueous solubility and redox potential. We develop novel evaluation methods to probe the utility of the uncertainty estimates for both in-domain and out-of-domain data sets. Finally, we leverage selected uncertainty estimation methods for active learning to evaluate their capacity to support experimental design.

Active Learning-Driven Siamese Network for Hyperspectral Image Classification

Hyperspectral image (HSI) classification has recently been successfully explored by using deep learning (DL) methods. However, DL models rely heavily on a large number of labeled samples, which are laborious to obtain. Therefore, finding a way to efficiently embed DL models in limited labeled samples is a hot topic in the field of HSI classification. In this paper, an active learning-based siamese network (ALSN) is proposed to solve the limited labeled samples problem in HSI classification. First, we designed a dual learning-based siamese network (DLSN), which consists of a contrastive learning module and a classification module. Secondly, in view of the problem that active learning is difficult to effectively sample under the extremely limited labeling cost, we proposed an adversarial uncertainty-based active learning (AUAL) method to query valuable samples, and to promote DLSN to learn a more complete feature distribution by fine-tuning. Finally, an active learning architecture, based on inter-class uncertainty (ICUAL), is proposed to construct a lightweight sample pair training set, fully extracting the inter-class information of sample pairs and improving classification accuracy. Experiments on three generic HSI datasets strongly demonstrated the effectiveness of ALSN for HSI classification, with performance improvements over other related DL methods.

Prediction of serious eye damage or eye irritation potential of compounds via consensus labelling models and active learning models based on uncertainty strategies

Serious eye damage and eye irritation have been authenticated to be significant human health issues in various fields such as ophthalmic pharmaceuticals. Due to the shortcomings of traditional animal testing methods, in silico methods have advanced to study eye toxicity. The models for predicting serious eye damage and eye irritation potential of compounds were developed using 2299 and 5214 compounds, respectively. The 40 global single models and 40 local models were developed by combining 5 molecular description methods and 4 machine learning methods. The 40 active learning models were developed by adopting uncertainty-based active learning strategies and taking local models as initial models. The 110 global consensus models based on 40 global single models were developed using a consensus strategy. Active learning models and global consensus models performed high prediction accuracy. The test accuracy of the best serious eye damage model and eye irritation model reached 0.972 and 0.959, respectively. The applicability domains for all models were calculated to verify the rationality of prediction effect. In addition, 8 structural alerts probably causing serious eye damage or eye irritation were sought out. The prediction models and structural alerts contributed to providing hazard identification and assessing chemical safety.

Deep active learning with Weighting filter for object detection

Active learning has been demonstrated to be effective in reducing labeling costs by selecting the most valuable data from the unlabeled pool. However, the training data of the first epoch in almost all active learning methods is randomly selected, which will cause an instability learning process. Additionally, current active learning, especially uncertainty-based active learning methods, is prone to the problem of data bias because model learning inevitably prefers partial data. For the above issues, we propose Weighting filter (W-filter) tailored for object detection in this paper, which is an image filtering algorithm that can calculate the contribution of a single image to the neural network training as well as remove similar ones in the entire selected data to optimize the sampling results. We first use W-filter to select the training data of the first epoch, which can guarantee better performance and a more stable learning process. Then, we propose to resample the uncertain data from the perspective of the frequency domain to alleviate the problem of data bias. Finally, we redesign several classical uncertainty methods specifically for classification to make them more suitable for the task of object detection. We do rigorous experiments on standard benchmark datasets to validate our work. Several classical detectors such as Faster R-CNN, SSD, R-FCN, CenterNet, EfficientDet, and effective networks including ResNet, DarkNet, MobileNet are used in experiments, which shows our framework is detector-agnostic and network-agnostic and thus can meet any detection scenario.

Active learning for new-fault class sample recovery in electrical submersible pump fault diagnosis

Applying effective supervised learning methods is difficult when dealing with complex industrial equipment such as Electrical Submersible Pump as the labeling work is costly and requires the dedication of an expert in the field. Moreover, the appearance of a new, unknown fault may require more time from the expert, not only for labeling the data but also for an arduous search for samples of this new type of fault. The solution developed in this paper uses uncertainty-based active learning for aiding experts through intelligent fault diagnosis and the search for potential samples of a new type of fault. The proposed approach was tested with a classical classification algorithm (random forest) applied to features extracted from the vibration signals. Two different feature extraction methods were investigated and contrasted: one using features defined by the expert and another using automatic feature extraction by a deep learning procedure. The algorithms were trained and tested on a dataset composed of 5616 vibration signals labeled in the frequency domain. The results show enough evidence to claim, at a significance value of 5%, that the active learning procedure proposed in this paper outperforms active learning based on random search. A new acquisition strategy for active learning is proposed to aid experts to find potential samples of a new fault. The recovered samples of the new fault aid improving performance of feature extractors based on deep learning since this type of method is highly affected by imbalanced data.

Enlarging Applicability Domain of Quantitative Structure–Activity Relationship Models through Uncertainty-Based Active Learning

Enlarging Applicability Domain of Quantitative Structure–Activity Relationship Models through Uncertainty-Based Active Learning

LTP: A New Active Learning Strategy for CRF-Based Named Entity Recognition

In recent years, deep learning has achieved great success in many natural language processing tasks, including named entity recognition. The shortcoming is that a large quantity of manually annotated data is usually required. Previous studies have demonstrated that active learning can considerably reduce the cost of data annotation, but there is still plenty of room for improvement. In real applications, we found that existing uncertainty-based active learning strategies have two shortcomings. First, these strategies prefer to choose long sequences explicitly or implicitly, which increases the annotation burden of annotators. Second, some strategies need to revise and modify the model to generate additional information for sample selection, which increases the workload of the developer and increases the training/prediction time of the model. In this paper, we first examine traditional active learning strategies in specific cases of Word2Vec-BiLSTM-CRF and Bert-CRF that have been widely used in named entity recognition on several typical datasets. Then, we propose an uncertainty-based active learning strategy called the lowest token probability (LTP), which combines the input and output of conditional random field (CRF) to select informative instances. LTP is a simple and powerful strategy that does not favor long sequences and does not need to revise the model. We test LTP on multiple real-world datasets, the experiment results show that compared with existing state-of-the-art selection strategies, LTP can reduce about 20% annotation tokens while maintaining competitive performance on both sentence-level accuracy and entity-level F1-score. Additionally, LTP significantly outperformed all other strategies in selecting valid samples, which dramatically reduced the invalid annotation times of the labelers.

How Useful Is Image-Based Active Learning for Plant Organ Segmentation?

Training deep learning models typically requires a huge amount of labeled data which is expensive to acquire, especially in dense prediction tasks such as semantic segmentation. Moreover, plant phenotyping datasets pose additional challenges of heavy occlusion and varied lighting conditions which makes annotations more time-consuming to obtain. Active learning helps in reducing the annotation cost by selecting samples for labeling which are most informative to the model, thus improving model performance with fewer annotations. Active learning for semantic segmentation has been well studied on datasets such as PASCAL VOC and Cityscapes. However, its effectiveness on plant datasets has not received much importance. To bridge this gap, we empirically study and benchmark the effectiveness of four uncertainty-based active learning strategies on three natural plant organ segmentation datasets. We also study their behaviour in response to variations in training configurations in terms of augmentations used, the scale of training images, active learning batch sizes, and train-validation set splits.

Geometric Analysis of Uncertainty Sampling for Dense Neural Network Layer.

For model adaptation of fully connected neural network layers, we provide an information geometric and sample behavioral active learning uncertainty sampling objective analysis. We identify conditions under which several uncertainty-based methods have the same performance and show that such conditions are more likely to appear in the early stages of learning. We define riskier samples for adaptation, and demonstrate that, as the set of labeled samples increases, margin-based sampling outperforms other uncertainty sampling methods by preferentially selecting these risky samples. We support our derivations and illustrations with experiments using Meta-Dataset, a benchmark for few-shot learning. We compare uncertainty-based active learning objectives using features produced by SimpleCNAPS (a state-of-the-art few-shot classifier) as input for a fully-connected adaptation layer. Our results indicate that margin-based uncertainty sampling achieves similar performance as other uncertainty based sampling methods with fewer labelled samples as discussed in the novel geometric analysis.

Active Learning for Node Classification: An Evaluation.

Current breakthroughs in the field of machine learning are fueled by the deployment of deep neural network models. Deep neural networks models are notorious for their dependence on large amounts of labeled data for training them. Active learning is being used as a solution to train classification models with less labeled instances by selecting only the most informative instances for labeling. This is especially important when the labeled data are scarce or the labeling process is expensive. In this paper, we study the application of active learning on attributed graphs. In this setting, the data instances are represented as nodes of an attributed graph. Graph neural networks achieve the current state-of-the-art classification performance on attributed graphs. The performance of graph neural networks relies on the careful tuning of their hyperparameters, usually performed using a validation set, an additional set of labeled instances. In label scarce problems, it is realistic to use all labeled instances for training the model. In this setting, we perform a fair comparison of the existing active learning algorithms proposed for graph neural networks as well as other data types such as images and text. With empirical results, we demonstrate that state-of-the-art active learning algorithms designed for other data types do not perform well on graph-structured data. We study the problem within the framework of the exploration-vs.-exploitation trade-off and propose a new count-based exploration term. With empirical evidence on multiple benchmark graphs, we highlight the importance of complementing uncertainty-based active learning models with an exploration term.

Batch Active Learning With Two-Stage Sampling

Due to its effectiveness in training precise model using significant fewer labeled instances, active learning has been widely researched and applied. In order to reduce the time complexity of active learning so that the oracle need not wait for the algorithm to provide instance in labeling, we proposed a new active learning method, which leverages batch sampling and direct boundary annotation with a two-stage sampling strategy. In the first stage sampling, the initial seed, which determines the location of boundary annotation, is selected with reject sampling based on the clustering structure of instances to ensure the initial seeds can approximate the distribution of data and with high diversity. In the second stage sampling, by treating the instance sampling as the selection of representative in a local region and maximizing the rewards that can get from selecting a instance as the new representative, we proposed a novel mechanism to maintain local representativeness and diversity of query instances. Compared with the conventional pool-based active learning method, our proposed method does not need to train the model in each iteration, which reduces the amount of calculation and time consumption. The experimental results in three public datasets show that the proposed method has comparable performance with the uncertainty-based active learning methods, which proves that the sampling mechanism in our method is effective. It performs well without retraining the model in each iteration and does not rely on the precision of the model.

Uncertainty-Based Active Learning via Sparse Modeling for Image Classification.

Uncertainty sampling-based active learning has been well studied for selecting informative samples to improve the performance of a classifier. In batch-mode active learning, a batch of samples are selected for a query at the same time. The samples with top uncertainty are encouraged to be selected. However, this selection strategy ignores the relations among the samples, because the selected samples may have much redundant information with each other. This paper addresses this problem by proposing a novel method that combines uncertainty, diversity, and density via sparse modeling in the sample selection. We use sparse linear combination to represent the uncertainty of unlabeled pool data with Gaussian kernels, in which the diversity and density are well incorporated. The selective sampling method is proposed before optimization to reduce the representation error. To deal with ${l}_{0}$ norm constraint in the sparse problem, two approximated approaches are adopted for efficient optimization. Four image classification data sets are used for evaluation. Extensive experiments related to batch size, feature space, seed size, significant analysis, data transform, and time efficiency demonstrate the advantages of the proposed method.

AL-ELM: One uncertainty-based active learning algorithm using extreme learning machine

It is well known that in supervised learning, active learning could effectively decrease the complexity of training instances without obvious loss of the classification performance. Generally, active learning is applied in the scenario that lots of instances are easy to be acquired, but labeling them is expensive and/or time-consuming. In this study, we try to implement active learning by using extreme learning machine (ELM) classifier based on three reasons as follows: (1) ELM has light computational costs, (2) ELM has strong generalization ability which is even comparable with support vector machine (SVM) and (3) ELM could be directly applied on both binary-class and multiclass problems. Specifically, an active learning algorithm based on ELM classifier named AL-ELM is proposed in this paper. During active learning, AL-ELM estimates the uncertainty of each unlabeled instance by creating a mapping relation between the actual outputs of the instance in ELM and the approximated membership probability of the same instance. In other words, ELM is converted as the equivalent Bayes classifier. On each iteration, those most uncertain instances are extracted and labeled to promote the quality of classification model. The learning procedure stops until it satisfies a pre-designed criterion. Experimental results on 20 benchmark data sets show that AL-ELM is better than or at least comparable to several state-of-the-art uncertainty-based active learning algorithms. Also, in contrast with several other algorithms, AL-ELM could effectively decrease the running time of learning procedure.