Annotation Cost Research Articles

Multiscale progressive text prompt network for medical image segmentation

The accurate segmentation of medical images is a crucial step in obtaining reliable morphological statistics. However, training a deep neural network for this task requires a large amount of labeled data to ensure high-accuracy results. To address this issue, we propose using progressive text prompts as prior knowledge to guide the segmentation process. Our model consists of two stages. In the first stage, we perform contrastive learning on natural images to pretrain a powerful prior prompt encoder (PPE). This PPE leverages text prior prompts to generate multimodality features. In the second stage, medical image and text prior prompts are sent into the PPE inherited from the first stage to achieve the downstream medical image segmentation task. A multiscale feature fusion block (MSFF) combines the features from the PPE to produce multiscale multimodality features. These two progressive features not only bridge the semantic gap but also improve prediction accuracy. Finally, an UpAttention block refines the predicted results by merging the image and text features. This design provides a simple and accurate way to leverage multiscale progressive text prior prompts for medical image segmentation. Compared with using only images, our model achieves high-quality results with low data annotation costs. Moreover, our model not only has excellent reliability and validity on medical images but also performs well on natural images. The experimental results on different image datasets demonstrate that our model is effective and robust for image segmentation.

SatelliteCloudGenerator: Controllable Cloud and Shadow Synthesis for Multi-Spectral Optical Satellite Images

Optical satellite images of Earth frequently contain cloud cover and shadows. This requires processing pipelines to recognize the presence, location, and features of the cloud-affected regions. Models that make predictions about the ground behind the clouds face the challenge of lacking ground truth information, i.e., the exact state of Earth’s surface. Currently, the solution to that is to either (i) create pairs from samples acquired at different times or (ii) simulate cloudy data based on a clear acquisition. This work follows the second approach and proposes an open-source simulation tool capable of generating a diverse and unlimited number of high-quality simulated pair data with controllable parameters to adjust cloud appearance, with no annotation cost. The tool is available as open-source. An indication of the quality and utility of the generated clouds is demonstrated by the models for cloud detection and cloud removal trained exclusively on simulated data, which approach the performance of their equivalents trained on real data.

Automatic intracranial abnormality detection and localization in head CT scans by learning from free-text reports

Deep learning has yielded promising results for medical image diagnosis but relies heavily on manual image annotations, which are expensive to acquire. We present Cross-DL, a cross-modality learning framework for intracranial abnormality detection and localization in head computed tomography (CT) scans by learning from free-text imaging reports. Cross-DL has a discretizer that automatically extracts discrete labels of abnormality types and locations from reports, which are utilized to train an image analyzer by a dynamic multi-instance learning approach. Benefiting from the low annotation cost and a consequent large-scale training set of 28,472 CT scans, Cross-DL achieves accurate performance, with an average area under the receiver operating characteristic curve (AUROC) of 0.956 (95% confidence interval: 0.952-0.959) in detecting 4 abnormality types in 17 regions while accurately localizing abnormalities at the voxel level. An intracranial hemorrhage classification experiment on the external dataset CQ500 achieves an AUROC of 0.928 (0.905-0.951). The model can also help review prioritization.

Uncertainty-aware and dynamically-mixed pseudo-labels for semi-supervised defect segmentation

Deep learning-based defect segmentation is one of the important tasks of machine vision in automated inspection. Supervised learning methods have recently achieved remarkable performance on this task. However, the effectiveness of the supervised methods is limited by the scarcity and high cost of pixel-level annotation of training data. Semi-supervised learning methods have been proposed for training deep learning networks using a limited amount of labeled data along with additional unlabeled data for image segmentation. Most of these methods are based on consistency regularization and pseudo labeling, where the predictions on unlabeled samples often come with noise and are unreliable, resulting in poor segmentation performance. To alleviate this problem, we propose uncertainty-aware pseudo labels, which are generated from dynamically mixed predictions of multiple decoders that leverage a shared encoder network. The estimated uncertainty guides the pseudo-label-based supervision and regularizes the training when using the unlabeled samples. In our experiments on four public datasets for defect segmentation, the proposed method outperformed the fully supervised baseline and six state-of-the-art semi-supervised segmentation methods. We also conducted an extensive ablation study to demonstrate the effectiveness of our approach in various settings. The implementation code for this work is available at https://github.com/djene-mengistu/UAPS.

HAL-IA: A Hybrid Active Learning framework using Interactive Annotation for medical image segmentation.

High performance of deep learning models on medical image segmentation greatly relies on large amount of pixel-wise annotated data, yet annotations are costly to collect. How to obtain high accuracy segmentation labels of medical images with limited cost (e.g. time) becomes an urgent problem. Active learning can reduce the annotation cost of image segmentation, but it faces three challenges: the cold start problem, an effective sample selection strategy for segmentation task and the burden of manual annotation. In this work, we propose a Hybrid Active Learning framework using Interactive Annotation (HAL-IA) for medical image segmentation, which reduces the annotation cost both in decreasing the amount of the annotated images and simplifying the annotation process. Specifically, we propose a novel hybrid sample selection strategy to select the most valuable samples for segmentation model performance improvement. This strategy combines pixel entropy, regional consistency and image diversity to ensure that the selected samples have high uncertainty and diversity. In addition, we propose a warm-start initialization strategy to build the initial annotated dataset to avoid the cold-start problem. To simplify the manual annotation process, we propose an interactive annotation module with suggested superpixels to obtain pixel-wise label with several clicks. We validate our proposed framework with extensive segmentation experiments on four medical image datasets. Experimental results showed that the proposed framework achieves high accuracy pixel-wise annotations and models with less labeled data and fewer interactions, outperforming other state-of-the-art methods. Our method can help physicians efficiently obtain accurate medical image segmentation results for clinical analysis and diagnosis.

Dissecting self-supervised learning methods for surgical computer vision.

The field of surgical computer vision has undergone considerable breakthroughs in recent years with the rising popularity of deep neural network-based methods. However, standard fully-supervised approaches for training such models require vast amounts of annotated data, imposing a prohibitively high cost; especially in the clinical domain. Self-Supervised Learning (SSL) methods, which have begun to gain traction in the general computer vision community, represent a potential solution to these annotation costs, allowing to learn useful representations from only unlabeled data. Still, the effectiveness of SSL methods in more complex and impactful domains, such as medicine and surgery, remains limited and unexplored. In this work, we address this critical need by investigating four state-of-the-art SSL methods (MoCo v2, SimCLR, DINO, SwAV) in the context of surgical computer vision. We present an extensive analysis of the performance of these methods on the Cholec80 dataset for two fundamental and popular tasks in surgical context understanding, phase recognition and tool presence detection. We examine their parameterization, then their behavior with respect to training data quantities in semi-supervised settings. Correct transfer of these methods to surgery, as described and conducted in this work, leads to substantial performance gains over generic uses of SSL - up to 7.4% on phase recognition and 20% on tool presence detection - as well as state-of-the-art semi-supervised phase recognition approaches by up to 14%. Further results obtained on a highly diverse selection of surgical datasets exhibit strong generalization properties. The code is available at https://github.com/CAMMA-public/SelfSupSurg.

MTMVC: Semi-supervised 3D hand pose estimation using multi-task and multi-view consistency

The high performance of state-of-the-art deep learning methods for 3D hand pose estimation heavily depends on a large annotated training set. However, it is difficult and time-consuming to obtain the annotations for 3D hand poses. To leverage unannotated images to reduce the annotation cost, we propose a semi-supervised method based on Multi-Task and Multi-View Consistency (MTMVC) for hand pose estimation. First, we obtain the joints based on heatmap prediction and coordinate regression parallelly and encourage their consistency. Second, we introduce multi-view consistency to encourage the predicted poses to be rotation-invariant. Thirdly, to make the network pay more attention to the hand region, we propose a spatially weighted consistency. Experiments on four public datasets showed that our proposed MTMVC outperformed existing semi-supervised hand pose estimation methods, and by only using half of the annotations, the accuracy of our method was comparable to those of several state-of-the-art fully supervised methods.

Prescriptive Method for Optimizing Cost of Data Collection and Annotation in Machine Learning of Clinical Ultrasound.

Using public breast, fetal, and lung ultrasound datasets we can: Optimize data collection by statistically predicting accuracy for a desired dataset size; and optimize labeling efficiency using Active Learning, where predictions with lowest certainty were labelled manually using feedback. A practical case study on BUSI data was used to demonstrate the method prescribed in this work. With small data subsets, ~10%, dataset size vs. final accuracy relations can be predicted with diminishing results after 50% usage. Manual annotation was reduced by ~10% using active learning to focus the annotation. This led to cost reductions of 50%-66%, depending on requirements and initial cost model, on BUSI dataset with a negligible accuracy drop of 3.75% from theoretical maximums.Clinical Relevance- This work provides methodology to optimize dataset size and manual data labelling, this allows generation of cost-effective datasets, of interest to all, but particularly for financially limited trials and feasibility studies, Reducing the time burden on annotating clinicians.

Density-based one-shot active learning for image segmentation

Image segmentation is a key step in image processing tasks, which has significant applications in computer vision field such as medical image analysis, scene understanding and video monitoring, etc. However, image segmentation tasks usually require a large number of labeled samples to obtain great performance of convolutional neural networks (CNNs). Active learning (AL) can select valuable samples for annotation, so as to reduce the annotation cost as much as possible while maintaining the performance of CNNs. Further, one-shot AL can select valuable samples by once, which eliminates the need for iterative sample selection and annotation. However, existing one-shot AL approaches extremely rely on complex clustering algorithm, which brings a limitation in practice, i.e., we often do not know how to set the hyperparameters. In this paper, we propose a clustering-free one-shot AL framework, which is based on self-supervised feature learning and density-based query strategy. Our framework can select samples with high local density robustly against hyperparameters. The experimental results are impressive that state-of-the-art one-shot active learning performance can be achieved with simple density-based sampling.

Dynamic facial expression recognition with pseudo‐label guided multi‐modal pre‐training

AbstractDue to the huge cost of manual annotations, the labelled data may not be sufficient to train a dynamic facial expression (DFR) recogniser with good performance. To address this, the authors propose a multi‐modal pre‐training method with a pseudo‐label guidance mechanism to make full use of unlabelled video data for learning informative representations of facial expressions. First, the authors build a pre‐training dataset of videos with aligned vision and audio modals. Second, the vision and audio feature encoders are trained through an instance discrimination strategy and a cross‐modal alignment strategy on the pre‐training data. Third, the vision feature encoder is extended as a dynamic expression recogniser and is fine‐tuned on the labelled training data. Fourth, the fine‐tuned expression recogniser is adopted to predict pseudo‐labels for the pre‐training data, and then start a new pre‐training phase with the guidance of pseudo‐labels to alleviate the long‐tail distribution problem and the instance‐class confliction. Fifth, since the representations learnt with the guidance of pseudo‐labels are more informative, a new fine‐tuning phase is added to further boost the generalisation performance on the DFR recognition task. Experimental results on the Dynamic Facial Expression in the Wild dataset demonstrate the superiority of the proposed method.

Efficient unsupervised learning of biological images with compressed deep features

Machine learning has significantly impacted the analysis of biological images and is now an important part of many biological data analysis pipelines. A variety of biological and biomedical domain-related tasks is gaining benefit from image analysis and pattern recognition tools developed currently. Applications include diagnostic histopathology, environmental monitoring, synthetic biology, genomics, and proteomics. Particularly in the last decade, several deep learning and advanced computer vision methods such as convolutional neural networks (CNNs), typically trained in a supervised fashion, have started to be largely employed in biological image classification. Moreover, the advancement of automatic acquisition systems has been generating a massive amount of biological data, which requires to be analyzed by domain experts. However, the cost of manual annotation of such data has become a bottleneck, impairing the application of supervised machine learning algorithms. Biological images generally have an intrinsic high variability, whose identity is sometimes hard to assign and strongly dependent on the annotator’s expertise. In this context, a limited number of annotation-free (i.e., unsupervised) learning solutions have been proposed, typically based on hand-crafted features, specifically tailored for a certain biological domain. Nonetheless, a successful unsupervised learning approach must be accurate, and sufficiently robust to deal with different biological domains. This paper aims at providing a viable solution to these issues, proposing an unsupervised learning algorithm based on compressed deep features for image classification. We exploit features extracted from ImageNet pre-trained transformers and CNNs, further compressed with a customized β-Variational AutoEncoder (β-VAE), that we call reconstruction VAE (R-VAE). We test our algorithm on biological images coming from diverse domains characterized by high variability in shape and texture information and acquired with widely differing imaging platforms. Considered image datasets range from multi-cellular organisms (plankton, coral) to sub-cellular organelles (budding yeast vacuoles, human cells’ nuclei, etc.). Our results show that the compressed deep features extracted from different pre-trained vision models establish new unsupervised learning state-of-the-art performances for the investigated datasets.

Learning from algorithm-generated pseudo-annotations for detecting ants in videos

Deep learning (DL) based detection models are powerful tools for large-scale analysis of dynamic biological behaviors in video data. Supervised training of a DL detection model often requires a large amount of manually-labeled training data which are time-consuming and labor-intensive to acquire. In this paper, we propose LFAGPA (Learn From Algorithm-Generated Pseudo-Annotations) that utilizes (noisy) annotations which are automatically generated by algorithms to train DL models for ant detection in videos. Our method consists of two main steps: (1) generate foreground objects using a (set of) state-of-the-art foreground extraction algorithm(s); (2) treat the results from step (1) as pseudo-annotations and use them to train deep neural networks for ant detection. We tackle several challenges on how to make use of automatically generated noisy annotations, how to learn from multiple annotation resources, and how to combine algorithm-generated annotations with human-labeled annotations (when available) for this learning framework. In experiments, we evaluate our method using 82 videos (totally 20,348 image frames) captured under natural conditions in a tropical rain-forest for dynamic ant behavior study. Without any manual annotation cost but only algorithm-generated annotations, our method can achieve a decent detection performance (77% in F_1 score). Moreover, when using only 10% manual annotations, our method can train a DL model to perform as well as using the full human annotations (81% in F_1 score).

Research on classification method of high myopic maculopathy based on retinal fundus images and optimized ALFA-Mix active learning algorithm.

To conduct a classification study of high myopic maculopathy (HMM) using limited datasets, including tessellated fundus, diffuse chorioretinal atrophy, patchy chorioretinal atrophy, and macular atrophy, and minimize annotation costs, and to optimize the ALFA-Mix active learning algorithm and apply it to HMM classification. The optimized ALFA-Mix algorithm (ALFA-Mix+) was compared with five algorithms, including ALFA-Mix. Four models, including ResNet18, were established. Each algorithm was combined with four models for experiments on the HMM dataset. Each experiment consisted of 20 active learning rounds, with 100 images selected per round. The algorithm was evaluated by comparing the number of rounds in which ALFA-Mix+ outperformed other algorithms. Finally, this study employed six models, including EfficientFormer, to classify HMM. The best-performing model among these models was selected as the baseline model and combined with the ALFA-Mix+ algorithm to achieve satisfactory classification results with a small dataset. ALFA-Mix+ outperforms other algorithms with an average superiority of 16.6, 14.75, 16.8, and 16.7 rounds in terms of accuracy, sensitivity, specificity, and Kappa value, respectively. This study conducted experiments on classifying HMM using several advanced deep learning models with a complete training set of 4252 images. The EfficientFormer achieved the best results with an accuracy, sensitivity, specificity, and Kappa value of 0.8821, 0.8334, 0.9693, and 0.8339, respectively. Therefore, by combining ALFA-Mix+ with EfficientFormer, this study achieved results with an accuracy, sensitivity, specificity, and Kappa value of 0.8964, 0.8643, 0.9721, and 0.8537, respectively. The ALFA-Mix+ algorithm reduces the required samples without compromising accuracy. Compared to other algorithms, ALFA-Mix+ outperforms in more rounds of experiments. It effectively selects valuable samples compared to other algorithms. In HMM classification, combining ALFA-Mix+ with EfficientFormer enhances model performance, further demonstrating the effectiveness of ALFA-Mix+.

WGAN-CL: A Wasserstein GAN with confidence loss for small-sample augmentation

The small-sample task is a current challenge in the field of deep learning, due to the huge annotation cost and the inherent limitations of targets, such as the acquisition of rare animal and plant images. Data augmentation is an effective method to solve the semantic sparseness and overfitting of deep convolution neural network in small-sample classification, but its effectiveness remains to be improved. We propose a Wasserstein GAN with confidence loss (WGAN-CL) to implement the expansion of small-sample plant dataset. Firstly, a shallower GAN’s structure is designed to adapt to less plant data. Meanwhile, shortcut-stream connections are brought into the basic network to enlarge the solution space of the model without producing additional training parameters. Secondly, the Wasserstein distance combined with confidence loss is used for optimizing the model. Experiments demonstrate that the Wasserstein distance with gradient penalty guarantees the stability of model training and the diversity of outputs. And the sample screening strategy based on confidence loss can ensure that the generated image is close to the real image in semantic features, which is critical for subsequent image classification. To verify the effectiveness of the WGAN-CL in plant small-sample augmentation, 2000 flower images of 5 categories in the “Flowers” dataset are utilized as training samples, while 2000 augmented images are employed for model training as well to improve the performance of a classical classifier. WGAN-CL has a significant performance improvement over state-of-the-art technologies, i.e., a 2.2% improvement in recall and a 2% improvement in F1-score. Experiments on the “Plant Leaves” dataset also achieved excellent results demonstrating that WGAN-CL can be migrated to other tasks. WGAN-CL uses less computational resources while considering both effectiveness and robustness, proved the practicality of our model.

CSAL: Cost sensitive active learning for multi-source drifting stream

Multi-source stream classification is a prominent real-world problem challenged by the limited real labels and non-stationary environment. Despite growing research achievements in this field, most existing works solved this problem by requiring all real labels of source or target streams to conduct domain adaption or transfer learning mechanisms, which brings high labeling costs. However, in real-world applications, there are usually insufficient labeled data in both source and target streams, and the annotation cost of the source and target streams are generally unequal. Thus, we propose a Cost-Sensitive Active Learning (CSAL) method for multi-source drifting streams. Specifically, a multi-source ensemble framework with an asymmetry weighting mechanism is presented to ensure beneficial knowledge transfer and avoid the negative transfer. Then, a multi-perspective similarity estimation method is proposed to evaluate the similarity of source and target streams. On this basis, a novel cost-sensitive hybrid labeling strategy that combines volatility strategy and uncertainty strategy with a cost-sensitive budget control mechanism is proposed, which adaptively selects representative samples at the appropriate time. At last, a parallel multiple hypothesis drift detection method is proposed, which can efficiently utilize real labels to detect concept drift. Experimental results on real-world and synthetic data streams show that our CSAL outperforms the state-of-the-art methods with even fewer labels.

SMIN: Semi-Supervised Multi-Modal Interaction Network for Conversational Emotion Recognition

Conversational emotion recognition is a crucial research topic in human-computer interactions. Due to the heavy annotation cost and inevitable label ambiguity, collecting large amounts of labeled data is challenging and expensive, which restricts the performance of current fully-supervised methods in this domain. To address this problem, researchers attempt to distill knowledge from unlabeled data via semi-supervised learning. However, most of these semi-supervised methods ignore multimodal interactive information, although recent works have proven that such interactive information is essential for emotion recognition. To this end, we propose a novel framework to seamlessly integrate semi-supervised learning with multimodal interactions, called ‘`Semi-supervised Multi-modal Interaction Network (SMIN)’'. SMIN contains two well-designed semi-supervised modules, ‘`Intra-modal Interactive Module (IIM)’' and ‘`Cross-modal Interactive Module (CIM)’' to learn intra- and cross-modal interactions. These two modules leverage additional unlabeled data to extract emotion-salient representations. To capture additional contextual information, we utilize the hierarchical recurrent networks followed with the hybrid fusion strategy to integrate multimodal features. These multimodal features are further utilized for conversational emotion recognition. Experimental results on four benchmark datasets (i.e., IEMOCAP, MELD, CMU-MOSI and CMU-MOSEI) demonstrate that SMIN succeeds over existing state-of-the-art strategies on emotion recognition.

Unsupervised Cross-Corpus Speech Emotion Recognition Using a Multi-Source Cycle-GAN

Speech emotion recognition (SER) plays a crucial role in understanding user feelings when developing artificial intelligence services. However, the data mismatch and label distortion between the training (source) set and the testing (target) set significantly degrade the performances when developing the SER systems. Additionally, most emotion-related speech datasets are highly contextualized and limited in size. The manual annotation cost is often too high leading to an active investigation of unsupervised cross-corpus SER techniques. In this paper, we propose a framework in unsupervised cross-corpus emotion recognition using multi-source corpus in a data augmentation manner. We introduced Corpus-Aware Emotional CycleGAN (CAEmoCyGAN) including a corpus-aware attention mechanism to aggregate each source datasets to generate the synthetic target sample. We choose the widely used speech emotion corpora the IEMOCAP and the VAM as sources and the MSP-Podcast as the target. By generating synthetic target-aware samples to augment source datasets and by directly training on this augmented dataset, our proposed multi-source target-aware augmentation method outperforms other baseline models in activation and valence classification.

Novel Intent Detection and Active Learning Based Classification (Student Abstract)

Novel intent class detection is an important problem in real world scenario for conversational agents for continuous interaction. Several research works have been done to detect novel intents in a mono-lingual (primarily English) texts and images. But, current systems lack an end-to-end universal framework to detect novel intents across various different languages with less human annotation effort for mis-classified and system rejected samples. This paper proposes NIDAL (Novel Intent Detection and Active Learning based classification), a semi-supervised framework to detect novel intents while reducing human annotation cost. Empirical results on various benchmark datasets demonstrate that this system outperforms the baseline methods by more than 10% margin for accuracy and macro-F1. The system achieves this while maintaining overall annotation cost to be just ~6-10% of the unlabeled data available to the system.

LWSIS: LiDAR-Guided Weakly Supervised Instance Segmentation for Autonomous Driving

Image instance segmentation is a fundamental research topic in autonomous driving, which is crucial for scene understanding and road safety. Advanced learning-based approaches often rely on the costly 2D mask annotations for training. In this paper, we present a more artful framework, LiDAR-guided Weakly Supervised Instance Segmentation (LWSIS), which leverages the off-the-shelf 3D data, i.e., Point Cloud, together with the 3D boxes, as natural weak supervisions for training the 2D image instance segmentation models. Our LWSIS not only exploits the complementary information in multimodal data during training but also significantly reduces the annotation cost of the dense 2D masks. In detail, LWSIS consists of two crucial modules, Point Label Assignment (PLA) and Graph-based Consistency Regularization (GCR). The former module aims to automatically assign the 3D point cloud as 2D point-wise labels, while the atter further refines the predictions by enforcing geometry and appearance consistency of the multimodal data. Moreover, we conduct a secondary instance segmentation annotation on the nuScenes, named nuInsSeg, to encourage further research on multimodal perception tasks. Extensive experiments on the nuInsSeg, as well as the large-scale Waymo, show that LWSIS can substantially improve existing weakly supervised segmentation models by only involving 3D data during training. Additionally, LWSIS can also be incorporated into 3D object detectors like PointPainting to boost the 3D detection performance for free. The code and dataset are available at https://github.com/Serenos/LWSIS.

PARCS: A Deployment-Oriented AI System for Robust Parcel-Level Cropland Segmentation of Satellite Images

Cropland segmentation of satellite images is an essential basis for crop area and yield estimation tasks in the remote sensing and computer vision interdisciplinary community. Instead of common pixel-level segmentation results with salt-and-pepper effects, a parcel-level output conforming to human recognition is required according to the clients' needs during the model deployment. However, leveraging CNN-based models requires fine-grained parcel-level labels, which is an unacceptable annotation burden. To cure these practical pain points, in this paper, we present PARCS, a holistic deployment-oriented AI system for PARcel-level Cropland Segmentation. By consolidating multi-disciplinary knowledge, PARCS has two algorithm branches. The first branch performs pixel-level crop segmentation by learning from limited labeled pixel samples with an active learning strategy to avoid parcel-level annotation costs. The second branch aims at generating the parcel regions without a learning procedure. The final parcel-level segmentation result is achieved by integrating the outputs of these two branches in tandem. The robust effectiveness of PARCS is demonstrated by its outstanding performance on public and in-house datasets (an overall accuracy of 85.3% and an mIoU of 61.7% on the public PASTIS dataset, and an mIoU of 65.16% on the in-house dataset). We also include subjective feedback from clients and discuss the lessons learned from deployment.