Label Refinement Research Articles

Label refinement network from synthetic error augmentation for medical image segmentation

Deep convolutional neural networks for image segmentation do not learn the label structure explicitly and may produce segmentations with an incorrect structure, e.g., with disconnected cylindrical structures in the segmentation of tree-like structures such as airways or blood vessels. In this paper, we propose a novel label refinement method to correct such errors from an initial segmentation, implicitly incorporating information about label structure. This method features two novel parts: (1) a model that generates synthetic structural errors, and (2) a label appearance simulation network that produces segmentations with synthetic errors that are similar in appearance to the real initial segmentations. Using these segmentations with synthetic errors and the original images, the label refinement network is trained to correct errors and improve the initial segmentations. The proposed method is validated on two segmentation tasks: airway segmentation from chest computed tomography (CT) scans and brain vessel segmentation from 3D CT angiography (CTA) images of the brain. In both applications, our method significantly outperformed a standard 3D U-Net, four previous label refinement methods, and a U-Net trained with a loss tailored for tubular structures. Improvements are even larger when additional unlabeled data is used for model training. In an ablation study, we demonstrate the value of the different components of the proposed method.

PIS-Net: Efficient weakly supervised instance segmentation network based on annotated points for rice field weed identification

Weed damage in rice fields is one of the main causes of reduced rice yields and quality. Accurate and efficient weed identification is the prerequisite for realizing intelligent and precise weeding in paddies. Recently, Vision Transformers (ViTs) have emerged with superior performance on computer vision tasks compared to the convolutional neural network (CNN)-based models. However, the lack of fully labeled weed datasets hinders the potential application of deep learning models in weed identification. To address the above issues, this study customizes a novel point-supervised instance segmentation network (PIS-Net) for weakly supervised instance segmentation of weeds in rice fields. More correctly, we first propose a novel instance segmentation point labeling scheme that utilizes randomly generated annotation points within each instance, aiming to decrease both labeling time and difficulty. Additionally, to make optimal use of point labels, this study puts forth a mask generation strategy based on the adaptive selection of pyramid levels. In this sense, the network model can flexibly choose the pyramid level expected to generate the most suitable instance mask based on the network's reliability. Finally, we establish the pseudo label refinement network (PLR-Net) to refine rough instance masks. The proposed PIS-Net utilizes 13 randomly generated annotation points for each instance, yet achieving an AP of 38.5 and an AP50 of 68.3, which is superior to the baseline mask-R-CNN with an AP of 8.2 and AP50 of 6.9, achieving 90 % fully supervised performance. This method effectively utilizes point labels, annotated with high efficiency, as a robust source of weak supervision to address challenges in weed data annotation and the low accuracy of existing weakly supervised models. Experiments show that the point annotation scheme of the PIS-Net is faster than full-object mask annotation, and the AP is also higher than the current semi-supervised weed segmentation model, enjoying high potentials in practical paddy fields.

A novel Parallel Cooperative Mean-Teacher framework (PCMT) combined with prediction uncertainty guide and class contrastive learning for semi-supervised polyp segmentation

Polyp segmentation technology based on deep learning can quickly and accurately help doctors locate lesions, but its development is limited by pixel-level annotations. The polyp segmentation methods based on semi-supervised learning(SSL) is an effective solution to alleviate annotation pressure. For mainstream semi-supervised polyp segmentation methods, due to the extremely deep depth of the polyp segmentation network and the limited use of labeled data for training, the polyp segmentation network is prone to feature expression bottlenecks. And it is a deviation direction to refine pseudo-labels using regional mathematical evaluation, rather than using pixel-level pseudo label refinement schemes. To solve the limitation, based on SSL, a novel Parallel Cooperative Mean-Teacher framework (PCMT) is proposed. PCMT has several effective components: (1) using the segmentation network with deep supervision branches,which is trained by the Parallel Cooperative Mean-Teacher framework for solving feature expression bottlenecks, (2) prediction uncertainty guide for providing more reliable pseudo-labels, (3) class contrastive regularization for strengthening the affinity between the foreground and background features. PCMT and state-of-the-art semi-supervised methods have been compared using 10%, 20%, 50%, and 100% labeled data on five public datasets, respectively. A large number of experimental results show that PCMT has more prominent prediction performance and better generalization ability. Meanwhile, the segmentation performance of PCMT with only requiring 20% labeled data could reach that of full supervised methods with requiring 100% labeled data. Above the results prove that it is feasible of PCMT for semi-supervised polyp segmentation tasks.

Multi-stage ensemble with refinement for noisy labeled data classification

Deep neural networks (DNNs) have made remarkable progress in image classification. However, since DNNs can memorize all the label information in the training dataset due to their excellent feature learning ability, the generalization performance deteriorates when they are trained on the noisy labeled dataset that can be easily found in real-world problems. In this paper, we propose a multi-stage ensemble method with label refinement to build an effective classification model under noisy labels. The proposed method iteratively refines the dataset by re-labeling the samples at the end of each stage, which enables the models trained at each stage to learn different features. By integrating these models, the proposed multi-stage ensemble method exerts powerful generalization performance. Also, we suggest a novel dataset refinement method, demonstrating the effectiveness of a robust function in distinguishing corrupted samples. Experimental results on the benchmark and real-world datasets show that the proposed method outperforms the existing methods on the noisy labeled dataset classification.

CMM: A CNN-MLP Model for COVID-19 Lesion Segmentation and Severity Grading.

In this paper, a CNN-MLP model (CMM) is proposed for COVID-19 lesion segmentation and severity grading in CT images. The CMM starts by lung segmentation using UNet, and then segmenting the lesion from the lung region using a multi-scale deep supervised UNet (MDS-UNet), finally implementing the severity grading by a multi-layer preceptor (MLP). In MDS-UNet, shape prior information is fused with the input CT image to reduce the searching space of the potential segmentation outputs. The multi-scale input compensates for the loss of edge contour information in convolution operations. In order to enhance the learning of multiscale features, the multi-scale deep supervision extracts supervision signals from different upsampling points on the network. In addition, it is empirical that the lesion which has a whiter and denser appearance tends to be more severe in the COVID-19 CT image. So, the weighted mean gray-scale value (WMG) is proposed to depict this appearance, and together with the lung and lesion area to serve as input features for the severity grading in MLP. To improve the precision of lesion segmentation, a label refinement method based on the Frangi vessel filter is also proposed. Comparative experiments on COVID-19 public datasets show that our proposed CMM achieves high accuracy on COVID-19 lesion segmentation and severity grading.

GlyReShot: A glyph-aware model with label refinement for few-shot Chinese agricultural named entity recognition

Chinese agricultural named entity recognition (NER) has been studied with supervised learning for many years. However, considering the scarcity of public datasets in the agricultural domain, exploring this task in the few-shot scenario is more practical for real-world demands. In this paper, we propose a novel model named GlyReShot, integrating the knowledge of Chinese character glyph into few-shot NER models. Although the utilization of glyph has been proven successful in supervised models, two challenges still persist in the few-shot setting, i.e., how to obtain glyph representations and when to integrate them into the few-shot model. GlyReShot handles the two challenges by introducing a lightweight glyph representation obtaining module and a training-free label refinement strategy. Specifically, the glyph representations are generated based on the descriptive sentences by filling the predefined template. As most steps come before training, this module aligns well with the few-shot setting. Furthermore, by computing the confidence values for draft predictions, the refinement strategy selectively utilizes the glyph information only when the confidence values are relatively low, thus mitigating the influence of noise. Finally, we annotate a new agricultural NER dataset and the experimental results demonstrate effectiveness of GlyReShot for few-shot Chinese agricultural NER.

Reliable hybrid knowledge distillation for multi-source domain adaptive object detection

Although great efforts have been made in single source adaptation, most methods do not consider a more practical scenario where the source domain comprises multiple data distributions. In this paper, we propose a new multiple teacher–student framework (MTSF) for multi-source domain adaptive object detection. Our MTSF adopts multiple teacher–student sub-networks to distill domain specific knowledge from different source domains, yielding multiple precise pseudo labels. Considering that the samples from different domains are biased, we propose a new knowledge fusion module namely hybrid knowledge distillation (HKD) to fuse the knowledge by considering the contribution of each sample from different domains to the final decision on the target domain. Our HKD consists of a multi-source prototype construction (MSPC) module and a dense-to-sparse cross-domain pseudo label refinement (DCPR) module. According to the prototypes measured by MSPC, the DCPR distills the knowledge from dense to sparse in a hierarchical way, which makes the generated pseudo labels more reliable. When evaluated on benchmark adaptation scenarios (i.e. cross-camera and cross-time), our proposed method outperforms previous methods, establishing a new state-of-the-art.

Intelligent agent for hyperspectral image classification with noisy labels: a deep reinforcement learning framework

ABSTRACT Label accuracy plays an important role in supervised hyperspectral image classification problems. However, labelling can be prone to various types of noise, mainly including human errors and incorrect pseudo-label predictions. Learning from datasets with noisy labels is a critical challenge in practice. To address this issue, we first propose a deep reinforcement learning method named HyperRefine to tackle the label noise problem for hyperspectral image classification. The method utilizes an agent to learn the accuracy of each sample label and determine its suitability to be used in training the classifier model. The agent is trained using a reinforcement signal based on the reward obtained from a small validation set that reflects performance on the target task, where the validation set is generated from the training set by a momentum-based recruiter. Compared to complex hand-crafted methods of label noise cleaning, HyperRefine is easy to implement, yet highly effective. The robustness of the model to different types of label noises is significantly improved by further combining a novel label refinement approach. Experiments on artificially added label noise and pseudo-label noise hyperspectral datasets show that the proposed method can achieve promising performance.

Enhancing point annotations with superpixel and confident learning guided for improving semi-supervised OCT fluid segmentation

Automatic segmentation of fluid in Optical Coherence Tomography (OCT) images is beneficial for ophthalmologists to make an accurate diagnosis. Although semi-supervised OCT fluid segmentation networks enhance their performance by introducing additional unlabeled data, the performance enhancement is limited. To address this, we propose Superpixel and Confident Learning Guide Point Annotations Network (SCLGPA-Net) based on the teacher-student architecture, which can learn OCT fluid segmentation from limited fully-annotated data and abundant point-annotated data. Specifically, we use points to annotate fluid regions in unlabeled OCT images and the Superpixel-Guided Pseudo-Label Generation (SGPLG) module generates pseudo-labels and pixel-level label trust maps from the point annotations. The label trust maps provide an indication of the reliability of the pseudo-labels. Furthermore, we propose the Confident Learning Guided Label Refinement (CLGLR) module identifies error information in the pseudo-labels and leads to further refinement. Experiments on the RETOUCH dataset show that we are able to reduce the need for fully-annotated data by 94.22%, closing the gap with the best fully supervised baselines to a mean IoU of only 2%. Furthermore, we constructed a private 2D OCT fluid segmentation dataset for evaluation. Compared with other methods, comprehensive experimental results demonstrate that the proposed method can achieve excellent performance in OCT fluid segmentation.

When Sparse Neural Network Meets Label Noise Learning: A Multistage Learning Framework.

Recent methods in network pruning have indicated that a dense neural network involves a sparse subnetwork (called a winning ticket), which can achieve similar test accuracy to its dense counterpart with much fewer network parameters. Generally, these methods search for the winning tickets on well-labeled data. Unfortunately, in many real-world applications, the training data are unavoidably contaminated with noisy labels, thereby leading to performance deterioration of these methods. To address the above-mentioned problem, we propose a novel two-stream sample selection network (TS3-Net), which consists of a sparse subnetwork and a dense subnetwork, to effectively identify the winning ticket with noisy labels. The training of TS3-Net contains an iterative procedure that switches between training both subnetworks and pruning the smallest magnitude weights of the sparse subnetwork. In particular, we develop a multistage learning framework including a warm-up stage, a semisupervised alternate learning stage, and a label refinement stage, to progressively train the two subnetworks. In this way, the classification capability of the sparse subnetwork can be gradually improved at a high sparsity level. Extensive experimental results on both synthetic and real-world noisy datasets (including MNIST, CIFAR-10, CIFAR-100, ANIMAL-10N, Clothing1M, and WebVision) demonstrate that our proposed method achieves state-of-the-art performance with very small memory consumption for label noise learning. Code is available at https://github.com/Runqing-forMost/TS3-Net/tree/master.

MIMR: Modality-Invariance Modeling and Refinement for unsupervised visible-infrared person re-identification

Compared to visible modality person re-identification that handles only the intra-modality discrepancy, visible-infrared person re-identification (VI-ReID) aims to achieve the match between visible modality and infrared modality. However, supervised VI-ReID methods are limited in flexibility due to their dependence on labeled data. Although several unsupervised VI-ReID methods have been developed, they usually ignore the impact of hard samples and noisy labels. In this paper, we propose the modality-invariance modeling and refinement (MIMR) framework for unsupervised VI-ReID. For the first issue, we first conduct cross-modality image translation, then enable the encoder to extract modality-invariant feature representation by aligning translated versions with original samples, and finally use cross-modality clustering and hard sample contrastive loss to handle the hard samples. For the second issue, we believe that ambiguous samples are usually assigned noisy labels, so we design ambiguity-oriented pseudo label refinement (APLR), which evaluates ambiguity from both the sample itself and the corresponding translated version, rather than discards hard samples as existing methods. Extensive experiments demonstrate that MIMR achieves superior performance compared to state-of-the-art unsupervised methods and even surpasses early supervised methods.

DCLR-SF: distribution consistent label refinement and lighten similarity network fusion for multi-source domain-adaptive person re-identification

DCLR-SF: distribution consistent label refinement and lighten similarity network fusion for multi-source domain-adaptive person re-identification

Learning More from Mixed Emotions: A Label Refinement Method for Emotion Recognition in Conversations

Abstract One-hot labels are commonly employed as ground truth in Emotion Recognition in Conversations (ERC). However, this approach may not fully encompass all the emotions conveyed in a single utterance, leading to suboptimal performance. Regrettably, current ERC datasets lack comprehensive emotionally distributed labels. To address this issue, we propose the Emotion Label Refinement (EmoLR) method, which utilizes context- and speaker-sensitive information to infer mixed emotional labels. EmoLR comprises an Emotion Predictor (EP) module and a Label Refinement (LR) module. The EP module recognizes emotions and provides context/speaker states for the LR module. Subsequently, the LR module calculates the similarity between these states and ground-truth labels, generating a refined label distribution (RLD). The RLD captures a more comprehensive range of emotions than the original one-hot labels. These refined labels are then used for model training in place of the one-hot labels. Experimental results on three public conversational datasets demonstrate that our EmoLR achieves state-of-the-art performance.

ICLR: Instance Credibility-Based Label Refinement for label noisy person re-identification

Person re-identification (Re-ID) has demonstrated remarkable performance when trained on accurately annotated data. However, in practical applications, the presence of annotation errors is unavoidable, which can undermine the accuracy and robustness of the Re-ID model training. To address the adverse impacts of label noise, especially in scenarios with limited training samples for each identity (ID), a common approach is to utilize all the available sample labels. Unfortunately, these labels contain incorrect labels, leading to the model being influenced by noise and compromising its performance. In this paper, we propose an Instance Credibility-based Label Refinement and Re-weighting (ICLR) framework to exploit partially credible labels to refine and re-weight incredible labels effectively. Specifically, the Label-Incredibility Optimization (LIO) module is proposed to optimize incredible labels before model training, which partitions the samples into credible and incredible samples and propagates credible labels to others. Furthermore, we design an Incredible Instance Re-weight (I2R) strategy, aiming to emphasize instances that contribute more significantly and dynamically adjust the weight of each instance. The proposed method seamlessly reinforces accuracy without requiring additional information or discarding any samples. Extensive experimental results conducted on Market-1501 and Duke-MTMC datasets demonstrate the effectiveness of our proposed method, leading to a substantial improvement in performance under both random noise and pattern noise settings. Code will be available at https://github.com/whut16/ReID-Label-Noise.

Multiscale and multidirection depth map super resolution with semantic inference

AbstractDepth map super resolution has been paid much attention in 3D applications due to the limitation of depth sensors. Few textures in objects with clear contours along them is the most important characteristic of depth map. An efficient image representation should be directional, multiscale and anisotropic. From this we propose a novel multiscale and multidirection depth map super resolution framework with semantic inference to improve the quality of depth maps. In this framework, a multiscale and multidirection depth map contour fusion scheme captures and assembles intrinsic geometrical structures through a multiview non‐subsampled contourlet transform manner. This scheme not only isolates the discontinuities of contours but retains the smoothness along the contours. The semantic inference is also utilized to segment and label the depth map into objects/backgrounds‐level which are coplanar. Furthermore, a semantic‐aware label refinement strategy is introduced to correct the rarely inaccurate labels of the label map for upscaling the target pixel with pixels in the same object or background. Experimental results on benchmark depth map dataset demonstrate that the proposed multiscale and multidirection depth map super resolution framework with semantic inference has a significant improvement than the state‐of‐the‐art algorithms both visually and quantitatively.

Corrigendum

In the published article [Distribution Alignment Using Complement Entropy Objective and Adaptive Consensus-Based Label Refinement for Partial Domain Adaptation], there was an error in the Data Availability statement. The following corrections should be noted: In the section 4.2. “Parameters λ and η, which regulate the learning from adaptive complement entropy objective, are set to 1, 8,and 0.3, 1 for Office-31 and Office-Home, respectively.” should be “ Parameters λ and η, which regulate the learning from adaptive complement entropy objective, are set to 0.7, 8, and 0.3, 1 for Office-31 and Office-Home, respectively.” and the setting valuable of Table 1 for parameters λ should be 0.7; In Table 5 the mean accuracy for different values of parameters λ on Office-31 should be 96.63, 96.98, 97.07, 97.19, 97.25, 96.71 and 96.31 respectively. The authors apologize for this error and state that this does not change the scientific conclusions of the article in any way. Reference Choudhuri, S., Adeniye, S., & Sen, A. (2023). Distribution Alignment Using Complement Entropy Objective and Adaptive ConsensusBased Label Refinement For Partial Domain Adaptation. Artificial Intelligence and Applications, 1(1), 43–51. https://doi.org/10. 47852/bonviewAIA2202524

Learning class-agnostic masks with cross-task refinement for weakly supervised semantic segmentation

Weakly supervised semantic segmentation (WSSS) commonly relies on Class Activation Mapping (CAM) to produce pseudo semantic labels using image-level annotations. However, because CAM maps often form sparse object regions with poor boundaries, they cannot provide sufficient segmentation supervision. Because off-the-shelf saliency maps can provide rich object boundaries that can be leveraged to improve semantic segmentation, we propose to jointly learn semantic segmentation and class-agnostic masks by using image-level annotations and off-the-shelf saliency maps as supervision. We also propose a cross-task label refinement mechanism, which takes advantage of the learned class-agnostic masks and semantic segmentation masks, to refine the pseudo labels and provide more accurate supervision to both tasks. Moreover, we introduce a new normalization method for CAM to generate more complete class-specific localization maps. The improved CAM maps complement our learned class-agnostic masks, leading to high-quality pseudo semantic segmentation labels. Extensive experiments demonstrate the effectiveness of the proposed approach, with state-of-the-art WSSS results established on PASCAL VOC 2012 and MS COCO.

Frame-Level Label Refinement for Skeleton-Based Weakly-Supervised Action Recognition

In recent years, skeleton-based action recognition has achieved remarkable performance in understanding human motion from sequences of skeleton data, which is an important medium for synthesizing realistic human movement in various applications. However, existing methods assume that each action clip is manually trimmed to contain one specific action, which requires a significant amount of effort for annotation. To solve this problem, we consider a novel problem of skeleton-based weakly-supervised temporal action localization (S-WTAL), where we need to recognize and localize human action segments in untrimmed skeleton videos given only the video-level labels. Although this task is challenging due to the sparsity of skeleton data and the lack of contextual clues from interaction with other objects and the environment, we present a frame-level label refinement framework based on a spatio-temporal graph convolutional network (ST-GCN) to overcome these difficulties. We use multiple instance learning (MIL) with video-level labels to generate the frame-level predictions. Inspired by advances in handling the noisy label problem, we introduce a label cleaning strategy of the frame-level pseudo labels to guide the learning process. The network parameters and the frame-level predictions are alternately updated to obtain the final results. We extensively evaluate the effectiveness of our learning approach on skeleton-based action recognition benchmarks. The state-of-the-art experimental results demonstrate that the proposed method can recognize and localize action segments of the skeleton data.

Adapting Object Size Variance and Class Imbalance for Semi-supervised Object Detection

Semi-supervised object detection (SSOD) attracts extensive research interest due to its great significance in reducing the data annotation effort. Collecting high-quality and category-balanced pseudo labels for unlabeled images is critical to addressing the SSOD problem. However, most of the existing pseudo-labeling-based methods depend on a large and fixed threshold to select high-quality pseudo labels from the predictions of a teacher model. Considering different object classes usually have different detection difficulty levels due to scale variance and data distribution imbalance, conventional pseudo-labeling-based methods are arduous to explore the value of unlabeled data sufficiently. To address these issues, we propose an adaptive pseudo labeling strategy, which can assign thresholds to classes with respect to their “hardness”. This is beneficial for ensuring the high quality of easier classes and increasing the quantity of harder classes simultaneously. Besides, label refinement modules are set up based on box jittering for guaranteeing the localization quality of pseudo labels. To further improve the algorithm’s robustness against scale variance and make the most of pseudo labels, we devise a joint feature-level and prediction-level consistency learning pipeline for transferring the information of the teacher model to the student model. Extensive experiments on COCO and VOC datasets indicate that our method achieves state-of-the-art performance. Especially, it brings mean average precision gains of 2.08 and 1.28 on MS-COCO dataset with 5% and 10% labeled images, respectively.

Convolutional Multiple Instance Learning for Sleep Spindle Detection With Label Refinement

Sleep spindles are closely associated with cognitive functions and neurological disorders; thus, spindle detection has been an important topic in sleep medicine. Recently, machine learning approaches have shown the potential in automatic sleep spindle detection by learning optimized features in a data-driven way, while they highly rely on labeled data, and the performance can be degraded when labels are inaccurate. However, accurate annotation of the spindle is usually difficult to obtain and high intraexpert and interexpert variance exist. In this work, we propose a convolutional neural network (CNN) with a label refinement component to learn an effective spindle detector with imperfect labels. Our approach consists of two stages: 1) a feature learning stage and 2) a label refinement stage. In the feature learning stage, a CNN-based multiple instance learning framework (CNN-MIL) is built for spindle feature learning. By assuming only parts of each labeled spindle segment contain true spindle patterns, the CNN-MIL model can learn most-likely spindle-related features from ambiguous labels. In the label refinement stage, we adjust the spindle labels by merging the original labels and labels predicted by CNN-MIL, and the modified labels are then used in the next round CNN-MIL feature learning. The two stages perform alternately for detector optimization. Extensive experiments demonstrated that our approach achieved the state-of-the-art performance.