Label Ambiguity Research Articles

Generative Label Enhancement with Gaussian Mixture and Partial Ranking

Label distribution learning (LDL) is an effective learning paradigm for dealing with label ambiguity. When applying LDL, the datasets annotated with label distributions (i.e., the real-valued vectors like the probability distribution) are typically required. Unfortunately, most existing datasets only contain the logical labels, and manual annotating with label distributions is costly. To address this problem, we treat the label distribution as a latent vector and infer its posterior by variational Bayes. Specifically, we propose a generative label enhancement model to encode the process of generating feature vectors and logical label vectors from label distributions in a principled way. In terms of features, we assume that the feature vector is generated by a Gaussian mixture dominated by the label distribution, which captures the one-to-many relationship from the label distribution to the feature vector and thus reduces the feature generation error. In terms of logical labels, we design a probability distribution to generate the logical label vector from a label distribution, which captures partial label ranking in the logical label vector and thus provides a more accurate guidance for inferring the label distribution. Besides, to approximate the posterior of the label distribution, we design a inference model, and derive the variational learning objective. Finally, extensive experiments on real-world datasets validate our proposal.

Global Mixup: Eliminating Ambiguity with Clustering

Data augmentation with Mixup has been proven an effective method to regularize the current deep neural networks. Mixup generates virtual samples and corresponding labels simultaneously by linear interpolation. However, the one-stage generation paradigm and the use of linear interpolation have two defects: (1) The label of the generated sample is simply combined from the labels of the original sample pairs without reasonable judgment, resulting in ambiguous labels. (2) Linear combination significantly restricts the sampling space for generating samples. To address these issues, we propose a novel and effective augmentation method, Global Mixup, based on global clustering relationships. Specifically, we transform the previous one-stage augmentation process into two-stage by decoupling the process of generating virtual samples from the labeling. And for the labels of the generated samples, relabeling is performed based on clustering by calculating the global relationships of the generated samples. Furthermore, we are no longer restricted to linear relationships, which allows us to generate more reliable virtual samples in a larger sampling space. Extensive experiments for CNN, LSTM, and BERT on five tasks show that Global Mixup outperforms previous baselines. Further experiments also demonstrate the advantage of Global Mixup in low-resource scenarios.

Remarks on labelling and determinacy

AbstractWork by Chomsky et al. (2019) and Epstein et al. (2018) develops a third-factor principle of computational efficiency called “Determinacy”, which rules out “ambiguous” syntactic rule-applications by requiring one-to-one correspondences between the input or output of a rule and a single term in the domain of that rule. This article first adopts the concept of “Input Determinacy” articulated by Goto and Ishii (2019, 2020), who apply Determinacy specifically to the input of operations like Merge, and then proposes to extend Determinacy to the labeling procedure developed by Chomsky (2013, 2015). In particular, Input Determinacy can explain restrictions on labeling in contexts where multiple potential labels are available (labeling ambiguity), and it can also provide an explanation for Chomsky's (2013, 2015) proposal that syntactic movement of an item (“Internal Merge”) renders that item invisible to the labeling procedure.

Umbrella terms conceal the sale of threatened shark species: A DNA barcoding approach

Food fraud in the seafood industry is a growing concern, including the authenticity and provenance of seafood products that are often unable to be confidently verified along complex, and at times obscure, supply chains. Among the diverse types of seafood fraud, mislabelling and species substitution are common, and have potential implications on human health, the economy, and species conservation. In particular, shark meat can be difficult to monitor, associated with the use of an array of convoluted nomenclature or ambiguous umbrella terms. In Australia, shark meat fillets are a popular seafood item sold under the umbrella term flake in takeaways, and fish and chip shops. We collected samples of flake as sold to consumers at 104 retailers in South Australia and used DNA barcoding to determine the shark species being traded. At least nine distinct species were identified, with only 27% of all samples identified as gummy shark (Mustelus antarcticus), a species that has sustainable fisheries, is locally caught, and is one of two species that is recommended to be labelled as flake in Australia. Four species were threatened taxa, including the CITES Appendix II listed short-fin mako (Isurus oxyrinchus) and smooth hammerhead (Sphyrna zygaena). Of all the samples and shops analysed, only 11% of retailers correctly identified the species they sold, another 20% were mislabelled and the remaining only had ambiguous labelling. Ultimately, the umbrella term flake allowed for species misrepresentation but DNA barcoding was an effective tool to test ambiguous labelling in processed and cooked shark meat products, and can guide policy, management, and compliance efforts to mitigate mislabelling, empowering consumers to make informed decisions and champion sustainable seafood.

Less is More: Surgical Phase Recognition from Timestamp Supervision.

Surgical phase recognition is a fundamental task in computer-assisted surgery systems. Most existing works are under the supervision of expensive and time-consuming full annotations, which require the surgeons to repeat watching videos to find the precise start and end time for a surgical phase. In this paper, we introduce timestamp supervision for surgical phase recognition to train the models with timestamp annotations, where the surgeons are asked to identify only a single timestamp within the temporal boundary of a phase. This annotation can significantly reduce the manual annotation cost compared to the full annotations. To make full use of such timestamp supervisions, we propose a novel method called uncertainty-aware temporal diffusion (UATD) to generate trustworthy pseudo labels for training. Our proposed UATD is motivated by the property of surgical videos, i.e., the phases are long events consisting of consecutive frames. To be specific, UATD diffuses the single labelled timestamp to its corresponding high confident ( i.e., low uncertainty) neighbour frames in an iterative way. Our study uncovers unique insights of surgical phase recognition with timestamp supervision: 1) timestamp annotation can reduce 74% annotation time compared with the full annotation, and surgeons tend to annotate those timestamps near the middle of phases; 2) extensive experiments demonstrate that our method can achieve competitive results compared with full supervision methods, while reducing manual annotation costs; 3) less is more in surgical phase recognition, i.e., less but discriminative pseudo labels outperform full but containing ambiguous frames; 4) the proposed UATD can be used as a plug-and-play method to clean ambiguous labels near boundaries between phases, and improve the performance of the current surgical phase recognition methods. Code and annotations obtained from surgeons are available at https://github.com/xmed-lab/ TimeStamp-Surgical.

Learning from ambiguous labels for X-Ray security inspection via weakly supervised correction

Learning from ambiguous labels for X-Ray security inspection via weakly supervised correction

Single-channel speech separation using soft-minimum permutation invariant training

The goal of speech separation is to extract multiple speech sources from a single microphone recording. Recently, with the advancement of deep learning and availability of large datasets, speech separation has been formulated as a supervised learning problem. These approaches aim to learn discriminative patterns of speech, speakers, and background noise using a supervised learning algorithm, typically a deep neural network. A long-lasting problem in supervised speech separation is finding the correct label for each separated speech signal, referred to as label permutation ambiguity. Permutation ambiguity refers to the problem of determining the output-label assignment between the separated sources and the available single-speaker speech labels. Finding the best output-label assignment is required for calculation of separation error, which is later used for updating parameters of the model. Recently, Permutation Invariant Training (PIT) has been shown to be a promising solution in handling the label ambiguity problem. However, the overconfident choice of the output-label assignment by PIT results in a sub-optimal trained model. In this work, we propose a probabilistic optimization framework to address the inefficiency of PIT in finding the best output-label assignment. Our proposed method entitled trainable Soft-minimum PIT is then employed on the same Long-Short Term Memory (LSTM) architecture used in Permutation Invariant Training (PIT) speech separation method. The results of our experiments show that the proposed method outperforms conventional PIT speech separation significantly (p-value <0.01) by +1dB in Signal to Distortion Ratio (SDR) and +1.5dB in Signal to Interference Ratio (SIR).

Multi-Branch Attention Learning for Bone Age Assessment with Ambiguous Label

Bone age assessment (BAA) is a typical clinical technique for diagnosing endocrine and metabolic diseases in children’s development. Existing deep learning-based automatic BAA models are trained on the Radiological Society of North America dataset (RSNA) from Western populations. However, due to the difference in developmental process and BAA standards between Eastern and Western children, these models cannot be applied to bone age prediction in Eastern populations. To address this issue, this paper collects a bone age dataset based on the East Asian populations for model training. Nevertheless, it is laborious and difficult to obtain enough X-ray images with accurate labels. In this paper, we employ ambiguous labels from radiology reports and transform them into Gaussian distribution labels of different amplitudes. Furthermore, we propose multi-branch attention learning with ambiguous labels network (MAAL-Net). MAAL-Net consists of a hand object location module and an attention part extraction module to discover the informative regions of interest (ROIs) based only on image-level labels. Extensive experiments on both the RSNA dataset and the China Bone Age (CNBA) dataset demonstrate that our method achieves competitive results with the state-of-the-arts, and performs on par with experienced physicians in children’s BAA tasks.

Automated Sagittal Skeletal Classification of Children Based on Deep Learning.

Malocclusions are a type of cranio-maxillofacial growth and developmental deformity that occur with high incidence in children. Therefore, a simple and rapid diagnosis of malocclusions would be of great benefit to our future generation. However, the application of deep learning algorithms to the automatic detection of malocclusions in children has not been reported. Therefore, the aim of this study was to develop a deep learning-based method for automatic classification of the sagittal skeletal pattern in children and to validate its performance. This would be the first step in establishing a decision support system for early orthodontic treatment. In this study, four different state-of-the-art (SOTA) models were trained and compared by using 1613 lateral cephalograms, and the best performance model, Densenet-121, was selected was further subsequent validation. Lateral cephalograms and profile photographs were used as the input for the Densenet-121 model, respectively. The models were optimized using transfer learning and data augmentation techniques, and label distribution learning was introduced during model training to address the inevitable label ambiguity between adjacent classes. Five-fold cross-validation was conducted for a comprehensive evaluation of our method. The sensitivity, specificity, and accuracy of the CNN model based on lateral cephalometric radiographs were 83.99, 92.44, and 90.33%, respectively. The accuracy of the model with profile photographs was 83.39%. The accuracy of both CNN models was improved to 91.28 and 83.98%, respectively, while the overfitting decreased after addition of label distribution learning. Previous studies have been based on adult lateral cephalograms. Therefore, our study is novel in using deep learning network architecture with lateral cephalograms and profile photographs obtained from children in order to obtain a high-precision automatic classification of the sagittal skeletal pattern in children.

CAT: a coarse-to-fine attention tree for semantic change detection

Semantic change detection (SCD) and land cover mapping (LCM) are always treated as a dual task in the field of remote sensing. However, due to diverse real-world scenarios, many SCD categories are not easy to be clearly recognized, such as “water-vegetation” and “water-tree”, which can be regarded as fine-grained differences. In addition, even a single LCM category is usually difficult to define. For instance, some “vegetation” categories with litter vegetation coverage are easily confused with the general “ground” category. SCD/LCM becomes challenging under both challenges of its fine-grained nature and label ambiguity. In this paper, we tackle the SCD and LCM tasks simultaneously by proposing a coarse-to-fine attention tree (CAT) model. Specifically, it consists of an encoder, a decoder and a coarse-to-fine attention tree module. The encoder-decoder structure extracts the high-level features from input multi-temporal images first and then reconstructs them to return SCD and LCM predictions. Our coarse-to-fine attention tree, on the one hand, utilizes the tree structure to better model a hierarchy of categories by predicting the coarse-grained labels first and then predicting the fine-grained labels later. On the other hand, it applies the attention mechanism to capture discriminative pixel regions. Furthermore, to address label ambiguity in SCD/LCM, we also equip a label distribution learning loss upon our model. Experiments on the large-scale SECOND dataset justify that the proposed CAT model outperforms state-of-the-art models. Moreover, various ablation studies have demonstrated the effectiveness of tailored designs in the CAT model for solving semantic change detection problems.

Privileged information learning with weak labels

Privileged information learning is proposed to construct the classifier by incorporating privileged knowledge. At present, most of the privileged information learning methods assume that the instance is accurately labeled. However, in real-world applications, an instance may be weakly labeled. In this paper, we propose a novel privileged information learning method with weak labels (PLWB). The hypothesis of our work is that an instance may be annotated by a number of labelers and different labelers may give different labels to this instance due to distinct professional knowledge and subjective factors. It leads to ambiguous labels of instances, namely weak labels. To solve this problem, our methodology is to give each labeler a weight and incorporate these weights into a privileged information learning model. Our technique is to employ a heuristic framework to optimize the labeler weights and the privileged information learning model jointly. The existing privileged information learning methods do not consider the weak label problem, and assign an equal or random weight to each labeler. Our work is different from these methods. The novelty and theoretical contribution is that this is the first work to deal with the weak label problem in privileged information learning. The merit is that we assign an unknown weight to each labeler and solve the optimal values of these weights in the optimization process, such that the performance of the learning model can be improved with the optimal labeler weights. In the experiments, the tool that we use is MATLAB, in which we implement our algorithm. The experimental datasets include one handwritten categorization dataset, two image classification datasets (i.e., Animals-with-Attributes dataset and Caltech-101 dataset), and one disease diagnosis dataset (i.e., Alzheimer’s Disease Neuroimaging Initiative dataset), in which the number of instances used is 2000, 6180, 8677 and 202, respectively. The obtained results are that: (1) by optimizing the labeler weights, the proposed PLWB method obtains explicitly higher classification accuracy than the existing privileged information learning methods; (2) PLWB has relatively higher training time since it needs to solve the labeler weights in the optimization process.

Label distribution learning with high‐order label correlations

SummaryLabel distribution learning (LDL) is an emerging learning paradigm, which can be used to solve the label ambiguity problem. In spite of the recent great progress in LDL algorithms considering label correlations, the majority of existing methods only measure pairwise label correlations through the commonly used similarity metric, which is incapable of accurately reflecting the complex relationship between labels. To solve this problem, a novel label distribution learning method—based on high‐order label correlations (LDL‐HLC) is proposed. By virtue of the ‐regularization sparse reconstruction of the label space, the high‐order label correlations matrix is firstly obtained. Then, a new regular term can be constructed to fit the final prediction label distribution via the correction matrix. Furthermore, efficient classification performance and complete feature selection are guaranteed by common features learning via ‐regularization. Finally, the performance and effectiveness of the proposed algorithm are well illustrated through extensive experiments on 14 label distribution datasets and comparisons with some existing algorithms.

Animal Species Identification using Deep Learning

Bird identification is a difficult undertaking that frequently results in ambiguous labeling. Even skilled bird watchers dispute the species of a bird when presented with an image of one. It’s a demanding task that tests both people and computers’ visual talents. For example, for different scenarios, animals come with different sizes, forms, colors and from a human viewpoint with different angles. Indeed, the images show different differences that need to be recorded as image recognition of bird &amp; snake species. It is also easier for people to identify animals in the pictures. Identification of animal species is a challenging task often resulting in ambiguous labels. Even professional Wildlife watchers sometimes disagree on the species given an image of animals.

Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey

Using state-of-the-art deep learning (DL) models to diagnose cancer from histology data presents several challenges related to the nature and availability of labeled histology images, including image size, stain variations, and label ambiguity. In addition, cancer grading and the localization of regions of interest (ROIs) in such images normally rely on both image- and pixel-level labels, with the latter requiring a costly annotation process. Deep weakly-supervised object localization (WSOL) methods provide different strategies for low-cost training of DL models. Given only image-class annotations, these methods can be trained to simultaneously classify an image, and yield class activation maps (CAMs) for ROI localization. This paper provides a review of deep WSOL methods to identify and locate diseases in histology images, without the need for pixel-level annotations. We propose a taxonomy in which these methods are divided into bottom-up and top-down methods according to the information flow in models. Although the latter have seen only limited progress, recent bottom-up methods are currently driving a lot of progress with the use of deep WSOL methods. Early works focused on designing different spatial pooling functions. However, those methods quickly peaked in term of localization accuracy and revealed a major limitation, namely, – the under-activation of CAMs, which leads to high false negative localization. Subsequent works aimed to alleviate this shortcoming and recover the complete object from the background, using different techniques such as perturbation, self-attention, shallow features, pseudo-annotation, and task decoupling.&lt;br&gt;In the present paper, representative deep WSOL methods from our taxonomy are also evaluated and compared in terms of classification and localization accuracy using two challenging public histology datasets – one for colon cancer (GlaS), and a second, for breast cancer (CAMELYON16). Overall, the results indicate poor localization performance, particularly for generic methods that were initially designed to process natural images. Methods designed to address the challenges posed by histology data often use priors such as ROI size, or additional pixel-wise supervision estimated from a pre-trained classifier, allowing them to achieve better results. However, all the methods suffer from high false positive/negative localization. Classification performance is mainly affected by the model selection process, which uses either the classification or the localization metric. Finally, four key challenges are identified in the application of deep WSOL methods in histology, namely, – under-/over-activation of CAMs, sensitivity to thresholding, and model selection – and research avenues are provided to mitigate them. Our code is publicly available at &lt;a href='https://github.com/jeromerony/survey_wsl_histology'&gt;https://github.com/jeromerony/survey_wsl_histology&lt;/a&gt;

A hybrid deep belief network-based label distribution learning system for seismic damage estimation of liquid storage tanks

Liquid storage tanks play a vital role in the modern chemical process industry (CPI). The strong ground motion caused by large-scale earthquakes may easily impose severe structural damage on liquid storage tanks, leading to a series of catastrophic cascaded events. The seismic damage estimation of liquid storage tanks is a challenging problem, as the fluid-structure interaction exhibits extremely complicated and non-stationary response behavior. This study develops a novel data-driven methodology to estimate the seismic damage state probability distribution of liquid storage tanks in the contexts of label ambiguity and data imbalance. With the support of the advanced deep learning framework, synthetic oversampling methods, and label enhancement techniques, a hybrid deep belief network-based label distribution learning system (HDBN-LDLS) is proposed for probability distribution learning. The proposed HDBN-LDLS is evaluated on the widely used ALA database. Simulation results indicate that HDBN-LDLS can achieve a balanced estimation for all damage states while maintaining sufficient robustness to cope with label ambiguity. The reliability of the obtained data-driven model is validated by a damaged tank in the 2006 Silakhor earthquake. For practical applications, a more natural way to estimate a seismic damaged tank is to assign a membership degree to each possible damage state. The proposed methodology can quickly obtain the seismic damage state probability curves of a specific liquid storage tank, which can be used to support quantitative risk assessment and seismic design.

Convolutional Multiple Instance Learning for Sleep Spindle Detection With Label Refinement

Sleep spindles are closely associated with cognitive functions and neurological disorders; thus, spindle detection has been an important topic in sleep medicine. Recently, machine learning approaches have shown the potential in automatic sleep spindle detection by learning optimized features in a data-driven way, while they highly rely on labeled data, and the performance can be degraded when labels are inaccurate. However, accurate annotation of the spindle is usually difficult to obtain and high intraexpert and interexpert variance exist. In this work, we propose a convolutional neural network (CNN) with a label refinement component to learn an effective spindle detector with imperfect labels. Our approach consists of two stages: 1) a feature learning stage and 2) a label refinement stage. In the feature learning stage, a CNN-based multiple instance learning framework (CNN-MIL) is built for spindle feature learning. By assuming only parts of each labeled spindle segment contain true spindle patterns, the CNN-MIL model can learn most-likely spindle-related features from ambiguous labels. In the label refinement stage, we adjust the spindle labels by merging the original labels and labels predicted by CNN-MIL, and the modified labels are then used in the next round CNN-MIL feature learning. The two stages perform alternately for detector optimization. Extensive experiments demonstrated that our approach achieved the state-of-the-art performance.

Prior Knowledge Constrained Adaptive Graph Framework for Partial Label Learning

Partial label learning (PLL) aims to learn a robust multi-class classifier from the ambiguous data, where each instance is given with several candidate labels, among which only one label is real. Most existing methods usually cope with such problem by utilizing a feature similarity graph to conduct label disambiguation. However, these methods construct the feature graph by only employing original features, while the influences of latent outliers and the contributions of label space are regrettably ignored. To tackle these issues, in this article, we propose a P rior Kn O wledge Cons T rained A daptive G raph Fram E work ( POTAGE ) for partial label learning, which utilizes an adaptive graph fused with label information to accurately describe the instance relationship and guide the desired model training. Compared with the feature-induced fixed graph, the adaptive graph is deemed to be more robust and accurate to reveal the intrinsic manifold structure within the data, and the embedding label information is expected to effectively alleviate the label ambiguities and enlarge the gap of label confidences between two instances from different classes. Extensive experiments demonstrate that POTAGE achieves state-of-the-art performance.

Disambiguation-based partial label feature selection via feature dependency and label consistency

Partial label learning refers to the issue that each training sample corresponds to a candidate label set containing only one valid label. Feature selection can be viewed as an effective pre-processing technology to improve the generalization performance of learning models, while the partial label feature selection task is challenging due to ambiguous labeling information. To this end, this paper utilizes granular ball computing and neighborhood rough sets to put forward a disambiguation-based partial label feature selection algorithm via feature dependency and label consistency. Firstly, the proposed algorithm performs an adaptive neighbor aggregation based on granular balls to disambiguate candidate labels. Adaptive neighbors are flexible to make more efficient learning performance. Then, considering the labeling confidence induced by disambiguation, the significance of each feature is evaluated by fusing the feature dependency from the neighborhood granularity and the label consistency among the nearest neighbors. Neighborhood rough sets directly handle continuous features that can reduce the adverse impact of data discretization on feature selection. The label consistency among adjacent samples is important for measuring feature discrimination. The fusion of feature dependency and label consistency can facilitate performance improvement. Finally, experiments conducted on eight controlled UCI and five real-world partial label datasets demonstrate that the proposed algorithm can improve the generalization performance of partial label learning and achieve superior performance compared to the partial label feature selection methods.

PecidRL: Petition expectation correction and identification based on deep reinforcement learning

Identifying petition expectation for government response plays an important role in government administrative service. Although some petition platforms allow citizens to label the petition expectation when they submit e-petitions, the misunderstanding and misselection of petition labels still has necessitated manual classification involved. Automatic petition expectation identification has faced challenges in poor context information, heavy noise and casual syntactic structure of the petition text. In this paper we propose a novel deep reinforcement learning based method for petition expectation (citizens’ demands for the level of government response) correction and identification named PecidRL. We collect a dataset from Message Board for Leaders, the largest official petition platform in China, containing 237,042 petitions. Firstly, we introduce a deep reinforcement learning framework to automatically correct the mislabeled and ambiguous labels of the petitions. Then, multi-view textual features, including word-level and document-level semantic features, sentiment features and different textual graph representations are extracted and integrated to enrich more auxiliary information. Furthermore, based on the corrected petitions, 19 novel petition expectation identification models are constructed by extending 11 popular machine learning models for petition expectation detection. Finally, comprehensive comparison and evaluation are conducted to select the final petition expectation identification model with the best performance. After performing correction by PecidRL, each metric on all extended petition expectation identification models improves by an average of 8.3% with the highest increase ratio reaching 14.2%. The optimal model is determined as Peti-SVM-bert with the highest accuracy 93.66%. We also analyze the petition expectation label variation of the dataset by using PecidRL. We derive that 16.9% of e-petitioners tend to exaggerate the urgency of their petitions to make the government pay high attention to their appeals and 4.4% of the petitions urgency are underestimated. This study has substantial academic and practical value in improving government efficiency. Additionally, a web-server is developed to facilitate government administrators and other researchers, which can be accessed at http://www.csbg-jlu.info/PecidRL/.

Cross-modal dynamic sentiment annotation for speech sentiment analysis

Traditionally, one single hard label determines the sentiment label of an entire utterance for speech sentiment analysis. It obviously ignores the inherent dynamic and ambiguity of speech sentiments. Moreover, there are few segment-level ground truth labels in the most existing sentiment corpora, due to the label ambiguity and annotation cost. In this work, to capture segment-level sentiment fluctuations across one utterance, we propose sentiment profiles (SPs) to express segment-level soft labels. Meanwhile, we introduce massive multi-modal wild video data to solve the data shortage problem, and facial expression knowledge is used to guide audio segments generate soft labels through the Cross-modal Sentiment Annotation Module. Then, we design a Speech Encoder Module to encode audio segments into SPs. We further exploit the sentiment profile purifier (SPP) to iteratively improve the accuracy of SPs. Numerous experiments show that our model achieves state-of-the-art performance on CH-SIMS and IEMOCAP datasets with unlabeled data respectively.