Data Labeling Process Research Articles

Development of a machine learning model for predicting abnormalities of commercial airplanes

Airplanes are a social necessity for movement of humans, goods, and other. They are generally safe modes of transportation; however, incidents and accidents occasionally occur. To prevent aviation accidents, it is necessary to develop a machine-learning model to detect and predict commercial flights using automatic dependent surveillance–broadcast data. This study combined data-quality detection, anomaly detection, and abnormality-classification-model development. The research methodology involved the following stages: problem statement, data selection and labeling, prediction-model development, deployment, and testing. The data labeling process was based on the rules framed by the international civil aviation organization for commercial, jet-engine flights and validated by expert commercial pilots. The results showed that the best prediction model, the quadratic-discriminant-analysis, was 93% accurate, indicating a “good fit”. Moreover, the model’s area-under-the-curve results for abnormal and normal detection were 0.97 and 0.96, respectively, thus confirming its “good fit”.

Small is Beautiful: Compressing Deep Neural Networks for Partial Domain Adaptation.

Domain adaptation is a promising way to ease the costly data labeling process in the era of deep learning (DL). A practical situation is partial domain adaptation (PDA), where the label space of the target domain is a subset of that in the source domain. Although existing methods yield appealing performance in PDA tasks, it is highly presumable that computation overhead exists in deep PDA models since the target is only a subtask of the original problem. In this work, PDA and model compression are seamlessly integrated into a unified training process. The cross-domain distribution divergence is reduced by minimizing a soft-weighted maximum mean discrepancy (SWMMD), which is differentiable and functions as regularization during network training. We use gradient statistics to compress the overparameterized model to identify and prune redundant channels based on the corresponding scaling factors in batch normalization (BN) layers. The experimental results demonstrate that our method can achieve comparable classification performance to state-of-the-art methods on various PDA tasks, with a significant reduction in model size and computation overhead.

ASSL-HGAT: Active semi-supervised learning empowered heterogeneous graph attention network

Recently, heterogeneous graph attention network (HGAT) has been widely applied to various machine learning tasks and achieved remarkable results with sufficient labeled data. However, it is noteworthy that in many tasks, labeled data is scarce and the data labeling process is expensive. To that end, this paper presents a novel framework for learning from data with limited labels by organically integrating active learning (AL) and semi-supervised learning (SSL) into heterogeneous graph network. Our framework consists of three components: heterogeneous information network (HIN), HGAT and active semi-supervised learning strategy (ASSL). The adjustable HIN is first constructed by fusing multi-modal features. Then the HGAT is used to encode HIN and the double-layer attention mechanism can mitigate noises during information fusion. The ASSL is further designed to enrich the high-quality labeled training data for model train. Samples selected by uncertainty-aware AL and samples labeled by pseudo-label selection based SSL are finally mixed to iteratively train the ASSL-HGAT framework. Experimental results prove that ASSL-HGAT surpasses the compared state-of-the-art methods on five representative semi-supervised benchmark datasets under the same experimental settings.

Multiple-instance Learning from Triplet Comparison Bags

Multiple-instance learning (MIL) solves the problem where training instances are grouped in bags, and a binary (positive or negative) label is provided for each bag. Most of the existing MIL studies need fully labeled bags for training an effective classifier, while it could be quite hard to collect such data in many real-world scenarios, due to the high cost of data labeling process. Fortunately, unlike fully labeled data, triplet comparison data can be collected in a more accurate and human-friendly way. Therefore, in this article, we for the first time investigate MIL from only triplet comparison bags , where a triplet (X a , X b , X c ) contains the weak supervision information that bag X a is more similar to X b than to X c . To solve this problem, we propose to train a bag-level classifier by the empirical risk minimization framework and theoretically provide a generalization error bound. We also show that a convex formulation can be obtained only when specific convex binary losses such as the square loss and the double hinge loss are used. Extensive experiments validate that our proposed method significantly outperforms other baselines.

DomAda-FruitDet: Domain-Adaptive Anchor-Free Fruit Detection Model for Auto Labeling.

Recently, deep learning-based fruit detection applications have been widely used in the modern fruit industry; however, the training data labeling process remains a time-consuming and labor-intensive process. Auto labeling can provide a convenient and efficient data source for constructing smart orchards based on deep-learning technology. In our previous study, based on a labeled source domain fruit dataset, we used a generative adversarial network and a fruit detection model to achieve auto labeling of unlabeled target domain fruit images. However, since the current method uses one species source domain fruit to label multiple species target domain fruits, there is a problem of the domain gap in both the foreground and the background between the training data (retaining the source domain fruit label information) and the application data (target domain fruit images) of the fruit detection model. Therefore, we propose a domain-adaptive anchor-free fruit detection model, DomAda-FruitDet, and apply it to the previously proposed fruit labeling method to further improve the accuracy. It consists of 2 design aspects: (a) With a foreground domain-adaptive structure based on double prediction layers, an anchor-free method with multiscale detection capability is constructed to generate adaptive bounding boxes that overcome the foreground domain gap; (b) with a background domain-adaptive strategy based on sample allocation, we enhance the ability of the model to extract foreground object features to overcome the background domain gap. As a result, the proposed method can label actual apple, tomato, pitaya, and mango datasets, with an average precision of 90.9%, 90.8%, 88.3%, and 94.0%, respectively. In conclusion, the proposed DomAda-FruitDet effectively addressed the problem of the domain gap and improved effective auto labeling for fruit detection tasks.

Data-Centric AI for Healthcare Fraud Detection.

Automated methods for detecting fraudulent healthcare providers have the potential to save billions of dollars in healthcare costs and improve the overall quality of patient care. This study presents a data-centric approach to improve healthcare fraud classification performance and reliability using Medicare claims data. Publicly available data from the Centers for Medicare & Medicaid Services (CMS) are used to construct nine large-scale labeled data sets for supervised learning. First, we leverage CMS data to curate the 2013-2019 Part B, Part D, and Durable Medical Equipment, Prosthetics, Orthotics, and Supplies (DMEPOS) Medicare fraud classification data sets. We provide a review of each data set and data preparation techniques to create Medicare data sets for supervised learning and we propose an improved data labeling process. Next, we enrich the original Medicare fraud data sets with up to 58 new provider summary features. Finally, we address a common model evaluation pitfall and propose an adjusted cross-validation technique that mitigates target leakage to provide reliable evaluation results. Each data set is evaluated on the Medicare fraud classification task using extreme gradient boosting and random forest learners, multiple complementary performance metrics, and 95% confidence intervals. Results show that the new enriched data sets consistently outperform the original Medicare data sets that are currently used in related works. Our results encourage the data-centric machine learning workflow and provide a strong foundation for data understanding and preparation techniques for machine learning applications in healthcare fraud.

SMPT: A Semi-Supervised Multi-Model Prediction Technique for Food Ingredient Named Entity Recognition (FINER) Dataset Construction

To pursue a healthy lifestyle, people are increasingly concerned about their food ingredients. Recently, it has become a common practice to use an online recipe to select the ingredients that match an individual’s meal plan and healthy diet preference. The information from online recipes can be extracted and used to develop various food-related applications. Named entity recognition (NER) is often used to extract such information. However, the problem in building an NER system lies in the massive amount of data needed to train the classifier, especially on a specific domain, such as food. There are food NER datasets available, but they are still quite limited. Thus, we proposed an iterative self-training approach called semi-supervised multi-model prediction technique (SMPT) to construct a food ingredient NER dataset. SMPT is a deep ensemble learning model that employs the concept of self-training and uses multiple pre-trained language models in the iterative data labeling process, with a voting mechanism used as the final decision to determine the entity’s label. Utilizing the SMPT, we have created a new annotated dataset of ingredient entities obtained from the Allrecipes website named FINER. Finally, this study aims to use the FINER dataset as an alternative resource to support food computing research and development.

Data Poisoning Attacks in Internet-of-Vehicle Networks: Taxonomy, State-of-The-Art, and Future Directions

With the unprecedented development of deep learning, autonomous vehicles (AVs) have achieved tremendous progress nowadays. However, AV supported by DNN models is vulnerable to data poisoning attacks, hindering the large-scale application of autonomous driving. For example, by injecting carefully designed poisons into the training dataset of the DNN model in the traffic sign recognition system, the attacker can mislead the system to make targeted misclassification or cause a reduction in model classification accuracy. In this article, we conduct a thorough investigation of the state-of-the-art data poisoning attacks and defenses against AVs. According to whether the attacker needs to manipulate the data labeling process, we divide the state-of-the-art attack approaches into two categories, i.e., dirty-label attacks and clean-label attacks. We also differentiate the existing defense methods into two categories based on whether to modify the training data or the models, i.e., data-based defenses and model-based defenses. In addition to a detailed review of attacks and defenses in each category, we also give a qualitative comparison of the existing attacks and defenses. Besides, we provide a quantitative comparison of the existing attack and defense methods through experiments. Last but not least, we pinpoint several future directions for data poisoning attacks and defenses in AVs, providing possible ways for further research.

Post-Stroke Gait Classification Based on Feature Space Transformation and Data Labeling

Despite scientific and clinical advances, stroke is still considered one of the main causes of disability, including gait disorders. The search for more effective methods of gait re-education in post-stroke patients is one of the most important issues in contemporary neurorehabilitation. In this paper, we propose a transformation of the feature space and definition of class labels in the post-stroke gait problem to more efficiently study related phenomena and assess gait faster. Clustering is used to define two class labels (improvement and recurrence) in the data labeling process. The proposed approach was tested on a real-world dataset consisting of 50 patients (male and female, aged 49–82 years) after ischemic stroke who participated in a gait rehabilitation program. Gait in the study was described using speed, cadence, and stride length and their normalized values. Ten treatment sessions (10 therapy days) were conducted over two weeks (10 working days). The same specialist took measurements, and hence inter-rater reliability can be neglected. Machine learning methods, support vector machine and quadratic discriminant analysis were used to classify post-stroke gait for three cases with different class labels. The proposed novel approach, characterized by its speed of execution and accuracy of classification, may be helpful for screening, better targeting, and rehabilitation monitoring. The proposed approach minimizes clinical testing and supports the work of physicians, physiotherapists, and diagnosticians.

PLENARY: Explaining black-box models in natural language through fuzzy linguistic summaries

We introduce an approach called PLENARY (exPlaining bLack-box modEls in Natural lAnguage thRough fuzzY linguistic summaries), which is an explainable classifier based on a data-driven predictive model. Neural learning is exploited to derive a predictive model based on two levels of labels associated with the data. Then, model explanations are derived through the popular SHapley Additive exPlanations (SHAP) tool and conveyed in a linguistic form via fuzzy linguistic summaries. The linguistic summarization allows translating the explanations of the model outputs provided by SHAP into statements expressed in natural language. PLENARY accounts for the imprecision related to model outputs by summarizing them into simple linguistic statements and for the imprecision related to the data labeling process by including additional domain knowledge in the form of middle-layer labels. PLENARY is validated on preprocessed speech signals collected from smartphones from patients with bipolar disorder and on publicly available mental health survey data. The experiments confirm that fuzzy linguistic summarization is an effective technique to support meta-analyses of the outputs of AI models. Also, PLENARY improves explainability by aggregating low-level attributes into high-level information granules, and by incorporating vague domain knowledge into a multi-task sequential and compositional multilayer perceptron. SHAP explanations translated into fuzzy linguistic summaries significantly improve understanding of the predictive modelling process and its outputs.

Brain-inspired multiple-target tracking using Dynamic Neural Fields

Despite considerable progress in the field of automatic multi-target tracking, several problems such as data association remained challenging. On the other hand, cognitive studies have reported that humans can robustly track several objects simultaneously. Such circumstances happen regularly in daily life, and humans have evolved to handle the associated problems. Accordingly, using brain-inspired processing principles may contribute to significantly increase the performance of automatic systems able to follow the trajectories of multiple objects. In this paper, we propose a multiple-object tracking algorithm based on dynamic neural field theory which has been proven to provide neuro-plausible processing mechanisms for cognitive functions of the brain. We define several input neural fields responsible for representing previous location and orientation information as well as instantaneous linear and angular speed of the objects in successive video frames. Image processing techniques are applied to extract the critical object features including target location and orientation. Two prediction fields anticipate the objects’ locations and orientations in the upcoming frame after receiving excitatory and inhibitory inputs from the input fields in a feed-forward architecture. This information is used in the data association and labeling process. We tested the proposed algorithm on a zebrafish larvae segmentation and tracking dataset and an ant-tracking dataset containing non-rigid objects with spiky movements and frequently occurring occlusions. The results showed a significant improvement in tracking metrics compared to state-of-the-art algorithms.

Stacking Neural Network Models for Automatic Short Answer Scoring

Automatic short answer scoring is one of the text classification problems to assess students’ answers during exams automatically. Several challenges can arise in making an automatic short answer scoring system, one of which is the quantity and quality of the data. The data labeling process is not easy because it requires a human annotator who is an expert in their field. Further, the data imbalance process is also a challenge because the number of labels for correct answers is always much less than the wrong answers. In this paper, we propose the use of a stacking model based on neural network and XGBoost for classification process with sentence embedding feature. We also propose to use data upsampling method to handle imbalance classes and hyperparameters optimization algorithm to find a robust model automatically. We use Ukara 1.0 Challenge dataset and our best model obtained an F1-score of 0.821 exceeding the previous work at the same dataset.

Uncertainty Inspired RGB-D Saliency Detection.

We propose the first stochastic framework to employ uncertainty for RGB-D saliency detection by learning from the data labeling process. Existing RGB-D saliency detection models treat this task as a point estimation problem by predicting a single saliency map following a deterministic learning pipeline. We argue that, however, the deterministic solution is relatively ill-posed. Inspired by the saliency data labeling process, we propose a generative architecture to achieve probabilistic RGB-D saliency detection which utilizes a latent variable to model the labeling variations. Our framework includes two main models: 1) a generator model, which maps the input image and latent variable to stochastic saliency prediction, and 2) an inference model, which gradually updates the latent variable by sampling it from the true or approximate posterior distribution. The generator model is an encoder-decoder saliency network. To infer the latent variable, we introduce two different solutions: i) a Conditional Variational Auto-encoder with an extra encoder to approximate the posterior distribution of the latent variable; and ii) an Alternating Back-Propagation technique, which directly samples the latent variable from the true posterior distribution. Qualitative and quantitative results on six challenging RGB-D benchmark datasets show our approach's superior performance in learning the distribution of saliency maps. The source code is publicly available via our project page: https://github.com/JingZhang617/UCNet.

Batch Prioritization of Data Labeling Tasks for Training Classifiers

In a data labeling process for building machine learning, the choice of labeling data instances is known to have a significant impact on the performance of classifiers. So far, the study of active learning has addressed the issue of how to choose the subset by prioritizing the data instances based on the state of the current classifier. However, the active learning approach has two drawbacks that (i) require a training loop to update the priorities of labeling tasks and (ii) require us to choose a specific active learner while we do not know the optimal classification model. In this paper, we propose a new framework of priority-aware labeling system that allows a parallel task assignment to crowd workers without assuming a particular classifier, which is based on novel methods called “batch prioritization” and “label expansion”. We conducted experiments with multiple datasets to examine the effectiveness of the approach and found that the proposed method improves the performance of the final classifiers more quickly than the active learning approach despite that the labeling tasks can be processed in a fully parallel manner.

DCBT-Net: Training Deep Convolutional Neural Networks With Extremely Noisy Labels

Obtaining data with correct labels is crucial to attain the state-of-the-art performance of Convolutional Neural Network (CNN) models. However, labeling datasets is significantly time-consuming and expensive process because it requires expert knowledge in a particular domain. Therefore, real-life datasets often exhibit incorrect labels due to the involvement of nonexperts in the data-labeling process. Consequently, there are many cases of incorrectly labeled data in the wild. Although the issue of poorly labeled datasets has been studied, the existing methods are complex and difficult to reproduce. Thus, in this study, we proposed a simpler algorithm called “Deep Clean Before Training Net” (DCBT-Net) that is based on cleaning wrongly labeled data points using the information from eigenvalues of the Laplacian matrix obtained from similarities between the data samples. The cleaned data were trained using deep CNN (DCNN) to attain the state-of-the-art results. This system achieved better performance than the existing approaches. In conducted experiments, the performance of the DCBT-Net was tested on three commercially available datasets, namely, Modified National Institute of Standards and Technology (MNIST) database of handwritten digits, Canadian Institute for Advanced Research (CIFAR) and WebVision1000 datasets. The proposed method achieved better results when assessed using several evaluation metrics compared with the existing state-of-the-art methods. Specifically, the DCBT-Net attained an average 15%, 20%, and 3% increase in accuracy score using MNIST database, CIFAR-10 dataset, and WebVision dataset, respectively. Also, the proposed approach demonstrated better results in specificity, sensitivity, positive predictive value, and negative predictive value evaluation metrics.

Regularized Instance Embedding for Deep Multi-Instance Learning

In the era of Big Data, multi-instance learning, as a weakly supervised learning framework, has various applications since it is helpful to reduce the cost of the data-labeling process. Due to this weakly supervised setting, learning effective instance representation/embedding is challenging. To address this issue, we propose an instance-embedding regularizer that can boost the performance of both instance- and bag-embedding learning in a unified fashion. Specifically, the crux of the instance-embedding regularizer is to maximize correlation between instance-embedding and underlying instance-label similarities. The embedding-learning framework was implemented using a neural network and optimized in an end-to-end manner using stochastic gradient descent. In experiments, various applications were studied, and the results show that the proposed instance-embedding-regularization method is highly effective, having state-of-the-art performance.

P.788 Associative learning and resting-state connectivity changes in the adult brain

Label distribution learning (LDL) is one of the paradigms for dealing with label ambiguity, and it can learn the relative importance of each label to a particular instance. Most of the existing LDL approaches require strong supervision information, which always involves high cost of the data-labeling process. To this end, we propose a novel transductive weakly supervised LDL algorithm based on matrix completion, which simultaneously explores the discriminant knowledge of both training instances and testing instances. Moreover, the manifold regularization is exploited to capture the samples’ relevance to enhance the performance of matrix completion. Experimental results on real-world data sets with various missing percentages validate the effectiveness of our method.

A Novel Active Learning Regression Framework for Balancing the Exploration-Exploitation Trade-Off.

Recently, active learning is considered a promising approach for data acquisition due to the significant cost of the data labeling process in many real world applications, such as natural language processing and image processing. Most active learning methods are merely designed to enhance the learning model accuracy. However, the model accuracy may not be the primary goal and there could be other domain-specific objectives to be optimized. In this work, we develop a novel active learning framework that aims to solve a general class of optimization problems. The proposed framework mainly targets the optimization problems exposed to the exploration-exploitation trade-off. The active learning framework is comprehensive, it includes exploration-based, exploitation-based and balancing strategies that seek to achieve the balance between exploration and exploitation. The paper mainly considers regression tasks, as they are under-researched in the active learning field compared to classification tasks. Furthermore, in this work, we investigate the different active querying approaches—pool-based and the query synthesis—and compare them. We apply the proposed framework to the problem of learning the price-demand function, an application that is important in optimal product pricing and dynamic (or time-varying) pricing. In our experiments, we provide a comparative study including the proposed framework strategies and some other baselines. The accomplished results demonstrate a significant performance for the proposed methods.

Weakly supervised label distribution learning based on transductive matrix completion with sample correlations

Label distribution learning (LDL) is one of the paradigms for dealing with label ambiguity, and it can learn the relative importance of each label to a particular instance. Most of the existing LDL approaches require strong supervision information, which always involves high cost of the data-labeling process. To this end, we propose a novel transductive weakly supervised LDL algorithm based on matrix completion, which simultaneously explores the discriminant knowledge of both training instances and testing instances. Moreover, the manifold regularization is exploited to capture the samples’ relevance to enhance the performance of matrix completion. Experimental results on real-world data sets with various missing percentages validate the effectiveness of our method.

Automotive radar and camera fusion using Generative Adversarial Networks

Radar sensors are considered to be very robust under harsh weather and poor lighting conditions. Largely owing to this reputation, they have found broad application in driver assistance and highly automated driving systems. However, radar sensors have considerably lower precision than cameras. Low sensor precision causes ambiguity in the human interpretation of the measurement data and makes the data labeling process difficult and expensive. On the other hand, without a large amount of high-quality labeled training data, it is difficult, if not impossible, to ensure that the supervised machine learning models can predict, classify, or otherwise analyze the phenomenon of interest with the required accuracy. This paper presents a method for fusing the radar sensor measurements with the camera images. A proposed fully-unsupervised machine learning algorithm converts the radar sensor data to artificial, camera-like, environmental images. Through such data fusion, the algorithm produces more consistent, accurate, and useful information than that provided solely by the radar or the camera. The essential point of the work is the proposal of a novel Conditional Multi-Generator Generative Adversarial Network (CMGGAN) that, being conditioned on the radar sensor measurements, can produce visually appealing images that qualitatively and quantitatively contain all environment features detected by the radar sensor.