Correct Label Research Articles

26-LB: Identifying the Risk of Diabetic Retinopathy Progression Using Machine Learning on Ultrawide Field Retinal Images

Estimating the risk of diabetic retinopathy (DR) progression is one of the most important and difficult tasks when caring for individuals with diabetic eye disease. Current DR severity scales have informed clinicians on the progression risk and provided recommendations for follow-up and treatment. The use of artificial intelligence (AI) algorithms may improve in this process. In this study, we developed and validated machine learning (ML) models for DR progression from ultrawide field (UWF) retinal images, which were labeled for baseline DR severity and progression based on clinician review of the images and 3-year longitudinal follow-up using the Early Treatment Diabetic Retinopathy Study (ETDRS) severity scale. This dataset has 8 classes: no DR nonprogression (14.62%), Mild nonproliferative DR (NPDR) progression (10.16%) /nonprogression (10.73%), Moderate NPDR progression (10.1%) /nonprogression (15.85%), Severe NPDR progression (11.27%) /nonprogression (10.68%), and proliferative DR (16.55%). A total of 9970 unique images were split into the train, the validation, and the test datasets based on 60-20-20 proportions. The class imbalance was addressed during model building through data augmentation. The ResNet model fine-tuned on this dataset has a classification test accuracy of 81% and an AUC of 0.967 on the test dataset. The objective of the model is to reduce false negatives, which refers to predicting a class that is less progressive than the true label. The predicted labels for 91% of the images were either correct labels or were the labels with greater progression than the original labels. These findings demonstrate the accuracy and feasibility of using machine learning models for identifying DR progression developed using UWF images. Potentially, the use of machine learning algorithms may further refine the risk of disease progression and personalize screening intervals that may reduce costs and improve vision-related outcomes. Disclosure A. Nigam: None. J. Sun: None. V. Subhash: None. P. S. Silva: Research Support; Optos plc., Optomed, Speaker's Bureau; Novartis, Roche Pharmaceuticals, Bayer Inc.

Exploratory Training: When Annonators Learn About Data

Data systems often present examples and solicit labels from users to learn a target model, i.e., active learning. However, due to the complexity of the underlying data, users may not initially have a perfect understanding of the effective model and do not know the accurate labeling. For example, a user who is training a model for detecting noisy or abnormal values may not perfectly know the properties of typical and clean values in the data. Users may improve their knowledge about the data and target model as they observe examples during training. As users gradually learn about the data and model, they may revise their labeling strategies. Current systems assume that users always provide correct labeling with potentially a fixed and small chance of annotation mistakes. Nonetheless, if the trainer revises its belief during training, such mistakes become significant and non-stationarity. Hence, current systems consume incorrect labels and may learn inaccurate models. In this paper, we build theoretical underpinnings and design algorithms to develop systems that collaborate with users to learn the target model accurately and efficiently. At the core of our proposal, a game-theoretic framework models the joint learning of user and system to reach a desirable eventual stable state, where both user and system share the same belief about the target model. We extensively evaluate our system using user studies over various real-world datasets and show that our algorithms lead to accurate results with a smaller number of interactions compared to existing methods.

A semi-automatic workflow to process images from small mammal camera traps

Camera traps have become popular for monitoring biodiversity, but the huge amounts of image data that arise from camera trap monitoring represent a challenge and artificial intelligence is increasingly used to automatically classify large image data sets. However, it is still challenging to combine automatic classification with other steps and tools needed for efficient, quality-assured and adaptive processing of camera trap images in long-term monitoring programs. Here we propose a semi-automatic workflow to process images from small mammal cameras that combines all necessary steps from downloading camera trap images in the field to a quality checked data set ready to be used in ecological analyses. The workflow is implemented in R and includes (1) managing raw images, (2) automatic image classification, (3) quality check of automatic image labels, as well as the possibilities to (4) retrain the model with new images and to (5) manually review subsets of images to correct image labels. We illustrate the application of this workflow for the development of a new monitoring program of an Arctic small mammal community. We first trained a classification model for the specific small mammal community based on images from an initial set of camera traps. As the monitoring program evolved, the classification model was retrained with a small subset of images from new camera traps. This case study highlights the importance of model retraining in adaptive monitoring programs based on camera traps as this step in the workflow increases model performance and substantially decreases the total time needed for manually reviewing images and correcting image labels. We provide all R scripts to make the workflow accessible to other ecologists.

FedDM: Federated Weakly Supervised Segmentation via Annotation Calibration and Gradient De-Conflicting.

Weakly supervised segmentation (WSS) aims to exploit weak forms of annotations to achieve the segmentation training, thereby reducing the burden on annotation. However, existing methods rely on large-scale centralized datasets, which are difficult to construct due to privacy concerns on medical data. Federated learning (FL) provides a cross-site training paradigm and shows great potential to address this problem. In this work, we represent the first effort to formulate federated weakly supervised segmentation (FedWSS) and propose a novel Federated Drift Mitigation (FedDM) framework to learn segmentation models across multiple sites without sharing their raw data. FedDM is devoted to solving two main challenges (i.e., local drift on client-side optimization and global drift on server-side aggregation) caused by weak supervision signals in FL setting via Collaborative Annotation Calibration (CAC) and Hierarchical Gradient De-conflicting (HGD). To mitigate the local drift, CAC customizes a distal peer and a proximal peer for each client via a Monte Carlo sampling strategy, and then employs inter-client knowledge agreement and disagreement to recognize clean labels and correct noisy labels, respectively. Moreover, in order to alleviate the global drift, HGD online builds a client hierarchy under the guidance of history gradient of the global model in each communication round. Through de-conflicting clients under the same parent nodes from bottom layers to top layers, HGD achieves robust gradient aggregation at the server side. Furthermore, we theoretically analyze FedDM and conduct extensive experiments on public datasets. The experimental results demonstrate the superior performance of our method compared with state-of-the-art approaches. The source code is available at https://github.com/CityU-AIM-Group/FedDM.

PreSoramimiset: Establishing dataset for Chinese misheard lyrics generation

Mondegreen is a common phenomenon during conversation and is more obvious during listening to songs. To simulate the impression that the audience will have when they hear a particular piece, the Soramiminet model is introduced in this paper. The model is a combination of wave2vec, and transformer used for generating misheard lyrics in Chinese with various input songs. This article will focus more on the establishment of the dataset for wave2vec model. The criteria and several methods of creating a high-quality dataset are summarized and present in this paper. Additionally, the negative impact of a defective dataset and how to avoid it is discussed. The main limitations and biases of this dataset, and how to address them and future work are explored. The dataset includes a set of audio clips from 21 tracks by 9 different singers in AVI form and a text file for annotation which can be processed by python dataset module. The dataset is relatively biased, since all the annotations are done by the author personally. There is no correct labelling, given that the model is generating misheard or wrong results.

Single-channel speech separation using soft-minimum permutation invariant training

The goal of speech separation is to extract multiple speech sources from a single microphone recording. Recently, with the advancement of deep learning and availability of large datasets, speech separation has been formulated as a supervised learning problem. These approaches aim to learn discriminative patterns of speech, speakers, and background noise using a supervised learning algorithm, typically a deep neural network. A long-lasting problem in supervised speech separation is finding the correct label for each separated speech signal, referred to as label permutation ambiguity. Permutation ambiguity refers to the problem of determining the output-label assignment between the separated sources and the available single-speaker speech labels. Finding the best output-label assignment is required for calculation of separation error, which is later used for updating parameters of the model. Recently, Permutation Invariant Training (PIT) has been shown to be a promising solution in handling the label ambiguity problem. However, the overconfident choice of the output-label assignment by PIT results in a sub-optimal trained model. In this work, we propose a probabilistic optimization framework to address the inefficiency of PIT in finding the best output-label assignment. Our proposed method entitled trainable Soft-minimum PIT is then employed on the same Long-Short Term Memory (LSTM) architecture used in Permutation Invariant Training (PIT) speech separation method. The results of our experiments show that the proposed method outperforms conventional PIT speech separation significantly (p-value <0.01) by +1dB in Signal to Distortion Ratio (SDR) and +1.5dB in Signal to Interference Ratio (SIR).

Deep learning-based classification of organs at risk and delineation guideline in pelvic cancer radiation therapy.

Deep learning (DL) models for radiation therapy (RT) image segmentation require accurately annotated training data. Multiple organ delineation guidelines exist; however, information on the used guideline is not provided with the delineation. Extraction of training data with coherent guidelines can therefore be challenging. We present a supervised classification method for pelvis structure delineations where bowel cavity, femoral heads, bladder, and rectum data, with two guidelines, were classified. The impact on DL-based segmentation quality using mixed guideline training data was also demonstrated. Bowel cavity was manually delineated on CT images for anal cancer patients (n=170) according to guidelines Devisetty and RTOG. The DL segmentation quality from using training data with coherent or mixed guidelines was investigated. A supervised 3D squeeze-and-excite SENet-154 model was trained to classify two bowel cavity delineation guidelines. In addition, a pelvis CT dataset with manual delineations from prostate cancer patients (n=1854) was used where data with an alternative guideline for femoral heads, rectum, and bladder were generated using commercial software. The model was evaluated on internal (n=200) and external test data (n=99). By using mixed, compared to coherent, delineation guideline training data mean DICE score decreased 3% units, mean Hausdorff distance (95%) increased 5mm and mean surface distance (MSD) increased 1mm. The classification of bowel cavity test data achieved 99.8% unweighted classification accuracy, 99.9% macro average precision, 97.2% macro average recall, and 98.5% macro average F1. Corresponding metrics for the pelvis internal test data were all 99% or above and for the external pelvis test data they were 96.3%, 96.6%, 93.3%, and 94.6%. Impaired segmentation performance was observed for training data with mixed guidelines. The DL delineation classification models achieved excellent results on internal and external test data. This can facilitate automated guideline-specific data extraction while avoiding the need for consistent and correct structure labels.

Training Data Selection for Record Linkage Classification

This paper presents a new two-step approach for record linkage, focusing on the creation of high-quality training data in the first step. The approach employs the unsupervised random forest model as a similarity measure to produce a similarity score vector for record matching. Three constructions were proposed to select non-match pairs for the training data, with both balanced (symmetry) and imbalanced (asymmetry) distributions tested. The top and imbalanced construction was found to be the most effective in producing training data with 100% correct labels. Random forest and support vector machine classification algorithms were compared, and random forest with the top and imbalanced construction produced an F1-score comparable to probabilistic record linkage using the expectation maximisation algorithm and EpiLink. On average, the proposed approach using random forests and the top and imbalanced construction improved the F1-score by 1% and recall by 6.45% compared to existing record linkage methods. By emphasising the creation of high-quality training data, this new approach has the potential to improve the accuracy and efficiency of record linkage for a wide range of applications.

A spatial–spectral adaptive learning model for textile defect images recognition with few labeled data

Textile defect recognition is a significant technique in the production processes of the textile industry. However, in the practical processes, it is hard to acquire large amounts of textile defect samples. Meanwhile, the textile samples with correct defect labels are rare. To address these two limitations, in this paper, we propose a novel semi-supervised graph convolutional network for few labeled textile defect recognition. First, we construct the graph convolutional network and convolution neural network to extract spectral features and spatial features. Second, the adaptive convolution structure is proposed to generate adaptive kernels according to their dynamically learned features. Finally, the spatial–spectral adaptive unified learning network (SSA-ULNet) is built for limited labeled defective samples, and graph-based semi-supervised learning is constructed. The textile defect recognition model can extract the textile image features through the image descriptors, enabling the whole network to be end-to-end trainable. To evaluate the proposed method, one public dataset and two unique self-built textile defect datasets are used to textile defect recognition. The evaluation results demonstrate that the proposed SSA-ULNet obviously outperforms existing state-of-the-art deep learning methods.

SNOWED: Automatically Constructed Dataset of Satellite Imagery for Water Edge Measurements.

Monitoring the shoreline over time is essential to quickly identify and mitigate environmental issues such as coastal erosion. Monitoring using satellite images has two great advantages, i.e., global coverage and frequent measurement updates; but adequate methods are needed to extract shoreline information from such images. To this purpose, there are valuable non-supervised methods, but more recent research has concentrated on deep learning because of its greater potential in terms of generality, flexibility, and measurement accuracy, which, in contrast, derive from the information contained in large datasets of labeled samples. The first problem to solve, therefore, lies in obtaining large datasets suitable for this specific measurement problem, and this is a difficult task, typically requiring human analysis of a large number of images. In this article, we propose a technique to automatically create a dataset of labeled satellite images suitable for training machine learning models for shoreline detection. The method is based on the integration of data from satellite photos and data from certified, publicly accessible shoreline data. It involves several automatic processing steps, aimed at building the best possible dataset, with images including both sea and land regions, and correct labeling also in the presence of complicated water edges (which can be open or closed curves). The use of independently certified measurements for labeling the satellite images avoids the great work required to manually annotate them by visual inspection, as is done in other works in the literature. This is especially true when convoluted shorelines are considered. In addition, possible errors due to the subjective interpretation of satellite images are also eliminated. The method is developed and used specifically to build a new dataset of Sentinel-2 images, denoted SNOWED; but is applicable to different satellite images with trivial modifications. The accuracy of labels in SNOWED is directly determined by the uncertainty of the shoreline data used, which leads to sub-pixel errors in most cases. Furthermore, the quality of the SNOWED dataset is assessed through the visual comparison of a random sample of images and their corresponding labels, and its functionality is shown by training a neural model for sea-land segmentation.

AI-driven ensemble three machine learning to enhance digital marketing strategies in the food delivery business

PurposeThis study focuses on the use of ensemble machine learning (ML) in digital marketing for the food delivery business. MethodologyArtificial intelligence (AI) techniques are used to analyze customer data, identify customer preferences, and predict customer behavior to provide AI-based recommendations. The ensemble method combines the outputs of decision trees, naïve Bayes, and nearest neighbor algorithms to generate a single prediction. FindingsThe accuracy matrix plots for both the decision tree and nearest neighbor algorithms yielded perfect predictions, with an accuracy of 100.000% and 0.000 error, respectively. Meanwhile, the naïve Bayes algorithm had an overall accuracy matrix of 97.175%, with a 0.028 error, indicating successful identification of the correct labels across all classes with a high level of accuracy. OriginalityThe majority voting method with a probability success rate greater than 90% can potentially integrate models into this process while utilizing less than half the randomized data, blended with customer experience data, thus reducing customer irritation. The driven ensemble of three ML algorithms is shown to successfully improve digital marketing strategies in the food delivery business by decreasing time and costs.

Semi-Supervised Lexicon-Aware Embedding for News Article Time Estimation

In the information retrieval community, Temporal Information Retrieval (TIR) has become increasingly popular. Documents focused on the time surrounding their publication are more likely to be accurate and contain information relevant to the reader. In this study, we explore the inverted pyramid paradigm by extracting temporal expressions from news documents, standardizing their values, and evaluating them based on their position within the text. We present a lexicon expansion method that employs WordNet as input. This approach enhances the lexicon by grouping words with similar meanings, potentially improving the accuracy of event detection algorithms. Additionally, this process can introduce new words and phrases to the lexicon, expanding the vocabulary. Using each tagged dataset, a classifier is trained with a pre-trained network. A pool of unlabeled data are processed, and high-confidence pseudo-labels are assigned. Pseudo-labels are generated by leveraging the partially trained model and the original labelled data. As the classifier predicts the correct label for a data sample, the pseudo-labels of other data samples are updated, and vice versa. At the end of this process, the predictions from different matching classifiers are combined. It takes several rounds to label the unlabeled inputs using this method. To evaluate the proposed solutions, we conducted experiments on 4,500 online news articles relevant to temporal retrieval. LSTM, BiLSTM, and BERT models with and without lexicon expansion were assessed based on log loss and relative divergence of entropy. A jointly trained semi-supervised learning model achieved a mean KL divergence of 0.89, an F1 score of 0.74 for temporal events, and 0.63 for non-temporal events. Besides alleviating data sparsity issues and enabling the training of more complex networks, this technique can also serve as an alternative to data augmentation methods.

Odor identification errors reveal cognitive aspects of age-associated smell loss

Human olfaction can be extraordinarily sensitive, and its most common assessment method is odor identification (OID), where everyday odors are matched to word labels in a multiple-choice format. However, many older persons are unable to identify familiar odors, a deficit that is associated with the risk of future dementia and mortality. The underlying processes subserving OID in older adults are poorly understood. Here, we analyzed error patterns in OID to test whether errors could be explained by perceptual and/or semantic similarities among the response alternatives. We investigated the OID response patterns in a large, population-based sample of older adults in Sweden (n = 2479; age 60–100 years). Olfaction was assessed by a ‘Sniffin ́ TOM OID test with 16 odors; each trial involved matching a target odor to a correct label among three distractors. We analyzed the pattern of misidentifications, and the results showed that some distractors were more frequently selected than others, suggesting cognitive or perceptual factors may be present. Relatedly, we conducted a large online survey of older adults (n = 959, age 60–90 years) who were asked to imagine and rate the perceptual similarity of the target odors and the three corresponding distractors (e.g. “How similar are these smells: apple and mint?”). We then used data from the Swedish web corpus and the Word2Vec neural network algorithm to quantify the semantic association strength between the labels of each target odor and its three distractors. These data sources were used to predict odor identification errors. We found that the error patterns were partly explained by both the semantic similarity between target-distractor pairs, and the imagined perceptual similarity of the target-distractor pair. Both factors had, however, a diminished prediction in older ages, as responses became gradually less systematic. In sum, our results suggest that OID tests not only reflect olfactory perception, but also likely involve the mental processing of odor-semantic associations. This may be the reason why these tests are useful in predicting dementia onset. Our insights into olfactory-language interactions could be harnessed to develop new olfactory tests that are tailored for specific clinical purposes.

Computational modeling of a generative grammar for clitic order in River Plate Spanish

Abstract The order of clitics (CLs) in Romance languages has been studied in depth by many scholars since Perlmutter’s first approach. This author points out that person and case features impose a filter that results in the following pattern in Spanish: Se- II- I- III dat - III acc . This pattern has been revisited because it is not restrictive enough: it gives rise to impossible sequences. In this context, the goal of this work is to define a pattern of morphosyntactic features for River Plate Spanish clitics, and an ordering pattern, with the purpose of developing a computational model for an automatic analysis of this phenomenon. This kind of analysis is justified because the algorithm output shows the degree of success of the descriptive proposal. To this end, we use NooJ, a linguistic development environment that has diverse tools. The computational modeling involves two stages: (i) the creation of an electronic dictionary and (ii) the creation of a computational grammar. The algorithm developed can generate all correct sentences conformed by Nominative Pronoun + CL + CL + Verb, Non-Finite Verb-CL-CL and Imperative Verb-CL-CL, and it can recognize these kinds of expressions in written text assigning the correct semantic labels. With these results, we conclude that our descriptive proposal is adequate for the analysis of sequences of clitics in River Plate Spanish.

CBCT-to-CT Translation Using Registration-Based Generative Adversarial Networks in Patients with Head and Neck Cancer

Recently, deep learning with generative adversarial networks (GANs) has been applied in multi-domain image-to-image translation. This study aims to improve the image quality of cone-beam computed tomography (CBCT) by generating synthetic CT (sCT) that maintains the patient's anatomy as in CBCT, while having the image quality of CT. As CBCT and CT are acquired at different time points, it is challenging to obtain paired images with aligned anatomy for supervised training. To address this limitation, the study incorporated a registration network (RegNet) into GAN during training. RegNet can dynamically estimate the correct labels, allowing supervised learning with noisy labels. The study developed and evaluated the approach using imaging data from 146 patients with head and neck cancer. The results showed that GAN trained with RegNet performed better than those trained without RegNet. Specifically, in the UNIT model trained with RegNet, the mean absolute error (MAE) was reduced from 40.46 to 37.21, the root mean-square error (RMSE) was reduced from 119.45 to 108.86, the peak signal-to-noise ratio (PSNR) was increased from 28.67 to 29.55, and the structural similarity index (SSIM) was increased from 0.8630 to 0.8791. The sCT generated from the model had fewer artifacts and retained the anatomical information as in CBCT.

Data-driven few-shot crop pest detection based on object pyramid for smart agriculture

Farmers are often annoyed by crop pests, which can lead to serious losses if not detected in time. Deep learning has gradually penetrated into the agricultural field due to its proven superior performance, promoting the development of smart agriculture. Its real-time performance and effectiveness can help farmers spot pests in time and reduce losses. However, with the increasing application of deep learning technology, its disadvantage of relying heavily on large amounts of data has gradually been exposed. Before training a network, people need to spend a lot of manpower, time, and money to prepare a complete dataset and make the correct labels. To solve this problem, we introduced a less-shot object detection method in crop pest detection. We provide the first few-shot pest detection dataset (FSPD), which contains 20 categories and 2000 samples for this research. We use a few-shot object detection network based on an object pyramid called multiscale positive sample refinement. Numerous experiments are conducted to prove that, with only few (1/2/3/5/10-shot) samples, the accuracy of our method on FSPD is better than that of traditional deep learning methods, such as YOLO and faster R-CNN. After embedding the object pyramid branch, the network performance improves by an average of 5.9% on three splits compared with that without embedding. With input of only 10-shot novel samples, our method can achieve 76%, 79%, and 82% for the mAP on three splits, respectively. As a research study on the detection of crop diseases and pests with less radiation, our work can give some new insight to the future scholars, which is of great significance to the development of smart agriculture.

The Safety of Ethnic Foods in Campania Region: A Preliminary Study

Ethnic food is produced by an ethnic group-using their familiarity with local ingredients of plants and/or animal origin-and it is consumed in a country other than the country of origin. In Italy, the ethnic food market has expanded over the last three decades. The aim of this study was to evaluate the correct labeling, the microbiological communities and the allergens present in 50 ethnic foods. The visual inspection of labels and microbiological and allergen analyses have been carried out for evaluating their food safety. The visual inspection of labels revealed the absence of labeling in Italian and/or a failure to specify the place of origin. Microbiological analyses showed the absence of pathogens (i.e., Salmonella spp., Listeria monocytogenes and E. coli 0157:H7) in all matrices, but the presence of process hygiene indicator bacteria (total bacterial count, Coagulase-positive Staphylococci, Bacillus cereus, coliforms, yeasts and molds) was found in 37 samples. With regard to allergens, 12 samples were non-compliant for the presence of at least one allergen, while only two products were of species different from those declared on the label. This research highlights the need to increase the control of ethnic foods and also to improve the labeling system by standardizing international regulations.

Data-driven human and bot recognition from web activity logs based on hybrid learning techniques

Distinguishing between web traffic generated by bots and humans is an important task in the evaluation of online marketing campaigns. One of the main challenges is related to only partial availability of the performance metrics: although some users can be unambiguously classified as bots, the correct label is uncertain in many cases. This calls for the use of classifiers capable of explaining their decisions. This paper demonstrates two such mechanisms based on features carefully engineered from web logs. The first is a man-made rule-based system. The second is a hierarchical model that first performs clustering and next classification using human-centred, interpretable methods. The stability of the proposed methods is analyzed and a minimal set of features that convey the class-discriminating information is selected. The proposed data processing and analysis methodology are successfully applied to real-world data sets from online publishers.

Abstract TP98: Deep Learning-based Automatic Ischemic Core Estimation Based On Non-contrastenhanced Computed Tomography

Background/Purpose: With the spread of thrombus retrieval procedures, the demand for immediate diagnosis and severity assessment of ischemic stroke is increasing, but current standard method using ASPECTS is semi-quantitative, and has low consistency among physicians. We develop and validate the fully automatic machine learning-based ischemic core segmentation model based only on non-contrast-enhanced computed tomography (NCCT). Materials and Methods: This multicenter retrospective study included patients with anterior circulation acute ischemic stroke who received both CT and MRI before thrombolysis or recanalization treatment between 2013 to 2019. The ischemic core on CT was manually delineated referencing the DWI and ADC map. A deep learning-based ischemic core segmentation model was developed using the derivation cohort data from 3 institutions (n=313), and the model performance was validated on the external validation cohort data from 3 institutions (n=106). Results: On the validation cohort, the median time between CT and MRI was 18 min. The ischemic core volume calculated by the DL model was significantly correlated with the reference standard (Pearson r=0.91, p&lt;.001). The correlation was significant both in early time window (&lt;4.5hr from onset; Pearson r=0.91, p&lt;.0.01) and the late time window (&gt;4.5hr from onset; Pearson r=0.93, p&lt;0.01). The median difference between model and reference standard was 4.7 mL (IQR, 0.8-12.4 mL). The DL model displayed high accuracy on discriminating large ischemic core (&gt;70mL) with a sensitivity of 84.2%, specificity of 97.7%, and AUC of 0.91. Conclusions: A segmentation model of cerebral ischemic core volume of the anterior circulation based on NCCT using deep learning showed good correlation with the correct labels generated with reference to MRI.

MultiSense: Cross Labelling and Learning Human Activities Using Multimodal Sensing Data

To tap-in the gold-mine of data generated by IoT devices with unprecedented volume and value, there is an urgent need to efficiently and accurately label raw sensor data. To this end, we explore and leverage the hidden connections among the multimodal data collected by various sensing devices, and propose to let different modal data complement and learn from each other. But it is challenging to align and fuse multimodal data without knowing their perception (and thus the correct labels). In this work, we propose MultiSense , a paradigm for automatically mining potential perception, cross labelling each modal data, and then updating the learning models for recognizing human activity to achieve higher accuracy or even recognize new activities. We design innovative solutions for segmenting, aligning, and fusing multimodal data from different sensors, as well as model updating mechanism. We implement our framework and conduct comprehensive evaluations on a rich set of data. Our results demonstrate that MultiSense significantly improves the data usability and the power of the learning models. With 9 diverse activities performed by users, our framework automatically labels multimodal sensing data generated by five different sensing mechanisms (video, smart watch, smartphone, audio, and wireless-channel) with an average accuracy \(98.5\% \) . Furthermore, it enables models of some modalities to learn unknown activities from other modalities and greatly improves the activity recognition ability.