Automatic Data Annotation Research Articles

A novel neural network architecture and cross-model transfer learning for multi-task autonomous driving

PurposeThe purpose of this research is to achieve multi-task autonomous driving by adjusting the network architecture of the model. Meanwhile, after achieving multi-task autonomous driving, the authors found that the trained neural network model performs poorly in untrained scenarios. Therefore, the authors proposed to improve the transfer efficiency of the model for new scenarios through transfer learning.Design/methodology/approachFirst, the authors achieved multi-task autonomous driving by training a model combining convolutional neural network and different structured long short-term memory (LSTM) layers. Second, the authors achieved fast transfer of neural network models in new scenarios by cross-model transfer learning. Finally, the authors combined data collection and data labeling to improve the efficiency of deep learning. Furthermore, the authors verified that the model has good robustness through light and shadow test.FindingsThis research achieved road tracking, real-time acceleration–deceleration, obstacle avoidance and left/right sign recognition. The model proposed by the authors (UniBiCLSTM) outperforms the existing models tested with model cars in terms of autonomous driving performance. Furthermore, the CMTL-UniBiCL-RL model trained by the authors through cross-model transfer learning improves the efficiency of model adaptation to new scenarios. Meanwhile, this research proposed an automatic data annotation method, which can save 1/4 of the time for deep learning.Originality/valueThis research provided novel solutions in the achievement of multi-task autonomous driving and neural network model scenario for transfer learning. The experiment was achieved on a single camera with an embedded chip and a scale model car, which is expected to simplify the hardware for autonomous driving.

DCServCG: A data-centric service code generation using deep learning

Modern software development paradigms, including Service-Oriented Architecture (SOA), tend to make use of available services e.g., web service Application Programming Interfaces (APIs) to generate new software. Thus, for the further advancement of SOA, the development of accurate automatic tasks, such as service discovery and composition, is necessary. Most of these automated tasks rely heavily on web service metadata annotation. The lack of machine-readable documentation and structured metadata reduces the accuracy and volume of automatic data annotation, negatively affecting the performance of automated SOA tasks. This study aims to propose automatic code completion for improving web service-based systems by identifying and capturing service usage collected from public repositories that share Open Source Software (OSS). To this end, a Data-Centric Service Code Generation (DCServCG) model is proposed to improve old-fashioned, general-purpose code generators that neglect essential service-based code characteristics e.g., sequence overlap and bias issues. DCServCG takes advantage of the data-centric concept, i.e., conditional text generation, to overcome the mentioned issues. We have evaluated the approach from the point of view of language modeling metrics. The obtained results indicate that the usage of the data-centric approach reduces perplexity by 1.125. Moreover, the DCServCG model uses de-noising and conditional text generation, which is trained on the transformer by distilling the knowledge, DistilGPT2 (82M parameters) trained faster and its perplexity is 0.363 lower than ServCG (124M parameters) without de-noising and conditional text generation, which lower perplexity value indicates better model generalization performance.

Vader Lexicon and Support Vector Machine Algorithm to Detect Customer Sentiment Orientation

Background: The concept of customer orientation, which is based on a set of fundamental beliefs that prioritize the interests of the customer, requires companies to detect these interests in order to maintain a high level of quality in their products or services. Furthermore, there are several indicators of customer orientation, and one of them is their opinion or taste, which provides valuable feedback for businesses. With the rapid development of social media, customers can express emotions, thoughts, and opinions about services or products that may not be easily conveyed in the real world. Objective: The objective of this study is to detect customer orientation towards product or service quality, as expressed in online or social media. Additionally, the study showcases the novelty and superiority of the annotation process used for detecting customer orientation classifications. Methods: This study employs a method to compare the classification performance of the Vader lexicon annotation process with manual annotation. To accomplish this, a dataset from the Amazon website will be analyzed and classified using the Support Vector Machine algorithm. The objective of this method is to determine the level of customer orientation present within the dataset. To evaluate the effectiveness of the Vader lexicon, the study will compare the results of manual and automatic data annotation. Results: The results showed that customer orientation towards product or service quality has a predominantly positive value, comprising up to 76% of the total responses analyzed. Conclusion: The findings demonstrate that using Vader in the annotation process results in superior accuracy values compared to manual annotation. Specifically, the accuracy value increased from 86% to 88.57%, indicating that Vader could be a reliable tool for annotating text. Therefore, future studies should consider using Vader as a classifier or integrating it into the annotation process to further enhance its performance. Keywords: Classification, Customer, Orientation, Text analysis, Vader lexicon,

VPP: Visual Pollution Prediction Framework Based on a Deep Active Learning Approach Using Public Road Images

Visual pollution (VP) is the deterioration or disruption of natural and man-made landscapes that ruins the aesthetic appeal of an area. It also refers to physical elements that limit the movability of people on public roads, such as excavation barriers, potholes, and dilapidated sidewalks. In this paper, an end-to-end visual pollution prediction (VPP) framework based on a deep active learning (DAL) approach is proposed to simultaneously detect and classify visual pollutants from whole public road images. The proposed framework is architected around the following steps: real VP dataset collection, pre-processing, a DAL approach for automatic data annotation, data splitting as well as augmentation, and simultaneous VP detection and classification. This framework is designed to predict VP localization and classify it into three categories: excavation barriers, potholes, and dilapidated sidewalks. A real dataset with 34,460 VP images was collected from various regions across the Kingdom of Saudi Arabia (KSA) via the Ministry of Municipal and Rural Affairs and Housing (MOMRAH), and this was used to develop and fine-tune the proposed artificial intelligence (AI) framework via the use of five AI predictors: MobileNetSSDv2, EfficientDet, Faster RCNN, Detectron2, and YOLO. The proposed VPP-based YOLO framework outperforms competitor AI predictors with superior prediction performance at 89% precision, 88% recall, 89% F1-score, and 93% mAP. The DAL approach plays a crucial role in automatically annotating the VP images and supporting the VPP framework to improve prediction performance by 18% precision, 27% recall, and 25% mAP. The proposed VPP framework is able to simultaneously detect and classify distinct visual pollutants from annotated images via the DAL strategy. This technique is applicable for real-time monitoring applications.

Embedding Weather Simulation in Auto-Labelling Pipelines Improves Vehicle Detection in Adverse Conditions.

The performance of deep learning-based detection methods has made them an attractive option for robotic perception. However, their training typically requires large volumes of data containing all the various situations the robots may potentially encounter during their routine operation. Thus, the workforce required for data collection and annotation is a significant bottleneck when deploying robots in the real world. This applies especially to outdoor deployments, where robots have to face various adverse weather conditions. We present a method that allows an independent car tansporter to train its neural networks for vehicle detection without human supervision or annotation. We provide the robot with a hand-coded algorithm for detecting cars in LiDAR scans in favourable weather conditions and complement this algorithm with a tracking method and a weather simulator. As the robot traverses its environment, it can collect data samples, which can be subsequently processed into training samples for the neural networks. As the tracking method is applied offline, it can exploit the detections made both before the currently processed scan and any subsequent future detections of the current scene, meaning the quality of annotations is in excess of those of the raw detections. Along with the acquisition of the labels, the weather simulator is able to alter the raw sensory data, which are then fed into the neural network together with the labels. We show how this pipeline, being run in an offline fashion, can exploit off-the-shelf weather simulation for the auto-labelling training scheme in a simulator-in-the-loop manner. We show how such a framework produces an effective detector and how the weather simulator-in-the-loop is beneficial for the robustness of the detector. Thus, our automatic data annotation pipeline significantly reduces not only the data annotation but also the data collection effort. This allows the integration of deep learning algorithms into existing robotic systems without the need for tedious data annotation and collection in all possible situations. Moreover, the method provides annotated datasets that can be used to develop other methods. To promote the reproducibility of our research, we provide our datasets, codes and models online.

Deep Learning Based SWIR Object Detection in Long-Range Surveillance Systems: An Automated Cross-Spectral Approach.

SWIR imaging bears considerable advantages over visible-light (color) and thermal images in certain challenging propagation conditions. Thus, the SWIR imaging channel is frequently used in multi-spectral imaging systems (MSIS) for long-range surveillance in combination with color and thermal imaging to improve the probability of correct operation in various day, night and climate conditions. Integration of deep-learning (DL)-based real-time object detection in MSIS enables an increase in efficient utilization for complex long-range surveillance solutions such as border or critical assets control. Unfortunately, a lack of datasets for DL-based object detection models training for the SWIR channel limits their performance. To overcome this, by using the MSIS setting we propose a new cross-spectral automatic data annotation methodology for SWIR channel training dataset creation, in which the visible-light channel provides a source for detecting object types and bounding boxes which are then transformed to the SWIR channel. A mathematical image transformation that overcomes differences between the SWIR and color channel and their image distortion effects for various magnifications are explained in detail. With the proposed cross-spectral methodology, the goal of the paper is to improve object detection in SWIR images captured in challenging outdoor scenes. Experimental tests for two object types (cars and persons) using a state-of-the-art YOLOX model demonstrate that retraining with the proposed automatic cross-spectrally created SWIR image dataset significantly improves average detection precision. We achieved excellent improvements in detection performance in various variants of the YOLOX model (nano, tiny and x).

Автоматическая разметка данных для сегментации изображений документов с использованием глубоких нейронных сетей

The article proposes a new method for automatic data annotation for solving the problem of document image segmentation using deep object detection neural networks. The format of marked PDF files is considered as the initial data for markup. The peculiarity of this format is that it includes hidden marks that describe the logical and physical structure of the document. To extract them, a tool has been developed that simulates the operation of a stack-based printing machine according to the PDF format specification. For each page of the document, an image and annotation are generated in PASCAL VOC format. The classes and coordinates of the bounding boxes are calculated during the interpretation of the labeled PDF file based on the labels. To test the method, a collection of marked up PDF files was formed from which images of document pages and annotations for three segmentation classes (text, table, figure) were automatically obtained. Based on these data, a neural network of the EfficientDet D2 architecture was trained. The model was tested on manually labeled data from the same domain, which confirmed the effectiveness of using automatically generated data for solving applied problems.

Pixel-level tunnel crack segmentation using a weakly supervised annotation approach

Automatic crack detection plays an essential role in ensuring the safe operation of tunnels, which is also challenging work in reality. In this paper, an innovative framework, which combines the weakly supervised learning methods (WSL) and the fully supervised learning methods (FSL), is presented to detect and segment the cracks in the tunnel images. Firstly, a WSL-based segmentation network Crack-CAM is proposed to annotate the collected data instead of using the traditional manual annotation process. By applying the proposed E-Res2Net101 structure and tuning some hyper-parameters, an FSL-based method named DeepLabv3+ is optimized to enhance the segmentation performance. After the crack segmentation, the risk levels of the detected cracks are judged using a new evaluation metric. In addition, the mean error of the lengths, the mean widths, and the areas are calculated for different types of cracks. A crack dataset in tunnel scenes that contain 3,921,726 sub-images that are cropped from 521 raw images is built to demonstrate the effectiveness of the presented methods. Based on the proposed dataset, the modified DeepLabv3+ achieves the highest MIoU of 0.786 and the best F1 of 0.865. Besides, the proposed framework combining WSL methods (automatic data annotation) and the FSL methods achieved a performance comparable to the framework that is based on manual annotation and the FSL methods, which demonstrates the WSL-based Crack-CAM can label images correctly.

Proximity based automatic data annotation for autonomous driving

The recent development in autonomous driving involves high-level computer vision and detailed road scene understanding. Today, most autonomous vehicles employ expensive high quality sensor-set such as light detection and ranging &#x0028 LIDAR &#x0029 and HD maps with high level annotations. In this paper, we propose a scalable and affordable data collection and annotation framework, image-to-map annotation proximity &#x0028 I2MAP &#x0029, for affordance learning in autonomous driving applications. We provide a new driving dataset using our proposed framework for driving scene affordance learning by calibrating the data samples with available tags from online database such as open street map &#x0028 OSM &#x0029 . Our benchmark consists of 40 000 images with more than 40 affordance labels under various day time and weather even with very challenging heavy snow. We implemented sample advanced driver-assistance systems &#x0028 ADAS &#x0029 functions by training our data with neural networks &#x0028 NN &#x0029 and cross-validate the results on benchmarks like KITTI and BDD100K, which indicate the effectiveness of our framework and training models.

DeepTrigger: A Watermarking Scheme of Deep Learning Models Based on Chaotic Automatic Data Annotation

With the rapid development of artificial intelligence, the intellectual property protection of deep learning models appeals widespread concerns of scientists and engineers. The black-box watermarking protection scheme has been favored by many scholars due to its many advantages. The trigger set containing data content and data annotation is the key of black-box watermarking technology. However, most of the trigger sets in literates were constructed by comprehensible features, such as Gaussian noise and badges on original data content. Then, the attacks based on machine learning can obtain the watermarking features and generate fake trigger set. Therefore, fraudulent ownership claim attacks may occur. In this paper, we turn our attention to data annotation and propose a black-box watermarking scheme based on chaotic automatic data annotation. Chaos has superior features, such as the sensitivity of initial value, aperiodic behavior and unpredictability of the chaotic sequence. We applies these chaotic features on data annotation so as to against the fraudulent ownership claim attacks. Firstly, this scheme applies chaotic automatic data annotation, which is time-saving and non-manual labeling. Secondly, chaotic sequences are unpredictable for long-terms, which can break the principle of empirical or statistical machine learning based attacks when chaotic labeling the trigger samples. Thirdly, the initial value and parameters in chaos offer a large range of key space, which can facilitate the commercialization of the intelligent models. The key formulation also guarantees the separation of the secret key and the trigger set. In addition, experiments and simulations show that the scheme is effective, secure and robust. It can resist fine-tuning attacks, compression attacks, fraudulent ownership claim attacks and overwriting attacks.

Deep Bayesian Self-Training

Supervised deep learning has been highly successful in recent years, achieving state-of-the-art results in most tasks. However, with the ongoing uptake of such methods in industrial applications, the requirement for large amounts of annotated data is often a challenge. In most real-world problems, manual annotation is practically intractable due to time/labour constraints; thus, the development of automated and adaptive data annotation systems is highly sought after. In this paper, we propose both a (1) deep Bayesian self-training methodology for automatic data annotation, by leveraging predictive uncertainty estimates using variational inference and modern neural network (NN) architectures, as well as (2) a practical adaptation procedure for handling high label variability between different dataset distributions through clustering of NN latent variable representations. An experimental study on both public and private datasets is presented illustrating the superior performance of the proposed approach over standard self-training baselines, highlighting the importance of predictive uncertainty estimates in safety-critical domains.

Automatic Annotation of Unlabeled Data from Smartphone-Based Motion and Location Sensors.

Automatic data annotation eliminates most of the challenges we faced due to the manual methods of annotating sensor data. It significantly improves users’ experience during sensing activities since their active involvement in the labeling process is reduced. An unsupervised learning technique such as clustering can be used to automatically annotate sensor data. However, the lingering issue with clustering is the validation of generated clusters. In this paper, we adopted the k-means clustering algorithm for annotating unlabeled sensor data for the purpose of detecting sensitive location information of mobile crowd sensing users. Furthermore, we proposed a cluster validation index for the k-means algorithm, which is based on Multiple Pair-Frequency. Thereafter, we trained three classifiers (Support Vector Machine, K-Nearest Neighbor, and Naïve Bayes) using cluster labels generated from the k-means clustering algorithm. The accuracy, precision, and recall of these classifiers were evaluated during the classification of “non-sensitive” and “sensitive” data from motion and location sensors. Very high accuracy scores were recorded from Support Vector Machine and K-Nearest Neighbor classifiers while a fairly high accuracy score was recorded from the Naïve Bayes classifier. With the hybridized machine learning (unsupervised and supervised) technique presented in this paper, unlabeled sensor data was automatically annotated and then classified.

An Annotation Technique for In-Home Smart Monitoring Environments

Advances in multimedia technologies have led to the emergence of smart home applications. In fact, mobile multimedia technologies provide the infrastructure to adopt smart solutions and track inhabitants’ activities. In-home activity recognition significantly enhances the performance of healthcare-monitoring and emergency-control applications for elderly and people with special needs. Developing and validating data models for such applications requires training sets that reflect a ground truth in the form of labeled or annotated data. With the accelerated development of Internet-of-Things applications, automated annotation processes have emerged understanding resident behavior in terms of activities. This paper presents a methodology for automatic data annotation by profiling sensing nodes. Our proposed methodology models activities based on spatially recognized actions, with every activity expected to have a direct relationship with a specific set of locations. Furthermore, the proposed technique validates the assignment of labels based on the temporal relations among consecutive actions. We performed experiments to evaluate our proposed methodology on CASAS data sets, which indicated that the proposed methodology achieved better performance, to a statistically significant extent, than the state-of-the-art methodologies presented in the literature.

A new F-score gradient-based training rule for the linear model

Delta rule is a standard, well-established approach to train perceptron recognition model. However, mean squared error, on which it is based, is not suitable estimate for some problems, like information retrieval or automatic data annotation. F-score, a combination of precision and recall, is one of the major quality measures and can be used as an alternative. In this paper we present perceptron training model based on f-score. An approximate of f-score is proposed, based on components which are both continuous and differentiable. It allows to formulate a gradient-descent training routine, conceptually similar to the standard delta rule.