Automatic Annotation Method Research Articles

Automatic Image Annotation Based on Deep Learning Models: A Systematic Review and Future Challenges

Recently, much attention has been given to image annotation due to the massive increase in image data volume. One of the image retrieval methods which guarantees the retrieval of images in the same way as texts are automatic image annotation (AIA). Consequently, numerous studies have been conducted on AIA, particularly on the classification-based and probabilistic modeling techniques. Several image annotation techniques that performed reasonably on standard datasets have been developed over the last decade. In this paper, a review of the image annotation method was conducted, focusing more on deep learning models. Automatic image annotation (AIA) methods were also classified into five categories, including i) Convolutional Neural Network (CNN) based on AIA, ii) Recurrent Neural Network (RNN) based on AIA, iii) Deep Neural Networks (DNN) based on AIA, iv) Long-Short-Term Memory (LSTM) based on AIA, and v) Stacked auto-encoder (SAE) based on AIA. An assessment of the five varieties of AIA methods was also offered based on their principal notion, feature mining technique, explanation precision, computational density, and examined aggregated data. Moreover, the evaluation metrics used to evaluate AIA methods were reviewed and discussed. The need for careful consideration of methods throughout the improvement of novel procedures and datasets for image annotation assignment was highly demanded. From the analysis of the achievements so far, it is certain that more attention should be paid to automatic image annotation.

Automatic Annotation Method of Terror Image Based on Integrated Deep Migration Learning

With the continuous development of Internet technology, anti-terrorism images have become an important part of network anti-terrorism. In this paper, a method of automatic annotation of terror images based on integrated deep migration learning is proposed by combining parameter migration and ensemble learning, which can help filter terror information in web pages. Firstly, the deep convolution neural network model is used to train the terror image model in the source domain, and then the migration from the source domain to the target domain is realized through the migration learning technology. Then, the integrated learning framework is used to integrate the transfer learning model. Experimental results show that the accuracy and recall rate of the proposed algorithm are obviously improved.

TableRobot: An Automatic Annotation Method for Heterogeneous Tables

Using deep learning networks to recognize the table attracts lots of attention. However, due to the lack of high-quality table datasets, the performance of using deep learning networks is limited. Therefore, TableRobot has been proposed, an automatic annotation method for heterogeneous tables. To be more specific, the annotations of table consist of the coordinates of the item block and the mapping relationship between item blocks and table cells. In order to transform the task, we successfully design an algorithm based on the greedy approach to find the optimum solution. To evaluate the performance of TableRobot, we check the annotation data of 3000 tables collected from the LaTex documents in arXiv.com, and the result shows that TableRobot can generate table annotation datasets with the accuracy of 93.2%. Besides, the table annotation data is feed into GraphTSR which is a state-of-the-art table recognition graph neural network, and the F1 value of the network has increased by nearly 10% compared to before.

Accurate, automatic annotation of peptidases with hotpep-protease

Peptidases are essential for intracellular protein processing, signaling and homeostasis, physiological processes and for digestion of food. Moreover, peptidases are important biotechnological enzymes used in processes such as industrial food processing, leather manufacturing and the washing industry. Identification of peptidases is a crucial step in their characterization but peptidase annotation is not a trivial task due to their large sequence diversity.In the present study short, conserved sequence profiles were generated for all peptidase families with more than four members in the comprehensive Merops peptidase database. The sequence profiles were combined with the Homology to Peptide Pattern (Hotpep) method for automatic annotation of peptidases. This method is a stand-alone software that annotates protease sequences to Merops family and subgroup and is suitable for large-scale sequence analysis. Compared to the Mammalian Degradome Database Hotpep-protease had an accuracy of 92% and a sensitivity of 96% for annotation of the human degradome. Annotation by commonly used methods (Blast and conserved domains) had an accuracy of 69% and a sensitivity of 78%. For fungal genomes, there were large differences between annotation with Hotpep-protease, Blast- and Hidden Markov Model-based annotation and the Merops annotation, which confirms the difficulty of large-scale peptidase annotation. Manual annotation indicated that Hotpep-protease had a positive prediction rate of 0.90 compared to a positive prediction rate of 0.67 for Blast search. Hence, Hotpep-protease is highly accurate method for fast and accurate annotation of peptidases.

Detecting Entities of Works for Chinese Chatbot

Chatbots such as Xiaoice have gained huge popularity in recent years. Users frequently mention their favorite works such as songs and movies in conversations with chatbots. Detecting these entities can help design better chat strategies and improve user experience. Existing named entity recognition methods are mainly designed for formal texts, and their performance on the informal chatbot conversation texts may not be optimal. In addition, these methods rely on massive manually annotated data for model training. In this article, we propose a neural approach to detect entities of works for Chinese chatbot. Our approach is based on a language model (LM) long-short term memory (LSTM) convolutional neural network (CNN) conditional random value (CRF), or LM-LSTM-CNN-CRF, framework, which contains a language model to generate context-aware character embeddings, a Bi-LSTM network to learn contextual character representations from global contexts, a CNN to learn character representations from local contexts, and a CRF layer to jointly decode the character label sequence. In addition, we propose an automatic text annotation method via quote marks to reduce the effort of manual annotation. Besides, we propose an iterative data purification method to improve the quality of the automatically constructed labeled data. Massive experiments on a real-world dataset validate that our approach can achieve good performance on entity detection for Chinese chatbots.

Automatic image annotation method based on a convolutional neural network with threshold optimization.

In this study, a convolutional neural network with threshold optimization (CNN-THOP) is proposed to solve the issue of overlabeling or downlabeling arising during the multilabel image annotation process in the use of a ranking function for label annotation along with prediction probability. This model fuses the threshold optimization algorithm to the CNN structure. First, an optimal model trained by the CNN is used to predict the test set images, and batch normalization (BN) is added to the CNN structure to effectively accelerate the convergence speed and obtain a group of prediction probabilities. Second, threshold optimization is performed on the obtained prediction probability to derive an optimal threshold for each class of labels to form a group of optimal thresholds. When the prediction probability for this class of labels is greater than or equal to the corresponding optimal threshold, this class of labels is used as the annotation result for the image. During the annotation process, the multilabel annotation for the image to be annotated is realized by loading the optimal model and the optimal threshold. Verification experiments are performed on the MIML, COREL5K, and MSRC datasets. Compared with the MBRM, the CNN-THOP increases the average precision on MIML, COREL5K, and MSRC by 27%, 28% and 33%, respectively. Compared with the E2E-DCNN, the CNN-THOP increases the average recall rate by 3% on both COREL5K and MSRC. The most precise annotation effect for CNN-THOP is observed on the MIML dataset, with a complete matching degree reaching 64.8%.

Neuron ID dataset facilitates neuronal annotation for whole-brain activity imaging of C. elegans

BackgroundAnnotation of cell identity is an essential process in neuroscience that allows comparison of cells, including that of neural activities across different animals. In Caenorhabditis elegans, although unique identities have been assigned to all neurons, the number of annotatable neurons in an intact animal has been limited due to the lack of quantitative information on the location and identity of neurons.ResultsHere, we present a dataset that facilitates the annotation of neuronal identities, and demonstrate its application in a comprehensive analysis of whole-brain imaging. We systematically identified neurons in the head region of 311 adult worms using 35 cell-specific promoters and created a dataset of the expression patterns and the positions of the neurons. We found large positional variations that illustrated the difficulty of the annotation task. We investigated multiple combinations of cell-specific promoters driving distinct fluorescence and generated optimal strains for the annotation of most head neurons in an animal. We also developed an automatic annotation method with human interaction functionality that facilitates annotations needed for whole-brain imaging.ConclusionOur neuron ID dataset and optimal fluorescent strains enable the annotation of most neurons in the head region of adult C. elegans, both in full-automated fashion and a semi-automated version that includes human interaction functionalities. Our method can potentially be applied to model species used in research other than C. elegans, where the number of available cell-type-specific promoters and their variety will be an important consideration.

2019 APHRS expert consensus statement on three-dimensional mapping systems for tachycardia developed in collaboration with HRS, EHRA, and LAHRS.

2019 APHRS expert consensus statement on three-dimensional mapping systems for tachycardia developed in collaboration with HRS, EHRA, and LAHRS.

Automatic “Ground Truth” Annotation and Industrial Workpiece Dataset Generation for Deep Learning

In industry, it is becoming common to detect and recognize industrial workpieces using deep learning methods. In this field, the lack of datasets is a big problem, and collecting and annotating datasets in this field is very labor intensive. The researchers need to perform dataset annotation if a dataset is generated by themselves. It is also one of the restrictive factors that the current method based on deep learning cannot expand well. At present, there are very few workpiece datasets for industrial fields, and the existing datasets are generated from ideal workpiece computer aided design (CAD) models, for which few actual workpiece images were collected and utilized. We propose an automatic industrial workpiece dataset generation method and an automatic ground truth annotation method. Included in our methods are three algorithms that we proposed: a point cloud based spatial plane segmentation algorithm to segment the workpieces in the real scene and to obtain the annotation information of the workpieces in the images captured in the real scene; a random multiple workpiece generation algorithm to generate abundant composition datasets with random rotation workpiece angles and positions; and a tangent vector based contour tracking and completion algorithm to get improved contour images. With our procedures, annotation information can be obtained using the algorithms proposed in this paper. Upon completion of the annotation process, a json format file is generated. Faster R-CNN (Faster R-convolutional neural network), SSD (single shot multibox detector) and YOLO (you only look once: unified, real-time object detection) are trained using the datasets proposed in this paper. The experimental results show the effectiveness and integrity of this dataset generation and annotation method.

Automatic semantic annotation method for mine Semantic Web of things

Automatic semantic annotation method for mine Semantic Web of things

Real-Time Inspection System for Ballast Railway Fasteners Based on Point Cloud Deep Learning

Rail fasteners are the most numerous components in railways and they should be inspected periodically. Manual inspection is currently a common solution, which is laborious and low-efficient. Some automatic inspection approaches are proposed. But for ballast railway fasteners inspection, debris, especially ballast along tracks may cover the fasteners, which is still a tricky problem. In this paper, a real-time inspection system for ballast railway fasteners based on point cloud deep learning is developed. Dense and precise point cloud of fastener is obtained from the structured light sensors in the system. The point cloud of fastener is segmented into different parts to avoid the interference of debris on fasteners. A ballast fastener point cloud semantic segmentation dataset is created based on automatic annotation method. Several deep learning point cloud segmentation models are tested in this dataset and PointNet++ is selected to be deployed in the real-time deep learning module of the system. Field tests on ballast railways show excellent accuracy and efficiency of this system.

Hierarchical attentive deep neural networks for semantic music annotation through multiple music representations

Automatically assigning a group of appropriate semantic tags to one music piece provides an effective way for people to efficiently utilize the massive and ever increasing online and off-line music data. In this paper, we propose a novel end-to-end deep neural network model for automatic music annotation, which effectively integrates available multiple complementary music representations and jointly accomplishes music representation learning, structure modeling, and tag prediction. The model first hierarchically leverages attentive convolutional networks and recurrent networks to learn informative descriptions from Mel-spectrogram and raw waveform of the music and depict time-varying structures embedded in the description sequence. A dual-state LSTM network is then employed to capture the correlations between two representation channels as supplementary music descriptions. Finally, the model aggregates music description sequence into a holistic embedding with a self-attentive multi-weighting mechanism, which adaptively captures multi-aspect summarized information of the music for tag prediction. Experiments on the public MagnaTagATune benchmark music dataset show that the proposed model outperforms state-of-the-art methods for automatic music annotation.

Web search model for Automatic Annotation of Tweets in Opinion Mining

In sentiment analysis, annotation is the crucial step of data processing to label the review or sentences as positive, negative, or neutral. An annotation process is usually performed by three key approaches: (i) manual, (ii) crowdsourcing, and (iii) automated annotation. Manual annotation is preferred in most of the literature’s and crowdsourcing tools are used in some of the works. This indicates that there is a scarce of automatic annotation and its service is highly essential to support more systematic research in sentiment analysis. Manual procedures mostly depends on external annotators, resulting in costly and time-consuming processes. Thus, we propose a method for automatic annotation using web search model to dynamically label reviews (positive, negative, or neutral) that are not available in the dictionary. Some research works consider the product opinions from e-commerce sites where most of the texts may not be available in the dictionary, in such cases, the web search model can be used instead of manual annotation. A large-scale opinosis dataset is used to evaluate the accuracy of the algorithms and feasibility of the model. The experimental results indicate that this model outperforms conventional methodologies and therefore we firmly believe it will be useful for current researchers in the field of opinion mining.

An Automatic Annotation Method for Discovering Semantic Information of Geographical Locations from Location-Based Social Networks

Location-Based Social Networks (LBSNs) contain rich information that can be used to identify and annotate points of interest (POIs). Discovering these POIs and annotating them with this information is not only helpful for understanding the social behavior of users, but it also provides benefits for location recommendations. However, current methods still have some limitations, such as a long annotating time and a low annotating accuracy. In this study, we develop a hybrid method to annotate POIs with meaningful information from LBSNs. The method integrates three patterns: temporal, spatial, and text patterns. Firstly, we present an approach for preprocessing data based on temporal patterns. Secondly, we describe a way to discover POIs through spatial patterns. Thirdly, we build a keyword dictionary for discovering the categories of POIs to be annotated via mining the text patterns. Finally, we integrate these three patterns to label each POI. Taking New York and London as the target areas, we accomplish automatic POI annotation by using Precision, Recall, and F-values to evaluate the effectiveness. The results show that our F-value is 78%, which is superior to that of the baseline method (Falcone's method) at 73% and this suggests that our method is effective in extracting POIs and assigning them categories.

Automatic Annotation of Events and Highlights Generation of Cricket Match Videos

In this paper, we propose a novel method for automatic annotation of events and highlights generation for cricket match videos. The videos are divided into time intervals representing one ball clip using a Convolutional Neural Network (CNN) and Optical Character Recognition (OCR). CNN detects the start frame of a ball. OCR is used to detect the end of the ball and to annotate it by recognizing the change in runs/wickets. The proposed framework is able to annotate events and generate sufficiently good highlights for four full length cricket matches.

REMOTE SENSING SCENE CLASSIFICATION USING MULTIPLE PYRAMID POOLING

Abstract. Remote sensing image scene classification has gained remarkable attention, due to its versatile use in different applications like geospatial object detection, ground object information extraction, environment monitoring and etc. The scene not only contains the information of the ground objects, but also includes the spatial relationship between the ground objects and the environment. With rapid growth of the amount of remote sensing image data, the need for automatic annotation methods for image scenes is more urgent. This paper proposes a new framework for high resolution remote sensing images scene classification based on convolutional neural network. To eliminate the requirement of fixed-size input image, multiple pyramid pooling strategy is equipped between convolutional layers and fully connected layers. Then, the fixed-size features generated by multiple pyramid pooling layer was extended to one-dimension fixed-length vector and fed into fully connected layers. Our method could generate a fixed-length representation regardless of image size, at the same time get higher classification accuracy. On UC-Merced and NWPU-RESISC45 datasets, our framework achieved satisfying accuracies, which is 93.24% and 88.62% respectively.

End-to-End Automatic Image Annotation Based on Deep CNN and Multi-Label Data Augmentation

Automatic image annotation is a key step in image retrieval and image understanding. In this paper, we present an end-to-end automatic image annotation method based on a deep convolutional neural network (CNN) and multi-label data augmentation. Different from traditional annotation models that usually perform feature extraction and annotation as two independent tasks, we propose an end-to-end automatic image annotation model based on deep CNN (E2E-DCNN). E2E-DCNN transforms the image annotation problem into a multi-label learning problem. It uses a deep CNN structure to carry out the adaptive feature learning before constructing the end-to-end annotation structure using multiple cross-entropy loss functions for training. It is difficult to train a deep CNN model using small-scale datasets or scale up multi-label datasets using traditional data augmentation methods; hence, we propose a multi-label data augmentation method based on Wasserstein generative adversarial networks (ML-WGAN). The ML-WGAN generator can approximate the data distribution of a single multi-label image. The images generated by ML-WGAN can assist in the reduction of the over-fitting problem of training a deep CNN model and enhance the generalization ability of the trained CNN model. We optimize the network structure by using deformable convolution and spatial pyramid pooling. We experiment the proposed E2E-DCNN model with data augmentation by the proposed ML-WGAN on several public datasets. The experimental results demonstrate that the proposed model outperforms the state-of-the-art automatic image annotation models.

Structuring Description for Product Image Data with Multilabel

Existing product data, presented as description documents, product images, etc., are very significant references for designing a new proposal. However, traditionally, the data collected manually only contain the product images without description documents. Obtaining reasonable description of product is a challenging task due to the high cost and labor-intensive process of human annotation. In this paper, a new approach is introduced to solve this problem and improve the efficiency of description by exploiting the potential information of product images. We propose a robust framework with multi-label learning to annotate each product image with several labels, which makes a brief description of the product from different aspects. An efficient method is proposed to accomplish product data acquisition, arrangement and analysis task to construct new data in images collection step. Furthermore, for the newly structured data, robust algorithms are employed to accomplish the data processing. Based on the processed data, automatic and semi-automatic multi-label annotation methods are applied to generate the labels for product images. We present the prediction results by several state-of-the-art multi-label learning classifiers based on the features extracted from different convolution neural framework. The results are evaluated by effective measures to validate the quality of the labeling result, as well as the predictive performance with respect to the number of the training examples. Experimental results indicate that the quality of the annotation is reasonable, and the proposed method can achieve excellent prediction accuracy based on the annotated data, generating satisfactory description for the products.

ECG-based Random Forest Classifier for Cardiac Arrest Rhythms.

Rhythm annotation of out-of-hospital cardiac episodes (OHCA) is key for a better understanding of the interplay between resuscitation therapy and OHCA patient outcome. OHCA rhythms are classified in five categories, asystole (AS), pulseless electrical activity (PEA), pulsed rhythms (PR), ventricular fibrillation (VF) and ventricular tachycardia (VT). Manual OHCA annotation by expert clinicians is onerous and time consuming, so there is a need for accurate and automatic OHCA rhythm annotation methods. For this study 852 OHCA episodes of patients treated with Automated External Defibrillators (AED) by the Emergency Medical Services of the Basque Country were analyzed. Six expert clinicians reviewed the electrocardiogram (ECG) of 4214 AED rhythm analyses and annotated the rhythm. Their consensus decision was used as ground truth. There were a total of 2418 AS, 294 PR, 1008 PEA, 472 VF and 22 VT. The ECG analysis intervals were extracted and used to develop an automatic rhythm annotator. Data was partitioned patient-wise into training (70%) and test (30%). Performance was evaluated in terms of per class sensitivity (Se) and F-score (F1). The unweighted mean of sensitivity (UMS) and F-score were used as global performance metrics. The classification method is composed of a feature extraction and denoising stage based on the stationary wavelet transform of the ECG, and on a random forest classifier. The best model presented a per rhythm Se/F1 of 95.8/95.7, 43.3/52.2, 85.3/81.3, 94.2/96.1, 81.9/72.2 for AS, PR, PEA, VF and VT, respectively. The UMS for the test set was 80.2%, 2-points above that of previous solutions. This method could be used to retrospectively annotate large OHCA datasets and ameliorate the workload of manual OHCA rhythm annotation.

SleepEEGNet: Automated sleep stage scoring with sequence to sequence deep learning approach.

Electroencephalogram (EEG) is a common base signal used to monitor brain activities and diagnose sleep disorders. Manual sleep stage scoring is a time-consuming task for sleep experts and is limited by inter-rater reliability. In this paper, we propose an automatic sleep stage annotation method called SleepEEGNet using a single-channel EEG signal. The SleepEEGNet is composed of deep convolutional neural networks (CNNs) to extract time-invariant features, frequency information, and a sequence to sequence model to capture the complex and long short-term context dependencies between sleep epochs and scores. In addition, to reduce the effect of the class imbalance problem presented in the available sleep datasets, we applied novel loss functions to have an equal misclassified error for each sleep stage while training the network. We evaluated the performance of the proposed method on different single-EEG channels (i.e., Fpz-Cz and Pz-Oz EEG channels) from the Physionet Sleep-EDF datasets published in 2013 and 2018. The evaluation results demonstrate that the proposed method achieved the best annotation performance compared to current literature, with an overall accuracy of 84.26%, a macro F1-score of 79.66% and κ = 0.79. Our developed model can be applied to other sleep EEG signals and aid the sleep specialists to arrive at an accurate diagnosis. The source code is available at https://github.com/SajadMo/SleepEEGNet.