Sequence Labeling Problem Research Articles

How Well Do Change Sequences Predict Defects? Sequence Learning from Software Changes

Software defect prediction, which aims to identify defective modules, can assist developers in finding bugs and prioritizing limited quality assurance resources. Various features to build defect prediction models have been proposed and evaluated. Among them, process metrics are one important category. Yet, existing process metrics are mainly encoded manually from change histories and ignore the sequential information arising from the changes during software evolution. Are the change sequences derived from such information useful to characterize buggy program modules? How can we leverage such sequences to build good defect prediction models? Unlike traditional process metrics used for existing defect prediction models, change sequences are mostly vectors of variable length. This makes it difficult to apply such sequences directly in prediction models that are driven by conventional classifiers. To resolve this challenge, we utilize Recurrent Neural Network (RNN), which is a deep learning technique, to encode features from sequence data automatically. In this paper, we propose a novel approach called Fences , which extracts six types of change sequences covering different aspects of software changes via fine-grained change analysis. It approaches defects prediction by mapping it to a sequence labeling problem solvable by RNN. Our evaluations on 10 open source projects show that Fences can predict defects with high performance. In particular, our approach achieves an average F-measure of 0.657, which improves the prediction models built on traditional metrics significantly. The improvements vary from 31.6 to 46.8 percent on average. In terms of AUC, Fences achieves an average value of 0.892, and the improvements over baselines vary from 4.2 to 16.1 percent. Fences also outperforms the state-of-the-art technique which learns semantic features automatically from static code via deep learning.

Causality extraction based on self-attentive BiLSTM-CRF with transferred embeddings

Causality extraction from natural language texts is a challenging open problem in artificial intelligence. Existing methods utilize patterns, constraints, and machine learning techniques to extract causality, heavily depending on domain knowledge and requiring considerable human effort and time for feature engineering. In this paper, we formulate causality extraction as a sequence labeling problem based on a novel causality tagging scheme. On this basis, we propose a neural causality extractor with the BiLSTM-CRF model as the backbone, named SCITE (Self-attentive BiLSTM-CRF wIth Transferred Embeddings), which can directly extract cause and effect without extracting candidate causal pairs and identifying their relations separately. To address the problem of data insufficiency, we transfer contextual string embeddings, also known as Flair embeddings, which are trained on a large corpus in our task. In addition, to improve the performance of causality extraction, we introduce a multihead self-attention mechanism into SCITE to learn the dependencies between causal words. We evaluate our method on a public dataset, and experimental results demonstrate that our method achieves significant and consistent improvement compared to baselines.

Domain Generalization for Named Entity Boundary Detection via Metalearning.

Named entity recognition (NER) aims to recognize mentions of rigid designators from text belonging to predefined semantic types, such as person, location, and organization. In this article, we focus on a fundamental subtask of NER, named entity boundary detection, which aims at detecting the start and end boundaries of an entity mention in the text, without predicting its semantic type. The entity boundary detection is essentially a sequence labeling problem. Existing sequence labeling methods either suffer from sparse boundary tags (i.e., entities are rare and nonentities are common) or they cannot well handle the issue of variable size output vocabulary (i.e., need to retrain models with respect to different vocabularies). To address these two issues, we propose a novel entity boundary labeling model that leverages pointer networks to effectively infer boundaries depending on the input sequence. On the other hand, training models on source domains that generalize to new target domains at the test time are a challenging problem because of the performance degradation. To alleviate this issue, we propose Metabdry, a novel domain generalization approach for entity boundary detection without requiring any access to target domain information. Especially, adversarial learning is adopted to encourage domain-invariant representations. Meanwhile, metalearning is used to explicitly simulate a domain shift during training so that metaknowledge from multiple resource domains can be effectively aggregated. As such, Metabdry explicitly optimizes the capability of "learning to generalize," resulting in a more general and robust model to reduce the domain discrepancy. We first conduct experiments to demonstrate the effectiveness of our novel boundary labeling model. We then extensively evaluate Metabdry on eight data sets under domain generalization settings. The experimental results show that Metabdry achieves state-of-the-art results against the recent seven baselines.

ATE-SPD: simultaneous extraction of aspect-term and aspect sentiment polarity using Bi-LSTM-CRF neural network

ABSTRACT Aspect-based sentiment analysis is one of the challenging problems among the various type of tasks in sentiment analysis. Sequential models specifically deep neural networks (like Recurrent Neural Networks) have been found to handle this problem in an efficient way. This paper presents a deep neural network model named ATE-SPD for aspect-based sentiment analysis that simultaneously extracts aspect-terms and their corresponding polarities in review sentences. This problem can be solved as a sequence labelling problem. We are using Bi-LSTM hybridised with CRF as both of these approaches are state-of-the-art approaches for sequence labelling tasks. It was observed that CRF is able to improve the performance of the traditional Bi-LSTM model. Another important contribution of this paper is that it provides a novel set of sequential tags for extracting aspect-terms along with their sentiment polarities. Aspect-terms and their polarities are determined without explicitly labelling the sentiment terms. The ATE-SPD is evaluated using a benchmark dataset of SemEval’14-Task4 and obtains state-of-the-art performance.

Multi-entity sentiment analysis using self-attention based hierarchical dilated convolutional neural network

In this paper, a self-attention based hierarchical dilated convolutional neural network (SA-HDCNN) is proposed for multi-entity sentiment analysis (MESA), in which the task is directly transformed into a sequence labeling problem avoiding decomposition and is also suitable for parallel computing. Specifically, SA-HDCNN is mainly composed of encoding, feature extraction and decoding modules. The encoding module is to map the input sentence into a word embedding matrix, which contains both semantic and sentiment information. Next, the feature extraction module is to learn multi-scale local features and inter-word global dependencies of the encoded sentence through HDCNN architecture and self-attention mechanism, respectively. Afterwards, the decoding module is to output the sequence of tags, thereby completing the automatic recognition of multiple target entities and their corresponding sentiment polarities. Finally, experimental results demonstrate that, without any domain specific features or prior information, our method surpasses the existing state-of-the-art methods, implying its feasibility and wide applicability.

Biomedical Knowledge Graphs Construction From Conditional Statements.

Conditions play an essential role in biomedical statements. However, existing biomedical knowledge graphs (BioKGs) only focus on factual knowledge, organized as a flat relational network of biomedical concepts. These BioKGs ignore the conditions of the facts being valid, which loses essential contexts for knowledge exploration and inference. We consider both facts and their conditions in biomedical statements and proposed a three-layered information-lossless representation of BioKG. The first layer has biomedical concept nodes, attribute nodes. The second layer represents both biomedical fact and condition tuples by nodes of the relation phrases, connecting to the subject and object in the first layer. The third layer has nodes of statements connecting to a set of fact tuples and/or condition tuples in the second layer. We transform the BioKG construction problem into a sequence labeling problem based on a novel designed tag schema. We design a Multi-Input Multi-Output sequence labeling model (MIMO) that learns from multiple input signals and generates proper number of multiple output sequences for tuple extraction. Experiments on a newly constructed dataset show that MIMO outperforms the existing methods. Further case study demonstrates that the BioKGs constructed provide a good understanding of the biomedical statements.

Prediction of students’ early dropout based on their interaction logs in online learning environment

ABSTRACTOnline learning has become more popular in higher education since it adds convenience and flexibility to students’ schedule. But, it has faced difficulties in the retention of the continuity of students and ensure continual growth in course. Dropout is a concerning factor in online course continuity. Therefore, it has sparked great interest among educators and researchers to offer many models and strategies to reduce online course students’ dropout by analyzing students’ behavior and their individual and academic information. However, online education platforms still face challenges of high dropout rate and the difficulty of accurate prediction to reduce it. The key aim of the present study is to construct a predictive model to early predict students who are at-risk of dropout. This model is useful to the course instructors to make effective and timely interventions. We have noticed that dropout prediction is basically a sequence labeling or time series prediction problem. For these reasons, we proposed two models; the Logistic Regression by adding a regularization term and the Input-Output Hidden Markov Model (IOHMM). Results showed that the proposed models achieved an accuracy of 84% compared to the baseline of Machine Learning Models for prediction of the students at-risk of dropping out.

DTranNER: biomedical named entity recognition with deep learning-based label-label transition model

BackgroundBiomedical named-entity recognition (BioNER) is widely modeled with conditional random fields (CRF) by regarding it as a sequence labeling problem. The CRF-based methods yield structured outputs of labels by imposing connectivity between the labels. Recent studies for BioNER have reported state-of-the-art performance by combining deep learning-based models (e.g., bidirectional Long Short-Term Memory) and CRF. The deep learning-based models in the CRF-based methods are dedicated to estimating individual labels, whereas the relationships between connected labels are described as static numbers; thereby, it is not allowed to timely reflect the context in generating the most plausible label-label transitions for a given input sentence. Regardless, correctly segmenting entity mentions in biomedical texts is challenging because the biomedical terms are often descriptive and long compared with general terms. Therefore, limiting the label-label transitions as static numbers is a bottleneck in the performance improvement of BioNER.ResultsWe introduce DTranNER, a novel CRF-based framework incorporating a deep learning-based label-label transition model into BioNER. DTranNER uses two separate deep learning-based networks: Unary-Network and Pairwise-Network. The former is to model the input for determining individual labels, and the latter is to explore the context of the input for describing the label-label transitions. We performed experiments on five benchmark BioNER corpora. Compared with current state-of-the-art methods, DTranNER achieves the best F1-score of 84.56% beyond 84.40% on the BioCreative II gene mention (BC2GM) corpus, the best F1-score of 91.99% beyond 91.41% on the BioCreative IV chemical and drug (BC4CHEMD) corpus, the best F1-score of 94.16% beyond 93.44% on the chemical NER, the best F1-score of 87.22% beyond 86.56% on the disease NER of the BioCreative V chemical disease relation (BC5CDR) corpus, and a near-best F1-score of 88.62% on the NCBI-Disease corpus.ConclusionsOur results indicate that the incorporation of the deep learning-based label-label transition model provides distinctive contextual clues to enhance BioNER over the static transition model. We demonstrate that the proposed framework enables the dynamic transition model to adaptively explore the contextual relations between adjacent labels in a fine-grained way. We expect that our study can be a stepping stone for further prosperity of biomedical literature mining.

Combining Multi-task Learning with Transfer Learning for Biomedical Named Entity Recognition

Abstract Multi-task learning approaches have shown significant improvements in different fields by training different related tasks simultaneously. The multi-task model learns common features among different tasks where they share some layers. However, it is observed that the multi-task learning approach can suffer performance degradation with respect to single task learning in some of the natural language processing tasks, specifically in sequence labelling problems. To tackle this limitation we formulate a simple but effective approach that combines multi-task learning with transfer learning. We use a simple model that comprises of bidirectional long-short term memory and conditional random field. With this simple model, we are able to achieve better F1-score compared to our single task and the multi-task models as well as state-of-the-art multi-task models.

SEML: A Semi-Supervised Multi-Task Learning Framework for Aspect-Based Sentiment Analysis

Aspect-Based Sentiment Analysis (ABSA) involves two sub-tasks, namely Aspect Mining (AM) and Aspect Sentiment Classification (ASC), which aims to extract the words describing aspects of a reviewed entity (e.g., a product or service) and analyze the expressed sentiments on the aspects. As AM and ASC can be formulated as a sequence labeling problem to predict the aspect or sentiment labels of each word in the review, supervised deep sequence learning models have recently achieved the best performance. However, these supervised models require a large number of labeled reviews which are very costly or unavailable, and they usually perform only one of the two sub-tasks, which limits their practical use. To this end, this paper proposes a SEmi-supervised Multi-task Learning framework (called SEML) for ABSA. SEML has three key features. (1) SEML applies Cross-View Training (CVT) to enable semi-supervised sequence learning over a small set of labeled reviews and a large set of unlabeled reviews from the same domain in a unified end-to-end architecture. (2) SEML solves the two sub-tasks simultaneously by employing three stacked bidirectional recurrent neural layers to learn the representations of reviews, in which the representations learned from different layers are fed into CVT, AM and ASC, respectively. (3) SEML develops a Moving-window Attentive Gated Recurrent Unit (MAGRU) for the three recurrent neural layers to enhance representation learning and prediction accuracy, as nearby contexts within a moving-window in a review can provide important semantic information for the prediction task in ABSA. Finally, we conduct extensive experiments on ABSA over four review datasets from the SemEval workshops. Experimental results show that SEML significantly outperforms the state-of-the-art models.

Applying a deep learning-based sequence labeling approach to detect attributes of medical concepts in clinical text

BackgroundTo detect attributes of medical concepts in clinical text, a traditional method often consists of two steps: named entity recognition of attributes and then relation classification between medical concepts and attributes. Here we present a novel solution, in which attribute detection of given concepts is converted into a sequence labeling problem, thus attribute entity recognition and relation classification are done simultaneously within one step.MethodsA neural architecture combining bidirectional Long Short-Term Memory networks and Conditional Random fields (Bi-LSTMs-CRF) was adopted to detect various medical concept-attribute pairs in an efficient way. We then compared our deep learning-based sequence labeling approach with traditional two-step systems for three different attribute detection tasks: disease-modifier, medication-signature, and lab test-value.ResultsOur results show that the proposed method achieved higher accuracy than the traditional methods for all three medical concept-attribute detection tasks.ConclusionsThis study demonstrates the efficacy of our sequence labeling approach using Bi-LSTM-CRFs on the attribute detection task, indicating its potential to speed up practical clinical NLP applications.

Family member information extraction via neural sequence labeling models with different tag schemes

BackgroundFamily history information (FHI) described in unstructured electronic health records (EHRs) is a valuable information source for patient care and scientific researches. Since FHI is usually described in the format of free text, the entire process of FHI extraction consists of various steps including section segmentation, family member and clinical observation extraction, and relation discovery between the extracted members and their observations. The extraction step involves the recognition of FHI concepts along with their properties such as the family side attribute of the family member concept.MethodsThis study focuses on the extraction step and formulates it as a sequence labeling problem. We employed a neural sequence labeling model along with different tag schemes to distinguish family members and their observations. Corresponding to different tag schemes, the identified entities were aggregated and processed by different algorithms to determine the required properties.ResultsWe studied the effectiveness of encoding required properties in the tag schemes by evaluating their performance on the dataset released by the BioCreative/OHNLP challenge 2018. It was observed that the proposed side scheme along with the developed features and neural network architecture can achieve an overall F1-score of 0.849 on the test set, which ranked second in the FHI entity recognition subtask.ConclusionsBy comparing with the performance of conditional random fields models, the developed neural network-based models performed significantly better. However, our error analysis revealed two challenging issues of the current approach. One is that some properties required cross-sentence inferences. The other is that the current model is not able to distinguish between the narratives describing the family members of the patient and those specifying the relatives of the patient’s family members.

CircSLNN: Identifying RBP-Binding Sites on circRNAs via Sequence Labeling Neural Networks.

The interactions between RNAs and RNA binding proteins (RBPs) are crucial for understanding post-transcriptional regulation mechanisms. A lot of computational tools have been developed to automatically predict the binding relationship between RNAs and RBPs. However, most of the methods can only predict the presence or absence of binding sites for a sequence fragment, without providing specific information on the position or length of the binding sites. Besides, the existing tools focus on the interaction between RBPs and linear RNAs, while the binding sites on circular RNAs (circRNAs) have been rarely studied. In this study, we model the prediction of binding sites on RNAs as a sequence labeling problem, and propose a new model called circSLNN to identify the specific location of RBP-binding sites on circRNAs. CircSLNN is driven by pretrained RNA embedding vectors and a composite labeling model. On our constructed circRNA datasets, our model has an average F 1 score of 0.790. We assess the performance on full-length RNA sequences, the proposed model outperforms previous classification-based models by a large margin.

Understanding what the users say in chatbots: A case study for the Vietnamese language

This paper11This paper is an improved and extended version of Tran and Luong. presents a study on understanding what the users say in chatbot systems: the situation where users input utterances bots would hopefully (1) detect intents and (2) recognize corresponding contexts implied by utterances. This helps bots better understand what users are saying, and act upon a much wider range of actions. To this end, we propose a framework which models the first task as a classification problem and the second one as a two-layer sequence labeling problem. The framework explores deep neural networks to automatically learn useful features at both character and word levels. We apply this framework to building a chatbot in a Vietnamese e-commerce domain to help retail brands better communicate with their customers. Experimental results on four newly-built datasets demonstrate that deep neural networks could be able to outperform strong conventional machine-learning methods. In detecting intents, we achieve the best F-measure of 82.32%. In extracting contexts, the proposed method yields promising F-measures ranging from 78% to 91% depending on specific types of contexts.

Hybrid neural conditional random fields for multi-view sequence labeling

In traditional machine learning, conditional random fields (CRF) is the mainstream probability model for sequence labeling problems. CRF considers the relation between adjacent labels other than decoding each label independently, and better performance is expected to achieve. However, there are few multi-view learning methods involving CRF which can be directly used for sequence labeling tasks. In this paper, we propose a novel multi-view CRF model to label sequential data, called MVCRF, which well exploits two principles for multi-view learning: consensus and complementary. We first use different neural networks to extract features from multiple views. Then, considering the consistency among the different views, we introduce a joint representation space for the extracted features and minimize the distance between the two views for regularization. Meanwhile, following the complementary principle, the features of multiple views are integrated into the framework of CRF. We train MVCRF in an end-to-end fashion and evaluate it on two benchmark data sets. The experimental results illustrate that MVCRF obtains state-of-the-art performance: F1 score 95.44% for chunking on CoNLL-2000, 95.06% for chunking and 96.99% for named entity recognition (NER) on CoNLL-2003.

Deep Cascade Multi-Task Learning for Slot Filling in Online Shopping Assistant

Slot filling is a critical task in natural language understanding (NLU) for dialog systems. State-of-the-art approaches treat it as a sequence labeling problem and adopt such models as BiLSTM-CRF. While these models work relatively well on standard benchmark datasets, they face challenges in the context of E-commerce where the slot labels are more informative and carry richer expressions. In this work, inspired by the unique structure of E-commerce knowledge base, we propose a novel multi-task model with cascade and residual connections, which jointly learns segment tagging, named entity tagging and slot filling. Experiments show the effectiveness of the proposed cascade and residual structures. Our model has a 14.6% advantage in F1 score over the strong baseline methods on a new Chinese E-commerce shopping assistant dataset, while achieving competitive accuracies on a standard dataset. Furthermore, online test deployed on such dominant E-commerce platform shows 130% improvement on accuracy of understanding user utterances. Our model has already gone into production in the E-commerce platform.

Compound Noun Decomposition by using Syllable-based Embedding and Deep Learning

기존의 복합명사 분해 알고리즘은 미등록어 단위명사들이 포함된 복합명사를 분해할 때 미등록어를 분리하기 어려운 문제가 발생한다. 이는 현실적으로 모든 고유명사, 신조어, 외래어 등의 모든 단위 명사를 사전에 등록하는 것은 불가능하다는 한계가 존재하기 때문이다. 이 문제를 해결하기 위하여 복합명사 분해 문제를 태그 열 부착(sequence labeling) 문제로 정의하고 음절 단위 임베딩과 딥러닝 기법을 이용하는 복합명사 분해 방법을 제안한다. 단위명사 사전을 구축하지 않고 미등록 단위명사를 인식하기 위하여 복합명사를 구성하는 각 음절들을 연속적인 벡터 공간에 표현하여 LSTM과 선형체인(linear-chain) CRF를 이용하는 방식으로 복합명사를 단위명사들로 분해한다.The existing compound noun decomposition algorithm has a problem that it is difficult to separate unregistered nouns when decomposing compound nouns that contain unregistered unit nouns in the dictionary. This is because there is a limit to the fact that it is impossible to register all unit nouns into the dictionary such as proper nouns, coined words, and foreign words in advance. In this paper, in order to solve this problem, compound noun decomposition problem is defined as tag sequence labeling problem and compound noun decomposition method to use syllable unit embedding and deep learning technique is proposed. In order to recognize unregistered unit nouns without constructing unit noun dictionary, compound nouns are decomposed into unit nouns by using LSTM and linear-chain CRF expressing each syllable that constitutes a compound noun in the continuous vector space.

CRFVoter: gene and protein related object recognition using a conglomerate of CRF-based tools

BackgroundGene and protein related objects are an important class of entities in biomedical research, whose identification and extraction from scientific articles is attracting increasing interest. In this work, we describe an approach to the BioCreative V.5 challenge regarding the recognition and classification of gene and protein related objects. For this purpose, we transform the task as posed by BioCreative V.5 into a sequence labeling problem. We present a series of sequence labeling systems that we used and adapted in our experiments for solving this task. Our experiments show how to optimize the hyperparameters of the classifiers involved. To this end, we utilize various algorithms for hyperparameter optimization. Finally, we present CRFVoter, a two-stage application of Conditional Random Field (CRF) that integrates the optimized sequence labelers from our study into one ensemble classifier.ResultsWe analyze the impact of hyperparameter optimization regarding named entity recognition in biomedical research and show that this optimization results in a performance increase of up to 60%. In our evaluation, our ensemble classifier based on multiple sequence labelers, called CRFVoter, outperforms each individual extractor’s performance. For the blinded test set provided by the BioCreative organizers, CRFVoter achieves an F-score of 75%, a recall of 71% and a precision of 80%. For the GPRO type 1 evaluation, CRFVoter achieves an F-Score of 73%, a recall of 70% and achieved the best precision (77%) among all task participants.ConclusionCRFVoter is effective when multiple sequence labeling systems are to be used and performs better then the individual systems collected by it.

Detection of abnormal behaviour for dementia sufferers using Convolutional Neural Networks.

In recent years, there is a rapid increase in the population of elderly people. However, elderly people may suffer from the consequences of cognitive decline, which is a mental health disorder that primarily affects cognitive abilities such as learning, memory, etc. As a result, the elderly people may get dependent on caregivers to complete daily life tasks. Detecting the early indicators of dementia before it gets worsen and warning the caregivers and medical doctors would be helpful for further diagnosis. In this paper, the problem of activity recognition and abnormal behaviour detection is investigated for elderly people with dementia. First of all, the paper presents a methodology for generating synthetic data reflecting on some behavioural difficulties of people with dementia given the difficulty of obtaining real-world data. Secondly, the paper explores Convolutional Neural Networks (CNNs) to model patterns in activity sequences and detect abnormal behaviour related to dementia. Activity recognition is considered as a sequence labelling problem, while abnormal behaviour is flagged based on the deviation from normal patterns. Moreover, the performance of CNNs is compared against the state-of-art methods such as Naïve Bayes (NB), Hidden Markov Models (HMMs), Hidden Semi-Markov Models (HSMM), Conditional Random Fields (CRFs). The results obtained indicate that CNNs are competitive with those state-of-art methods.

A deep learning method for named entity recognition in bidding document

Named entity recognition is a fundamental task of Natural Language Processing and belongs to the category of sequence labeling problems. In the text, the named entity is the main carrier of information, which is used to express the main content of the text. Accurately identifying these contents is essential for implementing various natural language processing techniques such as information extraction, information retrieval, machine translation, and question and answer system. Named entities in business documents contain a lot of important information that can bring significant business value to the business. In this paper, we propose the method of combining Bi-directional long-short-term memory network and conditional random field, combining n-gram features and character features, and introducing attention mechanism to identify the tenderee, bidding agent and project number three entities in the bidding documents. Compared with the LSTM, BiLSTM can obtain the context information better and extract more features. The CRF uses the features obtained by BiLSTM to decode and obtain the final labeling result. In the comparative experiment of the collected data sets of 20,000 bidding documents, the BiLSTM-CRF model proposed in this paper can produce better labeling effect than other models and meet our expectations.