BiLSTM-CRF Model Research Articles

특허 기술 개체명 식별을 위한 단어 임베딩 기반 Bi-LSTM CRF 연구

최근 4차 산업혁명의 시대에서 다양한 신기술들이 등장함에 따라 가치 있는 신기술들에 대한 권리를 주장하기 위한 목적으로 특허 출원 및 발명 건수가 급증하고 있다. 따라서 선행 특허의 기술 검색 및 핵심 기술추출 등의 연구들이 수행되고 있다. 특히 특허 문헌에 등장하는 기술 개체명은 특허 기술에 대한 분석 정확도 및 검색 재현율 성능 향상에 중요한 역할을 한다. 본 논문에서는 이러한 기술 개체명 식별을 위해 단어 임베딩 기반 특허 기술 개체명 식별 방법을 제안한다. 다양한 도메인 데이터를 대상으로 기술 개체명 식별 실험결과 제안 방법이 가장 우수함을 확인하였다. 따라서 제안 모델을 통해 다양한 특허 기술 분야에 포함된 기술 개체명 식별 및 기계 학습용 한국어 사전 자동 구축을 위한 기반 연구로 활용 가능한 것으로 기대된다.

Multi-Layout Unstructured Invoice Documents Dataset: A Dataset for Template-Free Invoice Processing and Its Evaluation Using AI Approaches

The daily transaction of an organization generates a vast amount of unstructured data such as invoices and purchase orders. Managing and analyzing unstructured data is a costly affair for the organization. Unstructured data has a wealth of hidden valuable information. Extracting such insights automatically from unstructured documents can significantly increase the productivity of an organization. Thus, there is a huge demand to develop a tool that can automate the extraction of key fields from unstructured documents. Researchers have used different approaches for extracting key fields, but the lack of annotated and high-quality datasets is the biggest challenge. Existing work in this area has used standard and custom datasets for extracting key fields from unstructured documents. Still, the existing datasets face some serious challenges, such as poor-quality images, domain-related datasets, and a lack of data validation approaches to evaluate data quality. This work highlights the detailed process flow for end-to-end key fields extraction from unstructured documents. This work presents a high-quality, multi-layout unstructured invoice documents dataset assessed with a statistical data validation technique. The proposed multi-layout unstructured invoice documents dataset is highly diverse in invoice layouts to generalize key field extraction tasks for unstructured documents. The proposed multi-layout unstructured invoice documents dataset is evaluated with various feature extraction techniques such as Glove, Word2Vec, FastText, and AI approaches such as BiLSTM and BiLSTM-CRF. We also present the comparative analysis of feature extraction techniques and AI approaches on the proposed multi-layout unstructured invoice document dataset. We attained the best results with BiLSTM-CRF model.

A small samples training framework for deep Learning-based automatic information extraction: Case study of construction accident news reports analysis

Knowledge management is crucial for construction safety management. Widely collected and well-organized safety-related documents are recognized to be significant in raising the workers' security awareness and then to prevent hazards and accidents. To improve document processing efficiency, automatic information extraction plays an important role. However, currently, automatic information extraction modeling requires large scale training datasets. It is a big challenge for the engineering industry, especially for the fields which heavily rely on the experts’ knowledge. Limited data sources, and high time and labor costs make it not practical to establish a large-scale dataset. This work proposed a natural language data augmentation-based small samples training framework for automatic information extraction modeling. With the designed cross combination-based text data augmentation algorithm, the deep neural network can be employed to build up automatic information extraction models without large-scale raw data and manual annotations. Characters semantic coding is employed to avoid word segmentation and make sure that the framework can be utilized in different writing language systems. The BiLSTM-CRF model is adopted as the detection core to conduct character classification. Through a case study of two independent accident news report datasets analysis, the proposed framework has been validated. A reliable and robust automatic information extraction model can be established, even though with small samples training.

BE-BLC: BERT-ELMO-Based Deep Neural Network Architecture for English Named Entity Recognition Task

One of the most important factors which considerably affects the quality of the neural sequence labeling model is the selection and encoding of input features to generate rich semantic and grammatical word representation vectors. Recently, pre-training text encoders have significantly ameliorated the performance on various Natural Language Processing (NLP) tasks. In this paper, we introduce a deep neural network model to address a challenging task of the sequence labeling problem, the task of named entity recognition. The model consists of three sub-networks to fully exploit current Google’s representations of the most powerful NLP transfer learning BERT and ELMO model, as well as the traditional state-of-the-art BiLSTM-CRF (BLC) model. The model outperforms the state-of-the-art systems on CoNLL-2003 by achieving a 95.56% F1-score and getting results comparable to the previous best-performing system models on OntoNotes 5.0 by achieving an 89.21% F1-score without using any external knowledge.

Research on key technologies for fault knowledge acquisition of power communication equipment

Abstract With the development of Energy Internet and Smart Grid, power communication equipment has gradually become an important pillar supporting the information transmission. However, when equipment failure occurs, more and more attention has been paid to the rapid realization of equipment fault research and judgment from a large amount of equipment fault information. Based on the fault data of power communication equipment, a fault model of power communication equipment is designed in this paper. It combines BERT and BiLSTM-CRF model for knowledge extraction, and constructs the equipment fault knowledge graph to realize the visual display of power communication equipment. Through this model, fault knowledge extraction can be realized accurately, and a visual model of fault analysis can be formed, which can effectively improve the fault research speed and work efficiency of maintenance personnel.

Aspect-based sentiment analysis in Chinese based on mobile reviews for BiLSTM-CRF

Aiming at the problem that the Aspect-based sentiment analysis in Chinese has low recognition rate due to many steps, this paper proposes an improved BiLSTM-CRF model based on combine the Chinese character vector and Chinese words position feature, which can extract attribute words and sentiment words jointly simultaneously, while extracting Polarity judges of sentiment words. Experiments show that the improved model improves the precision rate by 9.2% 13.32%, recall rate improves 0.48% 21.29%, F-measure improves 7.33% 15.74% compared with Conditional Random Fields (CRF) model and Long Short Term Memory (LSTM) model on the self-built 6357 mobile reviews dataset. The experimental results show that the model improves the accuracy of Aspect-based sentiment analysis and can effectively obtain the information required by users need in evaluation texts.

Nested Named Entity Recognition via an Independent-Layered Pretrained Model

When an entity contains one or more entities, these particular entities are referred to as nested entities. The Layered BiLSTM-CRF model can use multiple BiLSTM layers to identify nested entities. However, as the number of layers increases, the number of labels that the model can learn decreases, and it may not even predict any entities, thereby causing the model to stop stacking. Furthermore, the model will be constrained by the one-way propagation of information from the lower layer to the higher layer. The incorrect entities extracted by the outer layer will affect the performance of the inner layer. We propose a novel neural network for nested named entity recognition (NER) that dynamically stacks flat NER layers to address these issues. Each flat NER layer captures contextual information based on a pretrained model with more robust feature extraction capabilities. The model parameters of a flat NER layer and its input are entirely independent. The input of each layer is all of the word representations generated by the input sequence through the embedding layer. The independent input ensures that different flat NER layers will not be interfered with by other flat NER layers during model training and testing to reduce error propagation. Experiments show that our model obtains F1 scores of 76.9%, 78.1%, and 78.0% on the ACE2004, ACE2005, and GENIA datasets, respectively.

Construction of knowledge graph of forest musk deer based on BiLSTM-CRF model and DPA method

Forest musk deer is a national key protected species, but due to overhunting and habitat destruction, wild musk deer has disappeared in large areas. Constructing a knowledge graph in the field of forest musk deer can standardize the relevant knowledge in the field of forest musk deer and lay a good foundation for wildlife protection and ecological construction. In order to construct the knowledge map of the forest musk deer, the entities were further subdivided first, and the BiLSTM-CRF model was used to identify the entities of the BIO-labeled data. After analysis, it was found that the model used in the experiment performed better. Secondly, the Dependency Parsing Analysis method was used to identify the relationship, and the relationship between the entities was smoothly extracted.Finally, the Neoj4 graph database was used to realize the storage and visualization of the knowledge graph of the forest musk deer. This research transforms the unstructured text in the field of forest musk deer into structured data. Through the systematic sorting of relevant knowledge in the field of forest musk deer, it can provide wildlife protection, artificial breeding of forest musk deer, intelligent knowledge services and other aspects some new technical means in the field of forest musk deer.

A Knowledge Graph Based Framework in Relationship Modelling and Real-time Monitoring of Market Participants

Various types of relationships among participants exist in the foreign exchange market, such as transaction, credit relations inside the market and subcompany, investing and shareholding relations outside the market. In the market with complex relationships, an incident happened in one institution may have “butterfly effect”. To monitor systemic financial risks, we integrate various types of data and establish a Knowledge Graph describing the relationships between market participants. Based on the Knowledge Graph, we design a framework to measure the impact of various events on the entire market by connecting the internal market relationships to external market related incidents. By using the BiLSTM-CRF model, we extract the relevant entities in the market news with high accuracy and measure the impact on the market with graph related indicators such as closeness and betweenness centralities. In addition, to use external unstructured data, we use a document-level event extraction method to analyse these unstructured datasets and extract new events and relationships from them to dynamically update the Knowledge Graph.

Causality extraction based on self-attentive BiLSTM-CRF with transferred embeddings

Causality extraction from natural language texts is a challenging open problem in artificial intelligence. Existing methods utilize patterns, constraints, and machine learning techniques to extract causality, heavily depending on domain knowledge and requiring considerable human effort and time for feature engineering. In this paper, we formulate causality extraction as a sequence labeling problem based on a novel causality tagging scheme. On this basis, we propose a neural causality extractor with the BiLSTM-CRF model as the backbone, named SCITE (Self-attentive BiLSTM-CRF wIth Transferred Embeddings), which can directly extract cause and effect without extracting candidate causal pairs and identifying their relations separately. To address the problem of data insufficiency, we transfer contextual string embeddings, also known as Flair embeddings, which are trained on a large corpus in our task. In addition, to improve the performance of causality extraction, we introduce a multihead self-attention mechanism into SCITE to learn the dependencies between causal words. We evaluate our method on a public dataset, and experimental results demonstrate that our method achieves significant and consistent improvement compared to baselines.

POISE: Efficient Cross-Domain Chinese Named Entity Recognization via Transfer Learning

To improve the performance of deep learning methods in case of a lack of labeled data for entity annotation in entity recognition tasks, this study proposes transfer learning schemes that combine the character to be the word to convert low-resource data symmetry into high-resource data. We combine character embedding, word embedding, and the embedding of the label features using high- and low-resource data based on the BiLSTM-CRF model, and perform the feature-transfer and parameter-sharing tasks in two domains of the BiLSTM network to annotate with zero resources. Before transfer learning, we must first calculate the label similarity between two different domains and select the label features with large similarity for feature transfer mapping. All training parameters of the source domain in the model are shared during the BiLSTM network processing and CRF layer. In addition, we also use the method of combining characters and words to reduce the problem of word segmentation across domains and reduce the error rate in label mapping. The results of experiments show that in terms of the overall F1 score, the proposed model without supervision was superior by 9.76 percentage points to the general parametric shared transfer learning method, and by 9.08 and 12.38 percentage points, respectively, to two recent high–low resource learning methods. The proposed scheme improves performance in terms of transfer learning between the high- and low-resource data and can identify the predicted data in the target domain.

How to Construct a Power Knowledge Graph with Dispatching Data?

Knowledge graph is a kind of semantic network for information retrieval. How to construct a knowledge graph that can serve the power system based on the behavior data of dispatchers is a hot research topic in the area of electric power artificial intelligence. In this paper, we propose a method to construct the dispatch knowledge graph for the power grid. By leveraging on dispatch data from the power domain, this method first extracts entities and then identifies dispatching behavior relationship patterns. More specifically, the method includes three steps. First, we construct a corpus of power dispatching behaviors by semi-automated labeling. And then, we propose a model, called the BiLSTM-CRF model, to extract entities and identify the dispatching behavior relationship patterns. Finally, we construct a knowledge graph of power dispatching data. The knowledge graph provides an underlying knowledge model for automated power dispatching and related services and helps dispatchers perform better power dispatch knowledge retrieval and other operations during the dispatch process.

Entity perception of Two-Step-Matching framework for public opinions

Entity perception of ambiguous user comments is a critical problem of target identification for huge amount of public opinions. In this paper, a Two-Step-Matching method is proposed to identify the precise target entity from multiple entities mentioned. Firstly, potential entities are extracted by BiLSTM-CRF model and characteristic words by TF-IDF model from public comments. Secondly, the first matching is implemented between potential entities and an official business directory by Jaro–Winkler distance algorithm. Then, in order to find the precise one, an industry-characteristic dictionary is developed into the second matching process. The precise entity is identified according to the count of characteristic words matching to industry-characteristic dictionary. In addition, associated rate (global indicator) and accuracy rate (sample indicator) are defined for evaluation of matching accuracy. The results for three data sets of public opinions about major public health events show that the highest associated rate and accuracy rate arrive at 0.93 and 0.95, averagely enhanced by 32% and 30% above the case of using the first matching process alone. This framework provides the method to find the true target entity of really wanted expression from public opinions.

Efficiency Improvement of Function Point-Based Software Size Estimation With Deep Learning Model

Software cost estimation is crucial to software management, which has received considerable attention from both industry and academia. Software size is an important metric that forms the cornerstone of software cost estimation. The function point has been proven to be a useful software size unit for size estimation and has been successfully implemented in many countries. However, in current practice, the rule of function point size method is complicated and performed manually. Consequently, it is costly in both time and resources spent to apply these methods, especially in the scenario of large-scale software development in the industry. In this paper, a deep learning-based named entity recognition (NER) model was designed in place of manual function point recognition. In particular, a BiLSTM-CRF model was trained on previously labeled requirements in the industry to classify the function point type of new requirements in the same domain. The proposed method was verified on 29 real projects provided by our industry partner. A comparative experiment was designed for the quantitative evaluation of efficiency improvement of the proposed NER model aided function point estimation. The result suggests that, for the NER model, the precision and F1 of the BiLSTM-CRF-based function point analysis on test samples achieved 94.5% and 80.3%, respectively. Moreover, the improvement in the efficiency of the software size estimation process achieved an average of 38.6%, which is a significant enhancement for the function point-based software size estimation.

Robust Named Entity Recognition with Truecasing Pretraining

Although modern named entity recognition (NER) systems show impressive performance on standard datasets, they perform poorly when presented with noisy data. In particular, capitalization is a strong signal for entities in many languages, and even state of the art models overfit to this feature, with drastically lower performance on uncapitalized text. In this work, we address the problem of robustness of NER systems in data with noisy or uncertain casing, using a pretraining objective that predicts casing in text, or a truecaser, leveraging unlabeled data. The pretrained truecaser is combined with a standard BiLSTM-CRF model for NER by appending output distributions to character embeddings. In experiments over several datasets of varying domain and casing quality, we show that our new model improves performance in uncased text, even adding value to uncased BERT embeddings. Our method achieves a new state of the art on the WNUT17 shared task dataset.

Research on Chinese medical named entity recognition based on collaborative cooperation of multiple neural network models.

Medical named entity recognition (NER) in Chinese electronic medical records (CEMRs) has drawn much research attention, and plays a vital prerequisite role for extracting high-value medical information. In 2018, China Health Information Processing Conference (CHIP2018) organized a medical NER academic competition aiming to extract three types of malignant tumor entity from CEMRs. Since the three types of entity are highly domain-specific and interdependency, extraction of them cannot be achieved with a single neural network model. Based on comprehensive study of the three types of entity and the entity interdependencies, we propose a collaborative cooperation of multiple neural network models based approach, which consists of two BiLSTM-CRF models and a CNN model. In order to tackle the problem that target scene dataset is small and entity distributions are sparse, we introduce non-target scene datasets and propose sentence-level neural network model transfer learning. Based on 30,000 real-world CEMRs, we pre-train medical domain-specific Chinese character embeddings with word2vec, GloVe and ELMo, and apply them to our approach respectively to validate effects of pre-trained language models in Chinese medical NER. Also, as control experiments, we apply Gated Recurrent Unit to our approach. Finally, our approach achieves an overall F1-score of 87.60%, which is the state-of-the-art performance to the best of our knowledge. In addition, our approach has won the champion of the medical NER academic competition organized by 2019 China Conference on Knowledge Graph and Semantic Computing, which proves the outstanding generalization ability of our approach.

A Residual BiLSTM Model for Named Entity Recognition

As one of the most powerful neural networks, Long Short-Term Memory (LSTM) is widely used in natural language processing (NLP) tasks. Meanwhile, the BiLSTM-CRF model is one of the most popular models for named entity recognition (NER), and many state-of-the-art models for NER are based on it. In this paper, we propose a new residual BiLSTM model and perform it with a conditional random field (CRF) layer together on NER tasks. Based on the most popular BiLSTM-CRF model, we replace the BiLSTM with our residual BiLSTM blocks to encode words or characters. We evaluate our model on Chinese and English datasets. We utilize both word2vec and BERT to generate word or character vectors. Furthermore, we conduct experiments to compare the performance of NER by using different structures of residual blocks. The experimental results show that our model can improve the performance of both Chinese and English NER effectively without introducing any external knowledge.

A named entity recognition model based on ensemble learning

Knowledge Graph has gradually become one of core drivers advancing the Internet and AI in recent years, while there is currently no normal knowledge graph in the field of agriculture. Named Entity Recognition (NER), one important step in constructing knowledge graphs, has become a hot topic in both academia and industry. With the help of the Bidirectional Long Short-Term Memory Network (Bi-LSTM) and Conditional Random Field (CRF) model, we introduce a method of ensemble learning, and implement a named entity recognition model ELER. Our model achieves good results for the CoNLL2003 data set, the accuracy and F1 value in the best experimental results are respectively improved by 1.37% and 0.7% when compared with the BiLSTM-CRF model. In addition, our model achieves an F1 score of 91% for the agricultural data set AgriNER2018, which proves the validity of ELER model for small agriculture sample data sets and lays a foundation for the construction of agricultural knowledge graphs.

Investigating Self-Attention Network for Chinese Word Segmentation

Neural network has become the dominant method for Chinese word segmentation. Most existing models cast the task as sequence labeling, using BiLSTM-CRF for representing the input, and making output predictions. Recently, attention-based sequence models have emerged as a highly competitive alternative to LSTMs, which allow better running speed by parallelization of computation. We investigate self-attention network (SAN) for Chinese word segmentation, making comparisons between BiLSTM-CRF models. In addition, the influence of contextualized character embeddings is investigated using BERT, and a method is proposed for integrating word information into SAN segmentation. Results show that SAN gives highly competitive results compared with BiLSTMs, with BERT, and word information further improving segmentation for in-domain, and cross-domain segmentation. Our final models give the best results for 6 heterogenous domain benchmarks.

A self-attention based neural architecture for Chinese medical named entity recognition.

The combination of medical field and big data has led to an explosive growth in the volume of electronic medical records (EMRs), in which the information contained has guiding significance for diagnosis. And how to extract these information from EMRs has become a hot research topic. In this paper, we propose an ELMo-ET-CRF model based approach to extract medical named entity from Chinese electronic medical records (CEMRs). Firstly, a domain-specific ELMo model is fine-tuned on a common ELMo model with 4679 raw CEMRs. Then we use the encoder from Transformer (ET) as our model's encoder to alleviate the long context dependency problem, and the CRF is utilized as the decoder. At last, we compare the BiLSTM-CRF and ET-CRF model with word2vec and ELMo embeddings to CEMRs respectively to validate the effectiveness of ELMo-ET-CRF model. With the same training data and test data, the ELMo-ET-CRF outperforms all the other mentioned model architectures in this paper with 85.59% F1-score, which indicates the effectiveness of the proposed model architecture, and the performance is also competitive on the CCKS2019 leaderboard.