Character-level Model Research Articles

Label-Aware Chinese Event Detection with Heterogeneous Graph Attention Network

Event Detection (ED) aims to recognize instances of specified types of event triggers in text. Different from English ED, Chinese ED suffers from the problem of word-trigger mismatch due to the uncertain word boundaries. Existing approaches injecting word information into character-level models have achieved promising progress to alleviate this problem, but they are limited by two issues. First, the interaction between characters and lexicon words is not fully exploited. Second, they ignore the semantic information provided by event labels. We thus propose a novel architecture named Label enhanced Heterogeneous Graph Attention Networks (L-HGAT). Specifically, we transform each sentence into a graph, where character nodes and word nodes are connected with different types of edges, so that the interaction between words and characters is fully reserved. A heterogeneous graph attention networks is then introduced to propagate relational message and enrich information interaction. Furthermore, we convert each label into a trigger-prototype-based embedding, and design a margin loss to guide the model distinguish confusing event labels. Experiments on two benchmark datasets show that our model achieves significant improvement over a range of competitive baseline methods.

Exploring Character-Level Deep Learning Models for POS Tagging in Assamese Language

The proposed research investigates a novel approach of character-level Long Short-Term Memory (LSTM) and Bidirectional Long Short-Term Memory (Bi-LSTM) for part-of-speech (POS) tagging in the Assamese language. The proposed work contributes to Natural Language Processing (NLP) by exploring these models’ ability to assign grammatical labels (POS tags) to individual words within Assamese sentences. The corpus encompasses 60,000 Assamese words and utilizes the LDCIL Assamese tagset. For training and testing, the corpus is divided into an 80:20 ratio where 80% of the corpus is used for training the models, and the rest of 20% is used for evaluation. The character-level LSTM model achieves an accuracy of 92.80%, while the character-level Bi-LSTM model surpasses it by achieving an accuracy of 93.36%. The performance of the proposed research outperforms the existing research works in the Assamese language. The results of this work broaden the understanding of POS tagging in Assamese, offering valuable findings that could be applied to other languages with similar characteristics

Applying a Character-Level Model to a Short Arabic Dialect Sentence: A Saudi Dialect as a Case Study

Arabic dialect identification (ADI) has recently drawn considerable interest among researchers in language recognition and natural language processing fields. This study investigated the use of a character-level model that is effectively unrestricted in its vocabulary, to identify fine-grained Arabic language dialects in the form of short written text. The Saudi dialects, particularly the four main Saudi dialects across the country, were considered in this study. The proposed ADI approach consists of five main phases, namely dialect data collection, data preprocessing and labelling, character-based feature extraction, deep learning character-based model/classical machine learning character-based models, and model evaluation performance. Several classical machine learning methods, including logistic regression, stochastic gradient descent, variations of the naive Bayes models, and support vector classification, were applied to the dataset. For the deep learning, the character convolutional neural network (CNN) model was adapted with a bidirectional long short-term memory approach. The collected data were tested under various classification tasks, including two-, three- and four-way ADI tasks. The results revealed that classical machine learning algorithms outperformed the CNN approach. Moreover, the use of the term frequency–inverse document frequency, combined with a character n-grams model ranging from unigrams to four-grams achieved the best performance among the tested parameters.

Automatic dottization of Arabic text (Rasms) using deep recurrent neural networks

• First work on automatically adding dots to Arabic text without dots (i.e., Rasms). • Automatic adding dots using deep recurrent neural networks . • Evaluated on four different text corpora. • CER rates ranging from 2.0% to 5.5% on independent test sets. Arabic letters in their early stages were only shapes ( Rasm ) without dots. Dots were added later to ease reading and reduce ambiguity. Thereafter, diacritics were introduced for phonetic guidance, mainly for nonnative speakers. Many studies have been conducted to automatically diacritize Arabic texts using machine learning techniques . However, to the best of our knowledge, automatically adding dots to Arabic Rasms has not been reported in the literature. In this work, we present the automatic addition of dots to Arabic Rasms using deep recurrent neural networks. Different design choices were explored, including the use of character sequences and word sequences as tokens. The presented techniques were evaluated on four diverse publicly available datasets. Character-level models with stacked BiGRU architecture outperformed all the other architectures with character error rates ranging from 2.0% to 5.5% and dottization error rates ranging from 4.2% to 11.0% on independent test sets.

An effective approach for Arabic document classification using machine learning

Arabic text classification is one application of Natural Language Processing (NLP). It has been used to analyze and categorize Arabic text. Analyzing text has become an essential part of our lives because of the increasing number of text data which makes text classification a big data problem. Arabic text classification systems become significant to maintain vital information in many domains such as education, and health sector, and public services. In the presented research work, the Arabic text classification model is developed using various algorithms namely Multinomial Naïve Bayesian (MNB), Bernoulli Naïve Bayesian (BNB), Stochastic Gradient Descent (SGD), Logistic Regression (LR), Support vector classifier (SVC), Linear SVC, and convolutional neural networks (CNN). These algorithms have been implemented utilizing the Al-Khaleej dataset. The experiments are carried out with various representation models and it is observed that CNN with character level model outperforms others. The result of CNN exceeds the state-of-the-art machine learning method with an accuracy equal to 98. The presented methods will be useful in different domains, particularly on social media.

Integrating Character-level and Word-level Representation for Affect in Arabic Tweets

Affect tasks, which range from sentiment polarity classification to finer grained sentiment strength and emotional intensity detection, have become of increasing interest due to the vast amount of user-generated content and advanced learning models. Word representation models have been leveraged effectively within a variety of natural language processing tasks. However, these models are not always effective in the context of social media. When dealing with social media posts in Arabic, the use of Arabic dialects needs to be considered. Although using informal text to train word-level models can lead to the identification of words that convey the same meaning, these models are unable to capture the full extent of the words that are used in the real world due to out-of-vocabulary (OOV) words. The inability to identify such words is one of the main limitations of word-level models. One approach of overcoming OOV is through the use of character-level embeddings as they can effectively learn the vectors of word parts or character n-grams. This study uses a combination of character-level and word-level models to identify the most effective methods by which affective Arabic words in tweets can be represented semantically and morphologically. We evaluate our generated models and the proposed method by integrating them in a supervised learning framework that was used for a range of affect tasks and other related tasks. Our findings reveal that the developed models surpassed the performance of state-of-the-art Arabic pre-trained word embeddings over eight datasets. In addition, our models enhance previous state-of-the-art outcomes on tasks involving Arabic emotion intensity, outperforming the top-systems that used advanced ensemble learning models and several additional features.

Referent graph embedding model for name entity recognition of Chinese car reviews

Name entity recognition (NER) is one of the most basic tasks for extracting information from Internet text. Chinese NER remains a major challenge due to the language complexity. Although researchers have recently used domain knowledge to embed word-level information into the deep learning models to deal with the Chinese NER, they have not considered the global interdependence between word-level information, i.e., the entities in the same document should be semantically related to each other. In addition, domain knowledge often cannot be used efficiently due to the presence of irregular expressions in the Internet text, such as abbreviations and aliases. In this paper, we propose a referent graph embedding model for the NER, specifically concentrating on the Chinese car review. First, domain knowledge is used to generate character-level candidate entities and model the global interdependence between these entities based on the referent graph model. Second, the latest BERT-based character vectors and the character-level candidate entities are jointly embedded into the deep learning model to perform the NER. Last, Chinese car reviews are collected and labeled for use as the experimental dataset. The experimental results demonstrate the efficiency and effectiveness of the proposed model for the Chinese car NER task compared with the other start-of-the-art models.

Position-Aware Deep Character-Level CTR Prediction for Sponsored Search

Predicting the click-through rate of an advertisement is a critical component of online advertising platforms. In sponsored search, the click-through rate estimates the probability that a displayed advertisement is clicked by a user after she submits a query to the search engine. Commercial search engines typically rely on machine learning models trained with a large number of features to make such predictions. This inevitably requires a lot of engineering efforts to define, compute, and select the appropriate features. In this paper, we propose two novel approaches (one working at character level and the other working at word level) that use deep convolutional neural networks to predict the click-through rate of a query-advertisement pair. Specifically, the proposed architectures consider as input only the textual content appearing in a query-advertisement pair and the page position at which the advertisement appears on the search result page of the query, and produce as output a click-through rate prediction. By comparing the character-level model with the word-level model, we show that language representation can be learnt from scratch at character level when trained on enough data. Through extensive experiments using billions of query-advertisement pairs of a popular commercial search engine, we demonstrate that both approaches significantly outperform a baseline model built on well-selected text features and a state-of-the-art word2vec-based approach. We also show the importance of the position feature in the proposed approaches in improving the prediction accuracy. When combining the predictions of the deep models introduced in this study with the prediction of the model in production of the same commercial search engine, we significantly improve the accuracy and the calibration of the click-through rate prediction of the production system. We also show the potential of leveraging the CTR prediction of the proposed deep learning models for query-ad relevance modeling and query-ad matching tasks in sponsored search.

CHARM: An Improved Method for Chinese Precoding and Character-Level Embedding

The numerical transformation of text is a key step in natural language processing tasks, among which the word embedding model is the most representative. However, the word embedding model is insufficient in representing unregistered and low-frequency words, and the character-level embedding model makes up for it. Most Chinese character-level models focus on the independent use of Chinese character features such as strokes, radicals, and pinyin, or the shallow correlations between some features, while the inherent correlations among different features such as pronunciation, glyph, stroke order, and word frequency are not fully utilized. Through the statistical analyses of various features of Chinese characters, this paper proposes a precoding method based on Character Helix Alternative Representation Model (CHARM), which can realize the reversible mapping of Chinese characters or words to English-like sequences, and the advantage of this method is verified in the text classification test based on TextCNN. Experimental results on three test sets show that the model performs well, and can be a replacement character-level corpus for the original Chinese text.

Deep Graph-Based Character-Level Chinese Dependency Parsing

Character-level Chinese dependency parsing has been a concern of several studies that naturally handle word segmentation, POS (Part of Speech) tagging and dependency parsing jointly in an end-to-end way. Previous work mostly concentrates on a transition-based framework for this task because of its easy adaption, which is extremely important when feature representation relies heavily on the decoding strategy, particularly under the traditional statistical setting. Recently, on the one hand, sophisticated deep neural networks and deep contextualized word representations have greatly weakened the dependence between feature representation and decoding. On the other hand, (first-order) graph-based models, especially the biaffine parsers, are straightforward for dependency parsing, and meanwhile they can yield competitive parsing performance. In this paper, we make a comprehensive investigation of the deep graph-based character-level dependency parsing for Chinese. We start from an extension of a standard graph-based biaffine parser, and then exploit Chinese BERT as well as our improved encoders based on transformers to enhance the character-level dependency parsing model. We conduct a series of experiments on the Chinese benchmark datasets, showing the performances of various graph-based character-level models and analyzing the advantages of the character-level dependency parsing under the deep neural setting.

Novel Linguistic Steganography Based on Character-Level Text Generation

With the development of natural language processing, linguistic steganography has become a research hotspot in the field of information security. However, most existing linguistic steganographic methods may suffer from the low embedding capacity problem. Therefore, this paper proposes a character-level linguistic steganographic method (CLLS) to embed the secret information into characters instead of words by employing a long short-term memory (LSTM) based language model. First, the proposed method utilizes the LSTM model and large-scale corpus to construct and train a character-level text generation model. Through training, the best evaluated model is obtained as the prediction model of generating stego text. Then, we use the secret information as the control information to select the right character from predictions of the trained character-level text generation model. Thus, the secret information is hidden in the generated text as the predicted characters having different prediction probability values can be encoded into different secret bit values. For the same secret information, the generated stego texts vary with the starting strings of the text generation model, so we design a selection strategy to find the highest quality stego text from a number of candidate stego texts as the final stego text by changing the starting strings. The experimental results demonstrate that compared with other similar methods, the proposed method has the fastest running speed and highest embedding capacity. Moreover, extensive experiments are conducted to verify the effect of the number of candidate stego texts on the quality of the final stego text. The experimental results show that the quality of the final stego text increases with the number of candidate stego texts increasing, but the growth rate of the quality will slow down.

Character-level neural network model based on Nadam optimization and its application in clinical concept extraction

Clinical concept extraction aims to quickly and effectively extract available data from complex and diverse clinical information, which is a crucial task for medical diagnosis using electronic medical records. Named entity recognition (NER) accurately marks essential information in clinical records based on the characteristics of the target entity, providing a way to extract clinical concepts. In the clinical concept extraction task, the existing methods are not satisfactory to obtain accurate labelling results in the face of large-scale and complex clinical information. To solve this problem, we improve and optimize a named entity recognition method based on the LSTM-CRF model. First, the improved deep neural network model uses two optional configurations of Convolutional Neural Network (CNN) and Bidirectional Long Short-Term Memory (BLSTM) to achieve character-level representation. Then the BLSTM layer obtains the context information of the target word, and the Conditional Random Field (CRF) gives constraints to ensure the standardization of the label. On this basis, Nadam is used to optimize the training process of the network. The experimental results show that our new method has an F1 score of 84.61 on the public dataset of 2010 i2b2/VA concept extraction task, which exceeds the LSTM-CRF model. And the recall of 85.41 is ahead of all the methods evaluated on this data set.

Enhanced character-level deep convolutional neural networks for cardiovascular disease prediction

BackgroundElectronic medical records contain a variety of valuable medical information for patients. So, when we are able to recognize and extract risk factors for disease from EMRs of patients with cardiovascular disease (CVD), and are able to use them to predict CVD, we have the ability to automatically process clinical texts, resulting in an improved accuracy of supporting doctors for the clinical diagnosis of CVD. In the case where CVD is becoming more worldwide, predictive CVD based on EMRs has been studied by many researchers to address this important aspect of improving diagnostic efficiency.MethodsThis paper proposes an Enhanced Character-level Deep Convolutional Neural Networks (EnDCNN) model for cardiovascular disease prediction.ResultsOn the manually annotated Chinese EMRs corpus, our risk factor identification extraction model achieved 0.9073 of F-score, our prediction model achieved 0.9516 of F-score, and the prediction result is better than the most previous methods.ConclusionsThe character-level model based on text region embedding can well map risk factors and their labels as a unit into a vector, and downsampling plays a crucial role in improving the training efficiency of deep CNN. What’s more, the shortcut connections with pre-activation used in our model architecture implements dimension-matching free in training.

Character-level text classification via convolutional neural network and gated recurrent unit

Text categorization, or text classification, is one of key tasks for representing the semantic information of documents. Traditional deep leaning models for text categorization are generally time-consuming on large scale datasets due to slow convergence rate or heavily rely on the pre-trained word vectors. Motivated by fully convolutional networks in the field of image processing, we introduce fully convolutional layers to substantially reduce the number of parameters in the text classification model. A character-level model for short text classification, integrating convolutional neural network, bidirectional gated recurrent unit, highway network with the fully connected layers, is proposed to capture both the global and the local textual semantics at the fast convergence speed. Furthermore, In addition, error minimization extreme learning machine is incorporated into the proposed model to improve the classification accuracy further. Extensive experiments show that our approach achieves the state-of-the-art performance compared with the existing methods on the large scale text datasets.

Part-of-Speech Tagging Using Multiview Learning

In natural language processing, character-level representations are vector representations of the particular character. Character-level representations have recently focused on enriching subword information by stacking deep neural models. Ideally, applications of several character-level representations can help capture different aspects of the subword information. However, this approach has often failed in the past, mainly because of the nature of traditionally used simple concatenation models. In this study, we explore different character-level modeling techniques. During the learning process, long short-term memory-based character representations can introduce different views for a part-of-speech tagger. After investigating two previously reported techniques, we propose two additional extended methods: (1) a multihead-attention character-level representation for capturing several aspects of subword information, and (2) an optimal structure for training two different character-level embeddings based on joint learning. We evaluate our results on the part-of-speech (POS) tagging dataset of the Conference on Natural Language Learning (CoNLL) 2018 shared task in universal dependencies. We show that our method substantially improves POS tagging results for many morphologically rich languages where the character information should be considered more substantially. Moreover, we compare the performance of our model with recent state-of-the-art POS taggers, which are trained with language models such as Bidirectional Encoder Representations from Transformers (BERT) and Deep Contextualized Word Representations (ELMo); our multiview tagger shows better results for nine languages. The proposed character model shows significant improvements in Ancient Greek, with average gains of 8.89 points in accuracy compared to the previous word representation model. Therefore, our empirical experiments indicate that character-level representations are more important than word representations for morphologically rich languages in terms of performance.

Learning unsupervised contextual representations for medical synonym discovery

ObjectivesAn important component of processing medical texts is the identification of synonymous words or phrases. Synonyms can inform learned representations of patients or improve linking mentioned concepts to medical ontologies. However, medical synonyms can be lexically similar (“dilated RA” and “dilated RV”) or dissimilar (“cerebrovascular accident” and “stroke”); contextual information can determine if 2 strings are synonymous. Medical professionals utilize extensive variation of medical terminology, often not evidenced in structured medical resources. Therefore, the ability to discover synonyms, especially without reliance on training data, is an important component in processing training notes. The ability to discover synonyms from models trained on large amounts of unannotated data removes the need to rely on annotated pairs of similar words. Models relying solely on non-annotated data can be trained on a wider variety of texts without the cost of annotation, and thus may capture a broader variety of language.Materials and MethodsRecent contextualized deep learning representation models, such as ELMo (Peters et al., 2019) and BERT, (Devlin et al. 2019) have shown strong improvements over previous approaches in a broad variety of tasks. We leverage these contextualized deep learning models to build representations of synonyms, which integrate the context of surrounding sentence and use character-level models to alleviate out-of-vocabulary issues. Using these models, we perform unsupervised discovery of likely synonym matches, which reduces the reliance on expensive training data.ResultsWe use the ShARe/CLEF eHealth Evaluation Lab 2013 Task 1b data to evaluate our synonym discovery method. Comparing our proposed contextualized deep learning representations to previous non-neural representations, we find that the contextualized representations show consistent improvement over non-contextualized models in all metrics.ConclusionsOur results show that contextualized models produce effective representations for synonym discovery. We expect that the use of these representations in other tasks would produce similar gains in performance.

합성곱 신경망을 이용한 음절 피처맵 생성 및 감성 분석

Sentiment analysis is a technique for analyzing subjective attitudes, opinions, and emotions of people in a text. When conducting sentiment analysis understanding the structure of the language used in the text is very important. In this paper, we noted the characteristics of the Korean language that a syllable consists of three elements: Initial sound, Intermediate sound, Final sound. Thus, we compare sentiment classification models that can reflect the characteristics. These models, which expresses syllables by combination of initial sound, intermediate sound, final sound. One of them is improved in classification accuracy over the existing character-level model. But not only that, This model is robust to the misspelled word compared to Syllable-level model and Morph-level model because it uses a character-level representation of a sentence as input.

Empirical study on character level neural network classifier for Chinese text

Character level models are drawing attention recently. A number of these models have been proposed and shown successful in Natural Language Processing tasks. While most of the models are experimented mainly on English, or other alphabetic languages, a number of problems arise when they applied these models to non-alphabetic language such as Chinese. In this study, we investigated the problems encountered when transferring these models to the Chinese and put forward some solutions. We propose a double embedding neural network model that is also character level and consists of both CNN and RNN with two separate embeddings. The model is applied to a fundamental Natural Language Processing task, text classification. Experiment results conducted on the Chinese corpus demonstrated that our character level neural network model performs just as well as or better than those word level classification models. Our model is able to reach 95.9% accuracy on a Chinese Fudan news dataset, which outperforms the state-of-the-art models.

A Character-Level BiLSTM-CRF Model With Multi-Representations for Chinese Event Detection

Using the word as a basic unit may undermine Chinese event detection model’s performance because of the inaccurate word boundaries generated by segmentation tools. Besides, word embeddings are contextual independent and cannot handle the polysemy of event triggers, which may prevent us from obtaining the desired performance. To address these issues, we propose a BiLSTM-CRF (Bidirectional Long Short-Term Memory Conditional Random Field) model using contextualized representations, which regards event detection task as a character-level sequence labeling problem and uses contextualized representations to disambiguate event triggers. Experiments show that our proposed method sets a new state-of-the-art, which proves Chinese characters could replace words for the Chinese event detection task. Besides, using contextualized representation reduces the false positive case, which verifies that this kind of representation could remedy the weakness of the word embedding technique. Based on the results, we believe that character-level models are worth exploring in the future.

Wide-scope biomedical named entity recognition and normalization with CRFs, fuzzy matching and character level modeling.

We present a system for automatically identifying a multitude of biomedical entities from the literature. This work is based on our previous efforts in the BioCreative VI: Interactive Bio-ID Assignment shared task in which our system demonstrated state-of-the-art performance with the highest achieved results in named entity recognition. In this paper we describe the original conditional random field-based system used in the shared task as well as experiments conducted since, including better hyperparameter tuning and character level modeling, which led to further performance improvements. For normalizing the mentions into unique identifiers we use fuzzy character n-gram matching. The normalization approach has also been improved with a better abbreviation resolution method and stricter guideline compliance resulting in vastly improved results for various entity types. All tools and models used for both named entity recognition and normalization are publicly available under open license.