Character-level Convolutional Neural Networks Research Articles

Biomedical named entity recognition based on multi-cross attention feature fusion.

Currently, in the field of biomedical named entity recognition, CharCNN (Character-level Convolutional Neural Networks) or CharRNN (Character-level Recurrent Neural Network) is typically used independently to extract character features. However, this approach does not consider the complementary capabilities between them and only concatenates word features, ignoring the feature information during the process of word integration. Based on this, this paper proposes a method of multi-cross attention feature fusion. First, DistilBioBERT and CharCNN and CharLSTM are used to perform cross-attention word-char (word features and character features) fusion separately. Then, the two feature vectors obtained from cross-attention fusion are fused again through cross-attention to obtain the final feature vector. Subsequently, a BiLSTM is introduced with a multi-head attention mechanism to enhance the model's ability to focus on key information features and further improve model performance. Finally, the output layer is used to output the final result. Experimental results show that the proposed model achieves the best F1 values of 90.76%, 89.79%, 94.98%, 80.27% and 88.84% on NCBI-Disease, BC5CDR-Disease, BC5CDR-Chem, JNLPBA and BC2GM biomedical datasets respectively. This indicates that our model can capture richer semantic features and improve the ability to recognize entities.

Automatic analysis of public health service text based on character level convolutional neural network

The text information processing technology of public health service is one of the hot research topics at present. To improve the defects of public health service texts, such as inaccurate word segmentation, spelling errors and professional vocabulary understanding, this study designed a character-level deep neural network model on the characteristics of public health service texts. In this model, the bidirectional short and short time memory and the attention pooling operation layer are introduced to make the model better classify the text according to the context. In addition, counter perturbation is introduced in this study to improve the robustness and generalization ability of the model, thus improving its classification effect. The performance verification results show that the proposed model has better classification performance on the public health service text data set. The anti-disturbance samples generated by the model are all in the range of 0–0.2 when WMD deviation degree is measured, while most of the other methods are in the range of 0.4–0.6. The experimental object of this study is ultrasonic examination data. The experimental results show that the automatic analysis model of public health service text based on character level convolutional neural network constructed in this study has excellent accuracy and convergence speed, and has excellent performance in the classification of public health service text in different subject areas.

Enhancing Generalizability in Biomedical Entity Recognition: Self-Attention PCA-CLS Model.

One of the primary tasks in the early stages of data mining involves the identification of entities from biomedical corpora. Traditional approaches relying on robust feature engineering face challenges when learning from available (un-)annotated data using data-driven models like deep learning-based architectures. Despite leveraging large corpora and advanced deep learning models, domain generalization remains an issue. Attention mechanisms are effective in capturing longer sentence dependencies and extracting semantic and syntactic information from limited annotated datasets. To address out-of-vocabulary challenges in biomedical text, the PCA-CLS (Position and Contextual Attention with CNN-LSTM-Softmax) model combines global self-attention and character-level convolutional neural network techniques. The model's performance is evaluated on eight distinct biomedical domain datasets encompassing entities such as genes, drugs, diseases, and species. The PCA-CLS model outperforms several state-of-the-art models, achieving notable F1-scores, including 88.19% on BC2GM, 85.44% on JNLPBA, 90.80% on BC5CDR-chemical, 87.07% on BC5CDR-disease, 89.18% on BC4CHEMD, 88.81% on NCBI, and 91.59% on the s800 dataset.

CNN-Fusion: An effective and lightweight phishing detection method based on multi-variant ConvNet

Phishing scams are increasing as the technical skills and costs of phishing attacks diminish, emphasizing the need for rapid, precise, and low-cost prevention measures. Based on a character-level convolutional neural network (CNN), we present CNN-Fusion, an effective and lightweight phishing URL detection method. Our basic idea is to deploy multiple variants of one-layer CNN with various-sized kernels in parallel to extract multi-level features. Observing that differences between phishing and benign URLs might exhibit a strong spatial correlation, we choose SpatialDropout1D, making the model more robust and preventing it from memorizing the training data. To further reduce the probability of errors that may cause by irrelevant or noisy features, we apply a max-over time pooling technique over the feature map to pick only the most important feature. Finally, the model is evaluated using five publicly available datasets containing 1.85 million phishing and benign URLs. Other than that, we assess the model against AI adversarial attacks, known as “Offensive AI.” Compared to existing methods, experiments demonstrate that our approach enjoys advantages in 5 times less training time and much more in memory consumption, achieving an average accuracy above 99% on five different datasets as well as on AI-generated malicious attacks.

MOJI: Character-level convolutional neural networks for Malicious Obfuscated JavaScript Inspection

JavaScript malware is one of the major threats to web security. A big challenge in detecting such malicious JavaScript is obfuscation, which transforms a program code into a harder-to-understand representation while preserving its original functionality. Many malicious JavaScript detection methods perform code abstraction and prior feature extraction to uncover the functionality hidden by obfuscation. However, such preprocessing steps significantly limit the detectors’ efficiency in practical situations. This paper presents Malicious Obfuscated JavaScript Inspector (MOJI), a novel method for malicious JavaScript detection, which requires no code abstraction or prior feature extraction. Instead, our detector directly accepts a sequence of characters in JavaScript code as an input and outputs its maliciousness score. Specifically, we design a character-level convolutional neural network consisting mainly of several 1D convolutional layers and fully connected layers. We evaluate the proposed method on a dataset composed of 24,000 JavaScript codes and show that our method outperforms existing malicious JavaScript detectors in terms of both detection performance and running time. We also provide an analysis of the effect of additional obfuscation on the same dataset. Our results indicate that MOJI is far more robust to obfuscation than the existing methods and commercial antivirus software.

Application of Deep Learning Technology in the Recommendation System of Constitutional and Constitutional Cases

Due to the huge system in the field of constitution, the content of cases is complicated, and the number of documents related to constitutional cases is huge. It is incredibly tough to find the information that suits your demands in such a large volume of constitutional documents. The artificial neural network-based deep learning method has made significant progress. Deep learning has been used in the disciplines of computer image and video processing, as well as natural language processing, and has outperformed typical machine learning methods. To that purpose, the content-based recommendation algorithm in this work is improved using deep learning approaches. This paper primarily covers the following topics: (1) the current research status of deep learning technology in recommender systems is reviewed, and the theoretical knowledge of deep neural networks is introduced. (2) A text classification model OCCNN based on character-level convolutional neural network is proposed to solve the problem of document classification of massive cases. A semantic similarity calculation model TF-W2V based on word frequency and word vector is also proposed to solve the problem of semantic similarity calculation of cases. (3) The two methods are tested, and the experimental results show that the accuracy is much higher than that of the conventional method. The model can recommend cases that meet the needs of users according to the requirements put forward by users in the process of handling cases.

Classification of FinTech Patents by Machine Learning and Deep Learning Reveals Trends of FinTech Development in China

The development of the financial industry and its integration with information technology have promoted FinTech innovation. China is a major contributor to FinTech innovation, but few studies have systematically summarized FinTech innovation and development in China from the perspective of patents. This lacuna is attributable to the lack of a generally accepted classification of FinTech patents and the unavailability of classified Chinese FinTech patent text data. To fill this research gap, we developed a classification of Chinese FinTech patents and manually annotated a set of patent texts to train machine learning and deep learning models to classify massive Chinese patent application data and identify different types of FinTech innovations. Among the evaluated models, the character-level convolutional neural network (CNN) model and BERT model classified FinTech innovation most accurately. We used the character-level CNN to classify 20,529 Chinese FinTech patent applications from 2013 to 2020. The classified dataset was used to briefly analyze the history of FinTech innovation development in China and its future prospects.

Memory augmented recurrent neural networks for de-novo drug design.

A recurrent neural network (RNN) is a machine learning model that learns the relationship between elements of an input series, in addition to inferring a relationship between the data input to the model and target output. Memory augmentation allows the RNN to learn the interrelationships between elements of the input over a protracted length of the input series. Inspired by the success of stack augmented RNN (StackRNN) to generate strings for various applications, we present two memory augmented RNN-based architectures: the Neural Turing Machine (NTM) and the Differentiable Neural Computer (DNC) for the de-novo generation of small molecules. We trained a character-level convolutional neural network (CNN) to predict the properties of a generated string and compute a reward or loss in a deep reinforcement learning setup to bias the Generator to produce molecules with the desired property. Further, we compare the performance of these architectures to gain insight to their relative merits in terms of the validity and novelty of the generated molecules and the degree of property bias towards the computational generation of de-novo drugs. We also compare the performance of these architectures with simpler recurrent neural networks (Vanilla RNN, LSTM, and GRU) without an external memory component to explore the impact of augmented memory in the task of de-novo generation of small molecules.

Short text automatic scoring system based on BERT-BiLSTM model

Aiming at the problems of sparse features, polysemy of one word and less context related information in short text automatic scoring, a short text automatic scoring model based on bidirectional encoder representations from transformers - bidirectional long short-term memory (BERT-BiLSTM) is proposed. Firstly, the large-scale corpus is pre-trained with bidirectional encoder representations from transformers (BERT) language model to acquire the semantic features of the general language. Then the semantic features of short text and the semantics of keywords in a specific context are acquired through the short text data for the pre-fine tuning downstream specific tasks set pre-fined by BERT. And then the deep-seated context dependency is captured through bidirectional long short-term memory (BiLSTM). Finally, the obtained feature vectors are input into Softmax regression model for automatic scoring. The experimental results show that compared with other benchmark models of convolutional neural networks(CNN), character-level CNN (CharCNN), long short-term memory (LSTM) and BERT, the short text automatic scoring model based on BERT-BiLSTM achieves the best average value of quadratic weighted kappa coefficient.

MSCAT: A Machine Learning Assisted Catalog of Metabolomics Software Tools.

The bottleneck for taking full advantage of metabolomics data is often the availability, awareness, and usability of analysis tools. Software tools specifically designed for metabolomics data are being developed at an increasing rate, with hundreds of available tools already in the literature. Many of these tools are open-source and freely available but are very diverse with respect to language, data formats, and stages in the metabolomics pipeline. To help mitigate the challenges of meeting the increasing demand for guidance in choosing analytical tools and coordinating the adoption of best practices for reproducibility, we have designed and built the MSCAT (Metabolomics Software CATalog) database of metabolomics software tools that can be sustainably and continuously updated. This database provides a survey of the landscape of available tools and can assist researchers in their selection of data analysis workflows for metabolomics studies according to their specific needs. We used machine learning (ML) methodology for the purpose of semi-automating the identification of metabolomics software tool names within abstracts. MSCAT searches the literature to find new software tools by implementing a Named Entity Recognition (NER) model based on a neural network model at the sentence level composed of a character-level convolutional neural network (CNN) combined with a bidirectional long-short-term memory (LSTM) layer and a conditional random fields (CRF) layer. The list of potential new tools (and their associated publication) is then forwarded to the database maintainer for the curation of the database entry corresponding to the tool. The end-user interface allows for filtering of tools by multiple characteristics as well as plotting of the aggregate tool data to monitor the metabolomics software landscape.

Using Machine Learning to Improve Public Reporting on U.S. Government Contracts

The U.S. government procures more than $500 billion annually in goods and services on public contracts, which it classifies using a hierarchical product and service taxonomy. Classification serves several purposes, including transparency in the use of taxpayer funding; reporting, tracing, and segmenting government expenditures; budgeting; and forecasting. Government acquisition personnel have historically performed these classifications manually, resulting in a process that is time-consuming and error-prone and offers limited visibility into government purchases. The problem faced is not unique to the public sector and is common across retail, manufacturing, and healthcare, among other settings. Using almost 4 million historical data records on governmental purchases, we fit a series of classifiers and demonstrate (a) superior performance when explicitly modeling the hierarchical structure of information domains through the use of top-down strategies and (b) the effectiveness of character-level convolutional neural networks when textual inputs are terse and contain irregularities such as abnormal character combinations and misspellings, which are common in government contracts. Our machine learning models are embedded in multiple software applications, including a web application that we developed, used by federal government personnel and other contracting professionals.

Crowdsourced social media and mobile phone signaling data for disaster impact assessment: A case study of the 8.8 Jiuzhaigou earthquake

Rapid and accurate disaster impact assessment is crucial for disaster emergency decision-making. The perception and behavioral characteristics of people derived from crowdsourced data, such as social media and mobile phone signaling data, can provide information sources for disaster assessment and assist disaster emergency management. This topic has attracted extensive attention from scholars. However, these data types are incomplete and have wide-ranging quality, which makes it difficult to evaluate the disaster. Social media data contain many public descriptions of disasters, but the spatial distribution is uneven. Mobile phone signaling data cover a wide range and are spatiotemporal continuous, but semantic information is lacking. Thus, this paper develops a disaster impact assessment method that integrates social media data and mobile signaling data based on a classification of social media data with a character-level convolutional neural network (Char-CNN), anomaly detection in mobile signaling data, and feature mining of human activity. The result shows that social media and mobile signaling data complement each other well. The proposed method improves the assessment accuracy and provides detailed information on the spatial distribution of disasters. The results of this study provide reference for the combination of different sources of information for disaster response decisions.

Research on the Performance of Character Convolutional Neural Network in Different Text Encoding Formats

With the continuous progress and development of society, text classification has become an important task in the field of text data mining and text value exploration. Compared with existing text classification technology, deep learning technology has many advantages, such as high accuracy and effective feature extraction. This paper is mainly based on the character convolutional neural network to study the classification performance of different texts, hoping to provide experience for related research work. Through research, it is found that character-level convolutional neural network deep learning technology can achieve Chinese text classification more effectively.

The Impact of Transforming Unstructured Data into Structured Data on a Churn Prediction Model for Loan Customers

With various structured data, such as the company size, loan balance, and savings accounts, the voice of customer (VOC), which is text data containing contact history and counseling details was analyzed in this study. To analyze unstructured data, the term frequency-inverse document frequency (TF-IDF) analysis, semantic network analysis, sentiment analysis, and a convolutional neural network (CNN) were implemented. A performance comparison of the models revealed that the predictive model using the CNN provided the best performance with regard to predictive power, followed by the model using the TF-IDF, and then the model using semantic network analysis. In particular, a character-level CNN and a word-level CNN were developed separately, and the character-level CNN exhibited better performance, according to an analysis for the Korean language. Moreover, a systematic selection model for optimal text mining techniques was proposed, suggesting which analytical technique is appropriate for analyzing text data depending on the context. This study also provides evidence that the results of previous studies, indicating that individual customers leave when their loyalty and switching cost are low, are also applicable to corporate customers and suggests that VOC data indicating customers’ needs are very effective for predicting their behavior.

Text Classification on Large Scale Chinese News Corpus using Character-level Convolutional Neural Network

In view of the current problems in the field of Chinese news, such as chaotic news data and lack of automatic and efficient management, most of the existing Chinese text classification methods take word characteristics as the basic unit of text representation, but ignore the beneficial performance of character characteristics. In this paper, an automatic classification method of Chinese news based on character-level convolutional neural networks(char-CNN) is proposed. In order to compare with traditional models and deep learning models, we constructed a large scale Chinese news corpus to show that character-level convolutional neural networks could achieve the most advanced or competitive results.

Delayed Combination of Feature Embedding in Bidirectional LSTM CRF for NER

Named Entity Recognition (NER) plays a vital role in natural language processing (NLP). Currently, deep neural network models have achieved significant success in NER. Recent advances in NER systems have introduced various feature selections to identify appropriate representations and handle Out-Of-the-Vocabulary (OOV) words. After selecting the features, they are all concatenated at the embedding layer before being fed into a model to label the input sequences. However, when concatenating the features, information collisions may occur and this would cause the limitation or degradation of the performance. To overcome the information collisions, some works tried to directly connect some features to latter layers, which we call the delayed combination and show its effectiveness by comparing it to the early combination. As feature encodings for input, we selected the character-level Convolutional Neural Network (CNN) or Long Short-Term Memory (LSTM) word encoding, the pre-trained word embedding, and the contextual word embedding and additionally designed CNN-based sentence encoding using a dictionary. These feature encodings are combined at early or delayed position of the bidirectional LSTM Conditional Random Field (CRF) model according to each feature’s characteristics. We evaluated the performance of this model on the CoNLL 2003 and OntoNotes 5.0 datasets using the F1 score and compared the delayed combination model with our own implementation of the early combination as well as the previous works. This comparison convinces us that our delayed combination is more effective than the early one and also highly competitive.

An Effective Phishing Detection Model Based on Character Level Convolutional Neural Network from URL

Phishing is the easiest way to use cybercrime with the aim of enticing people to give accurate information such as account IDs, bank details, and passwords. This type of cyberattack is usually triggered by emails, instant messages, or phone calls. The existing anti-phishing techniques are mainly based on source code features, which require to scrape the content of web pages, and on third-party services which retard the classification process of phishing URLs. Although the machine learning techniques have lately been used to detect phishing, they require essential manual feature engineering and are not an expert at detecting emerging phishing offenses. Due to the recent rapid development of deep learning techniques, many deep learning-based methods have also been introduced to enhance the classification performance. In this paper, a fast deep learning-based solution model, which uses character-level convolutional neural network (CNN) for phishing detection based on the URL of the website, is proposed. The proposed model does not require the retrieval of target website content or the use of any third-party services. It captures information and sequential patterns of URL strings without requiring a prior knowledge about phishing, and then uses the sequential pattern features for fast classification of the actual URL. For evaluations, comparisons are provided between different traditional machine learning models and deep learning models using various feature sets such as hand-crafted, character embedding, character level TF-IDF, and character level count vectors features. According to the experiments, the proposed model achieved an accuracy of 95.02% on our dataset and an accuracy of 98.58%, 95.46%, and 95.22% on benchmark datasets which outperform the existing phishing URL models.

Machine learning applied in natural language processing

Machine learning applied in natural language processing

Joint Character-Level Convolutional and Generative Adversarial Networks for Text Classification

With the continuous renewal of text classification rules, text classifiers need more powerful generalization ability to process the datasets with new text categories or small training samples. In this paper, we propose a text classification framework under insufficient training sample conditions. In the framework, we first quantify the texts by a character-level convolutional neural network and input the textual features into an adversarial network and a classifier, respectively. Then, we use the real textual features to train a generator and a discriminator so as to make the distribution of generated data consistent with that of real data. Finally, the classifier is cooperatively trained by real data and generated data. Extensive experimental validation on four public datasets demonstrates that our method significantly performs better than the comparative methods.

Subword Attentive Model for Arabic Sentiment Analysis

Social media data is unstructured data where these big data are exponentially increasing day to day in many different disciplines. Analysis and understanding the semantics of these data are a big challenge due to its variety and huge volume. To address this gap, unstructured Arabic texts have been studied in this work owing to their abundant appearance in social media Web sites. This work addresses the difficulty of handling unstructured social media texts, particularly when the data at hand is very limited. This intelligent data augmentation technique that handles the problem of less availability of data are used. This article has proposed a novel architecture for hand Arabic words classification and understands based on convolutional neural networks (CNNs) and recurrent neural networks. Moreover, the CNN technique is the most powerful for the analysis of Arabic tweets and social network analysis. The main technique used in this work is character-level CNN and a recurrent neural network stacked on top of one another as the classification architecture. These two techniques give 95% accuracy in the Arabic texts dataset.