Neural Networks For Text Classification Research Articles

A detailed analysis of the interpretability of Convolutional Neural Networks for text classification

Abstract Convolutional Neural Networks (CNNs) have become ubiquitous in many NLP tasks. However, understanding its process is still an area with much to be done. In this paper, we introduce a method to study the interpretability of CNNs when used for text classification. More specifically, we work on the interpretability of the convolutional filters in the context of sentiment analysis. The framework used in this paper has allowed the understanding of the mechanics of the network when applied to the task of sentiment analysis. The results of the experiments reveal how certain parts of speech (POS) tags are more relevant than others to the classification of a sentence. Furthermore, we also observed a preference for shorter $n$-grams when classifying negative sentiment sequences. Additionally, we detected a certain amount of redundancy among the convolutional filters, thus allowing us to conclude that smaller architectures would have worked for this particular task. This was achieved by computing the relevant metrics to measure the influence of the pertinent property on the desired class. In our case, the concepts treated were POS tags that carry semantic information and properties were related to the convolutional filters.

Graph neural networks for text classification: a survey

Text Classification is the most essential and fundamental problem in Natural Language Processing. While numerous recent text classification models applied the sequential deep learning technique, graph neural network-based models can directly deal with complex structured text data and exploit global information. Many real text classification applications can be naturally cast into a graph, which captures words, documents, and corpus global features. In this survey, we bring the coverage of methods up to 2023, including corpus-level and document-level graph neural networks. We discuss each of these methods in detail, dealing with the graph construction mechanisms and the graph-based learning process. As well as the technological survey, we look at issues behind and future directions addressed in text classification using graph neural networks. We also cover datasets, evaluation metrics, and experiment design and present a summary of published performance on the publicly available benchmarks. Note that we present a comprehensive comparison between different techniques and identify the pros and cons of various evaluation metrics in this survey.

Quantum self-attention neural networks for text classification

Quantum self-attention neural networks for text classification

Optimizer algorithms and convolutional neural networks for text classification

<span lang="EN-US">Lately, deep learning has improved the algorithms and the architectures of several natural language processing (NLP) tasks. In spite of that, the performance of any deep learning model is widely impacted by the used optimizer algorithm; which allows updating the model parameters, finding the optimal weights, and minimizing the value of the loss function. Thus, this paper proposes a new convolutional neural network (CNN) architecture for text classification (TC) and sentiment analysis and uses it with various optimizer algorithms in the literature. Actually, in NLP, and particularly for sentiment classification concerns, the need for more empirical experiments increases the probability of selecting the pertinent optimizer. Hence, we have evaluated various optimizers on three types of text review datasets: small, medium, and large. Thereby, we examined the optimizers regarding the data amount and we have implemented our CNN model on three different sentiment analysis datasets so as to binary label text reviews. The experimental results illustrate that the adaptive optimization algorithms Adam and root mean square propagation (RMSprop) have surpassed the other optimizers. Moreover, our best CNN model which employed the RMSprop optimizer has achieved 90.48% accuracy and surpassed the state-of-the-art CNN models for binary sentiment classification problems.</span>

Semantic Template-based Convolutional Neural Network for Text Classification

We propose a semantic template-based distributed representation for the convolutional neural network called Semantic Template-based Convolutional Neural Network (STCNN) for text categorization that imitates the perceptual behavior of human comprehension. STCNN is a highly automatic approach that learns semantic templates that characterize a domain from raw text and recognizes categories of documents using a semantic-infused convolutional neural network that allows a template to be partially matched through a statistical scoring system. Our experiment results show that STCNN effectively classifies documents in about 140,000 Chinese news articles into predefined categories by capturing the most prominent and expressive patterns and achieves the best performance among all compared methods for Chinese topic classification. Finally, the same knowledge can be directly used to perform a semantic analysis task.

Young Peoples Attention to Health and Wellness after the Normalization of the Epidemic Situation

Nowadays, Covid-19 prevention and control measures have a large effect on society. In this social context, some young people are experiencing various degrees of physical and psychological problems, which prompts them to seek things that can provide them with health and psychological comfort. In todays era of developed social media, wellness has become the object they seek and promote. Considering the removal of Covid-19 prevention and control measures in China, as well as the psychological state of contemporary young people, this article explores the psychological state of young people during and after the pandemic, as well as their level of attention towards health preservation. The paper collects and analyzes comments with relevant tags from multiple platforms and discusses the psychological state of young people and their attention toward health preservation before and after the end of pandemic prevention and control measures. After preliminary data analysis, this paper manually annotated the data to train a neural network for text classification. The research results show that young people displayed psychological problems on social media before and after the pandemic, and their attention to health preservation remained high.

Hybrid photonic deep convolutional residual spiking neural networks for text classification.

Spiking neural networks (SNNs) offer powerful computation capability due to its event-driven nature and temporal processing. However, it is still limited to shallow structure and simple tasks due to the training difficulty. In this work, we propose a deep convolutional residual spiking neural network (DCRSNN) for text classification tasks. In the DCRSNN, the feature extraction is achieved via a convolution SNN with residual connection, using the surrogate gradient direct training technique. Classification is performed by a fully-connected network. We also suggest a hybrid photonic DCRSNN, in which photonic SNNs are used for classification with a converted training method. The accuracy of hard and soft reset methods, as well as three different surrogate functions, were evaluated and compared across four different datasets. Results indicated a maximum accuracy of 76.36% for MR, 91.03% for AG News, 88.06% for IMDB and 93.99% for Yelp review polarity. Soft reset methods used in the deep convolutional SNN yielded slightly better accuracy than their hard reset counterparts. We also considered the effects of different pooling methods and observation time windows and found that the convergence accuracy achieved by convolutional SNNs was comparable to that of convolutional neural networks under the same conditions. Moreover, the hybrid photonic DCRSNN also shows comparable testing accuracy. This work provides new insights into extending the SNN applications in the field of text classification and natural language processing, which is interesting for the resources-restrained scenarios.

Two End-to-End Quantum-Inspired Deep Neural Networks for Text Classification

In linguistics, the uncertainty of context due to polysemy is widespread, which attracts much attention. Quantum-inspired complex word embedding based on Hilbert space plays an important role in natural language processing (NLP), which fully leverages the similarity between quantum states and word tokens. A word containing multiple meanings could correspond to a single quantum particle which may exist in several possible states, and a sentence could be analogous to the quantum system where particles interfere with each other. Motivated by quantum-inspired complex word embedding, interpretable complex-valued word embedding (ICWE) is proposed to design two end-to-end quantum-inspired deep neural networks (ICWE-QNN and CICWE-QNN representing convolutional complex-valued neural network based on ICWE) for binary text classification. They have the proven feasibility and effectiveness in the application of NLP and can solve the problem of text information loss in CE-Mix [1] model caused by neglecting the important linguistic features of text, since linguistic feature extraction is presented in our model with deep learning algorithms, in which gated recurrent unit (GRU) extracts the sequence information of sentences, attention mechanism makes the model focus on important words in sentences and convolutional layer captures the local features of projected matrix. The model ICWE-QNN can avoid random combination of word tokens and CICWE-QNN fully considers textual features of the projected matrix. Experiments conducted on five benchmarking classification datasets demonstrate our proposed models have higher accuracy than the compared traditional models including CaptionRep BOW, DictRep BOW and Paragram-Phrase, and they also have great performance on F1-score. Eespecially, CICWE-QNN model has higher accuracy than the quantum-inspired model CE-Mix as well for four datasets including SST, SUBJ, CR and MPQA. It is a meaningful and effictive exploration to design quantum-inspired deep neural networks to promote the performance of text classification.

Improved Recurrent Neural Networks for Text Classification and Dynamic Sylvester Equation Solving

Improved Recurrent Neural Networks for Text Classification and Dynamic Sylvester Equation Solving

Automatic software vulnerability classification by extracting vulnerability triggers

AbstractVulnerability classification is a significant activity in software development and software maintenance. Natural Language Processing (NLP) techniques, which utilize the descriptions in public repositories, are widely used in automatic software vulnerability classification. However, vulnerability descriptions are ordinarily short and contain many technical terms, making them difficult for machines to automatically comprehend. In this paper, we present an approach based on vulnerability triggers to automatically classify vulnerabilities. First, we extract vulnerability triggers with Bert Question and Answer (Bert Q&A). Then, we use Recurrent Convolutional Neural Networks for Text classification (TextRCNN) to classify vulnerabilities based on Common Weakness Enumeration (CWE). We statistically perform an analysis of vulnerability triggers and comprehensively evaluate the classification performance of our approach on a set of 4769 prelabeled vulnerability entries, as well as compare it with state‐of‐the‐art vulnerability classification approaches. Experiment results show that our approach can achieve a F1‐measure of 95% on extraction and 80.8% on classification.

An Improved Convolutional Neural Network for Text Classification

This paper studies the text classification based on deep learning. Aiming at the problem of over fitting and training time consuming of CNN text classification model, a SDCNN model is constructed based on sparse dropout convolutional neural network. Experimental results show that, compared with CNN, SDCNN further improves the classification performance of the model, and its classification accuracy and precision can reach 98.96% and 85.61%, respectively, indicating that SDCNN has more advantages in text classification problems.

A Tensor Space Model-Based Deep Neural Network for Text Classification

Most text classification systems use machine learning algorithms; among these, naïve Bayes and support vector machine algorithms adapted to handle text data afford reasonable performance. Recently, given developments in deep learning technology, several scholars have used deep neural networks (recurrent and convolutional neural networks) to improve text classification. However, deep learning-based text classification has not greatly improved performance compared to that of conventional algorithms. This is because a textual document is essentially expressed as a vector (only), albeit with word dimensions, which compromises the inherent semantic information, even if the vector is (appropriately) transformed to add conceptual information. To solve this ‘loss of term senses’ problem, we develop a concept-driven deep neural network based upon our semantic tensor space model. The semantic tensor used for text representation features a dependency between the term and the concept; we use this to develop three deep neural networks for text classification. We perform experiments using three standard document corpora, and we show that our proposed methods are superior to both traditional and more recent learning methods.

Convolutional Recurrent Neural Networks for Text Classification

Recurrent neural network (RNN) and convolutional neural network (CNN) are two prevailing architectures used in text classification. Traditional approaches combine the strengths of these two networks by straightly streamlining them or linking features extracted from them. In this article, a novel approach is proposed to maintain the strengths of RNN and CNN to a great extent. In the proposed approach, a bi-directional RNN encodes each word into forward and backward hidden states. Then, a neural tensor layer is used to fuse bi-directional hidden states to get word representations. Meanwhile, a convolutional neural network is utilized to learn the importance of each word for text classification. Empirical experiments are conducted on several datasets for text classification. The superior performance of the proposed approach confirms its effectiveness.

An intelligent work order classification model for government service based on multi-label neural network

The classification of work orders for livelihood service play an important role in the e-government system, which goal is to push these work orders to the corresponding processing department. Although many studies have realized the classification of work orders automatically, these work orders are either related to many subjects or related to many departments. Therefore, the problem of work order classification is not a single label classification problem, but a multi-label classification problem. In view of the continuous development of deep neural networks, the goal of this paper is to modify a MultilabelMarginLoss function and design a MultilabelCrossEntropyLoss function to realize the multi-classification system of work orders, which can push the work orders to the related departments(one or more departments) automatically according to their appeal contents (consisting of short texts). We perform deep neural network for text classification including TextCNN, TextRCNN, TextRNN, TextRNN_Att, FastText and DPCNN on the real government hotline data set and compare the performances of these models. The results reveal that TextCNN and TextRCNN have a more stable performance on different loss functions, especially on the MultilabelCrossEntropyLoss function.

Classification of Chinese Hotel Reviews Based on GCN

Text classification is a fundamental part of natural language processing and can help with many downstream tasks, such as emotion analysis, question and answer systems, and recommendation systems. The graph convolution neural network has the natural superiority in the non - Euclidean space data. For Chinese text data, there is a lot of correlation between the data, using the graph convolutional neural network for text classification can achieve good results. In our experiment, we use a simple one-hot encoding of the word vector to process our words and use of the graph convolutional neural network can achieve 94.24% accuracy in our data set.

Fusion of heterogeneous attention mechanisms in multi-view convolutional neural network for text classification

The rapid proliferation of user generated content has given rise to large volumes of text corpora. Increasingly, scholars, researchers, and organizations employ text classification to mine novel insights for high-impact applications. Despite their prevalence, conventional text classification methods rely on labor-intensive feature engineering efforts that are task specific, omit long-term relationships, and are not suitable for the rapidly evolving domains. While an increasing body of deep learning and attention mechanism literature aim to address these issues, extant methods often represent text as a single view and omit multiple sets of features at varying levels of granularity. Recognizing that these issues often result in performance degradations, we propose a novel Spatial View Attention Convolutional Neural Network (SVA-CNN). SVA-CNN leverages an innovative and carefully designed set of multi-view representation learning, a combination of heterogeneous attention mechanisms and CNN-based operations to automatically extract and weight multiple granularities and fine-grained representations. Rigorously evaluating SVA-CNN against prevailing text classification methods on five large-scale benchmark datasets indicates its ability to outperform extant deep learning-based classification methods in both performance and training time for document classification, sentiment analysis, and thematic identification applications. To facilitate model reproducibility and extensions, SVA-CNN’s source code is also available via GitHub.

Improving convolutional neural network for text classification by recursive data pruning

In spite of the state-of-the-art performance of deep neural networks, shallow neural networks are still the choice in applications with limited computing and memory resources. Convolutional neural network (CNN), in particular the one-convolutional-layer CNN, is a widely-used shallow neural network in natural language processing tasks such as text classification. However, it was found that CNNs may misfit to task-irrelevant words in dataset, which in turn leads to unsatisfactory performance. To alleviate this problem, attention mechanism can be integrated into CNN, but this takes up the limited resources. In this paper, we propose to address the misfitting problem from a novel angle - pruning task-irrelevant words from the dataset. The proposed method evaluates the performance of each convolutional filter based on its discriminative power of the feature generated at the pooling layer, and prunes words captured by the poorly-performed filters. Experiment results show that our proposed model significantly outperforms the CNN baseline model. Moreover, our proposed model produces performance similar to or better than the benchmark models (attention integrated CNNs) while demanding less parameters and FLOPs, and is therefore a choice model for resource limited scenarios, such as mobile applications.

Learning neural networks for text classification by exploiting label relations

Text classification has been a fundamental problem in the realm of Natural Language Processing (NLP), and a variety of approaches have been proposed with the development of deep learning. Despite recent progress, most existing approaches deal with the problem of multi-class text classification in a flat way, assuming that text labels are semantically independent. This assumption doesn’t always hold in realistic settings, since there usually exist hierarchical or within-layer dependencies in the latent label space, especially when we consider larger label sets. In this paper, we propose a label clustering algorithm to exploit the underlying structure of label relations, and express the stacked concept relationships in the form of a two-layer label space. Next, we present two different neural network structures to capture inter-layer and intra-layer label relations. The first model HSNN organizes a group of local classifiers in a hierarchical way according to the exploited label space, while the second model LSNN takes advantages of text representations in different granularity levels and the bidirectional inferences with recurrent connections to make predictions. Finally, we evaluate our methods on three benchmark datasets. The results empirically demonstrate that both models are capable of leveraging the exploited label relations to improve text classification performance.

Words in Pairs Neural Networks for Text Classification

Existing methods utilized single words as text features. Some words contain multiple meanings, and it is difficult to distinguish its specific classification according to a single word, which probably affects the accuracy of the text classification. Propose a framework based on Words in pairs neural networks (WPNN) for text classification. Words in pairs include all single word combinations which have a high mutual association. Mine the crucial explicit and implicit Words in pairs as text features. These words in pairs as a text feature are easily classified. The words in pairs are utilized as the input of the neural network, which provides a better classification ability to the model, because they are more recognizable than the single word. Experimental results show that our model outperforms five benchmark algorithms.

Knowledge attention sandwich neural network for text classification

Text Classification is a fundamental and crucial issue in many Natural Language Processing (NLP) tasks. An effective and efficient representation model is the key to text classification. However, most existing representation models either learn insufficient structural information or just rely on pre-defined structures, resulting in degradation of performance and generalization capability. We propose a novel Sandwich Neural Network (SNN), which is able to learn local semantic and global structural representations automatically without parsers. To combine semantic and structural representations sensibly, we propose four fusion methods incorporated with SNN: Static Fusion, Adaptive Learning, Self-Attention, and Knowledge Attention methods. Static Fusion weights semantic and structural representations equally, Adaptive Learning learns the weights at corpus level, and Self-Attention learns the weights at instance level. More importantly, within Knowledge Attention fusion method, external semantic and structural knowledge are incorporated into SNN to improve the attention procedure and further boost the performance of SNN. Evaluated with four mainstream datasets: Text REtrieval Conference (TREC), SUBJectivity (SUBJ), Movie Reviews (MR) and Stanford Sentiment Treebank with Five Labels (SST-5), the proposed Knowledge Attention Sandwich Neural Network(KA-SNN) model achieves very competitive performance. Specifically, the accuracies achieve 96.2% (TREC), 94.1% (SUBJ), 82.3% (MR) and 51.5% (SST-5). Moreover, the proposed Knowledge Attention reduces the structural complexity of attention module by 77.66-81.43% with a computing time reduction of 21.47-34.05%, compared with Self-Attention fusion method.