Text Classification Applications Research Articles

Effective text classification using BERT, MTM LSTM, and DT

Text classification plays a critical role in managing large volumes of electronically produced texts. As the number of such texts increases, manual analysis becomes impractical, necessitating an intelligent approach for processing information. Deep learning models have witnessed widespread application in text classification, including the use of recurrent neural networks like Many to One Long Short-Term Memory (MTO LSTM). Nonetheless, this model is limited by its reliance on only the last token for text labelling. To overcome this limitation, this study introduces a novel hybrid model that combines Bidirectional Encoder Representations from Transformers (BERT), Many To Many Long Short-Term Memory (MTM LSTM), and Decision Templates (DT) for text classification. In this new model, the text is first embedded using the BERT model and then trained using MTM LSTM to approximate the target at each token. Finally, the approximations are fused using DT. The proposed model is evaluated using the well-known IMDB movie review dataset for binary classification and Drug Review Dataset for multiclass classification. The results demonstrate superior performance in terms of accuracy, recall, precision, and F1 score compared to previous models. The hybrid model presented in this study holds significant potential for a wide range of text classification tasks and stands as a valuable contribution to the field.

Research on the application of graph neural networks in text classification

With the rapid development of the Internet and social media, a large amount of text data is constantly generated, and text classification has become an important task. Text classification is an important branch in the field of natural language processing and has attracted the attention of many researchers in recent years. Due to the ease of editing text data, most texts in the network are manually constructed and uploaded by users. Therefore, the standardization of online texts and the classification of texts with different granularity are of great significance in the field of information retrieval. This article introduces the basic concepts of text classification and three text classification methods: knowledge engineering (rule) based text classification methods, traditional machine learning-based text classification methods, and deep neural network-based text classification methods. This article also analyzes the advantages and disadvantages of the three methods. Subsequently, the basic concepts and common models of graph neural networks were explored. An application case of graph neural networks in text classification is provided. This article provides effective suggestions for text classification applications in the field of natural language processing.

SMART DOCUMENT CLASSIFICATION

Smart document classification using Naive Bayes is a method used for organizing large volumes of unstructured data into categories or classes. This method uses the Naive Bayes algorithm's probabilistic approach to determine the likelihood that a text falls into a specific category based on the frequency of its words. A machine learning approach called Naive Bayes is used to forecast the likelihood of a particular category depending on the characteristics of the document. For text classification applications including spam filtering, sentiment analysis, and topic modeling, this method has been frequently employed. In this method, the algorithm is trained on a labeled dataset, which contains a set of documents and their corresponding categories The algorithm can be used to categorize fresh documents into the appropriate categories after it has been trained. The standard and applicability of the features used to represent the documents will determine how accurate the classification is. The Naive Bayes classifier is a popular option for smart document categorization since it is simple to use and needs little training data to attain high accuracy. Overall, automatic document classification using naive Bayes is an effective and efficient approach for organizing and managing large volumes of textual data. Key Words: classification, Naïve Bayes, machine learning, textual data

Genetic algorithm and support vector machine application in English text classification for intelligent teaching

Genetic algorithm and support vector machine application in English text classification for intelligent teaching

Heterogeneous Graph Transformer for Meta-structure Learning with Application in Text Classification

The prevalent heterogeneous Graph Neural Network (GNN) models learn node and graph representations using pre-defined meta-paths or only automatically discovering meta-paths. However, the existing methods suffer from information loss due to neglecting undiscovered meta-structures with richer semantics than meta-paths in heterogeneous graphs. To take advantage of the current rich meta-structures in heterogeneous graphs, we propose a novel approach called HeGTM to automatically extract essential meta-structures (i.e., meta-paths and meta-graphs) from heterogeneous graphs. The discovered meta-structures can capture more prosperous relations between different types of nodes that can help the model to learn representations. Furthermore, we apply the proposed approach for text classification. Specifically, we first design a heterogeneous graph for the text corpus, and then apply HeGTM on the constructed text graph to learn better text representations that contain various semantic relations. In addition, our approach can also be used as a strong meta-structure extractor for other GNN models. In other words, the auto-discovered meta-structures can replace the pre-defined meta-paths. The experimental results on text classification demonstrate the effectiveness of our approach to automatically extracting informative meta-structures from heterogeneous graphs and its usefulness in acting as a meta-structure extractor for boosting other GNN models.

Examining transaction-specific satisfaction and trust in Airbnb and hotels. An application of BERTopic and Zero-shot text classification

With a methodological approach, this article explores the application of data mining to the user-generated content of tourist accommodation on infomediation platforms and social networks. Its objective is to present an algorithm that allows the identification of service characteristics relevant to guest satisfaction and trust. Our study processes unstructured, natural-language data about Airbnb and hotel stays (the final dataset was 12,236 Airbnb sentences and 12,200 hotel sentences from 2018 until September 25 2021). Among the results is a computational algorithm that uses BERTopic to identify latent themes (or topics) in the narratives. Secondly, our analysis applies a Zero-shot classification approach for classifying guest reviews into labels related to guests' satisfaction and trust. Thirdly, we execute a Principal Component Analysis to investigate the sufficiency relationships between extracted topics, customer satisfaction, and trust-based labels. To sum up, and as practical implications, our study adds to the knowledge about the sharing economy by providing insights for developing marketing policies and a better understanding of hospitality services.

Deep Learning Based Text Classification Methods

Text classification tasks are indispensable in natural language processing. With the development of Internet technology, the way people transmit information has changed from letters to the Internet. With the increase in the amount of information, manual data annotation is inefficient. After 2010, the emergence of deep learning methods has brought text classification into an epoch-making stage. ReNN-> MLP-> RNN-> CNN-> Attention -> Transformer-> GNN and other text classification methods are gradually being developed and known by everyone, which also shows that text classification is developing towards a text feature that does not rely on manually acquired text features and is directly learning from the text content and modelling. This paper will start with basic knowledge, first let everyone understand the nature, application and historical background of text classification, will briefly introduce shallow learning, then enter deep learning, Select the classic model from the six classification methods from ReNN to Transformer for brief analysis, briefly analyse the model from the principle of the model, what problems it is good at solving, problems or shortcomings that it is not good at solving. Finally, the paper describes the performance of these models on the dataset.

Text classification in memristor-based spiking neural networks

Memristors, emerging non-volatile memory devices, have shown promising potential in neuromorphic hardware designs, especially in spiking neural network (SNN) hardware implementation. Memristor-based SNNs have been successfully applied in a wide range of applications, including image classification and pattern recognition. However, implementing memristor-based SNNs in text classification is still under exploration. One of the main reasons is that training memristor-based SNNs for text classification is costly due to the lack of efficient learning rules and memristor non-idealities. To address these issues and accelerate the research of exploring memristor-based SNNs in text classification applications, we develop a simulation framework with a virtual memristor array using an empirical memristor model. We use this framework to demonstrate a sentiment analysis task in the IMDB movie reviews dataset. We take two approaches to obtain trained SNNs with memristor models: (1) by converting a pre-trained artificial neural network (ANN) to a memristor-based SNN, or (2) by training a memristor-based SNN directly. These two approaches can be applied in two scenarios: offline classification and online training. We achieve the classification accuracy of 85.88% by converting a pre-trained ANN to a memristor-based SNN and 84.86% by training the memristor-based SNN directly, given that the baseline training accuracy of the equivalent ANN is 86.02%. We conclude that it is possible to achieve similar classification accuracy in simulation from ANNs to SNNs and from non-memristive synapses to data-driven memristive synapses. We also investigate how global parameters such as spike train length, the read noise, and the weight updating stop conditions affect the neural networks in both approaches. This investigation further indicates that the simulation using statistic memristor models in the two approaches presented by this paper can assist the exploration of memristor-based SNNs in natural language processing tasks.

A Survey of Cross-Lingual Text Classification and Its Applications on Fake News Detection

Cross-lingual text classification is a challenging task in natural language processing. The objective is to build accurate text classification models for low-resource languages by transferring the knowledge learned from high-resource languages. The task has been studied since 2003 and has attracted significantly growing attention in the last decade due to the success of deep learning models in natural language processing. Many new methods have been proposed to address the challenges in cross-lingual text classification. Meanwhile, cross-lingual fake news detection is one of the most important applications of cross-lingual text classification. It has already created significant social impacts on alleviating the infodemic problem in low-resource languages. The research works on cross-lingual text classification and cross-lingual fake news detection have been growing rapidly in recent years. Therefore, a comprehensive survey is imperative to summarize existing algorithms for cross-lingual text classification and explain the connections among them. This paper systematically reviews research works on cross-lingual text classifications and their applications in cross-lingual fake news detection. We categorize the evolution of cross-lingual text classification methods into four phases: (1) Traditional text classification models with translation; (2) Cross-lingual word embedding-based methods, (3) Pretraining then finetuning-based methods, and (4) Pretraining then prompting-based methods. We first discuss and analyze the representative methods in each phase in detail. Second, we provide a detailed review of their applications in the emerging fake news detection problem. Finally, we explore the potential issues of this open problem and also discuss possible future directions.

Comprehensive analysis of deep learning based text classification models and applications

Comprehensive analysis of deep learning based text classification models and applications

Metaheuristic Optimized Ensemble Model for Classification of SMS Spam in Computer Networks

By use of electronic communication, we are able to communicate a message to the recipient. In this digital age, a collaboration between several people is possible thanks to a variety of digital technologies. This interaction may take place in a variety of media formats, including but not limited to text, images, sound, and language. Today, a person's primary means of communication is their smart gadget, most commonly a cell phone. Spam is another side effect of our increasingly text-based modes of communication. We received a bunch of spam texts on our phones, and we know they're not from anyone we know. The vast majority of businesses nowadays use spam texts to advertise their wares, even when recipients have explicitly requested not to receive such messages. As a rule, there are many more spam emails than genuine ones. We apply text classification approaches to define short messaging service (SMS) and spam filtering in this study, which effectively categorizes messages. In this paper, we use machine learning algorithms and metaheuristic optimization to determine what percentage of incoming SMS messages are spam. This is why we used the optimized models to evaluate and contrast many classification strategies for gathering data.

Development of a fake news detection tool for Vietnamese based on deep learning techniques

With the development of the Internet, social networks and different communication channels, people can get information quickly and easily. However, in addition to real and useful news, we also receive false and unreal information. The problem of fake news has become a difficult and unresolved issue. For languages with few users, such as Vietnamese, the research on fake news detection is still very limited and has not received much attention. In this paper, we present research results on building a tool to support fake news detection for Vietnamese. Our idea is to apply text classification techniques to fake news detection. We have built a database of 4 groups of 2 topics about politics (fake news and real news) and about Covid-19 (fake news and real news). Then use deep learning techniques CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) to create the corresponding models. When there is new news that needs to be verified, we just need to apply the classification to see which of the four groups they label into to decide whether it is fake news or not. The tool was able to detect fake news quickly and easily with a correct rate of about 85 %. This result will be improved when getting a larger training data set and adjusting the parameters for the machine learning model. These results make an important contribution to the research on detecting fake news for Vietnamese and can be applied to other languages. In the future, besides using classification techniques (based on content analysis), we can combine many other methods such as checking the source, verifying the author's information, checking the distribution process to improve the quality of fake news detection.

Short Text Classification with Tolerance-Based Soft Computing Method

Text classification aims to assign labels to textual units such as documents, sentences and paragraphs. Some applications of text classification include sentiment classification and news categorization. In this paper, we present a soft computing technique-based algorithm (TSC) to classify sentiment polarities of tweets as well as news categories from text. The TSC algorithm is a supervised learning method based on tolerance near sets. Near sets theory is a more recent soft computing methodology inspired by rough sets where instead of set approximation operators used by rough sets to induce tolerance classes, the tolerance classes are directly induced from the feature vectors using a tolerance level parameter and a distance function. The proposed TSC algorithm takes advantage of the recent advances in efficient feature extraction and vector generation from pre-trained bidirectional transformer encoders for creating tolerance classes. Experiments were performed on ten well-researched datasets which include both short and long text. Both pre-trained SBERT and TF-IDF vectors were used in the experimental analysis. Results from transformer-based vectors demonstrate that TSC outperforms five well-known machine learning algorithms on four datasets, and it is comparable with all other datasets based on the weighted F1, Precision and Recall scores. The highest AUC-ROC (Area under the Receiver Operating Characteristics) score was obtained in two datasets and comparable in six other datasets. The highest ROC-PRC (Area under the Precision–Recall Curve) score was obtained in one dataset and comparable in four other datasets. Additionally, significant differences were observed in most comparisons when examining the statistical difference between the weighted F1-score of TSC and other classifiers using a Wilcoxon signed-ranks test.

An innovative approach to classify hierarchical remarks with multi-class using BERT and customized naïve bayes classifier

Text classification is the process of grouping text into distinct categories. Text classifiers may automatically assess text input and allocate a set of pre-defined tags or categories depending on its content or a pre-trained model using Natural Language Processing (NLP), which actually is a subset of Machine Learning (ML). The notion of text categorization is becoming increasingly essential in enterprises since it helps firms to get ideas from facts and automate company operations, lowering manual labor and expenses. Linguistic Detectors (the technique of determining the language of a given document), Sentiment Analysis (the process of identifying whether a text is favorable or unfavorable about a given subject), Topic Detection (determining the theme or topic of a group of texts), and so on are common applications of text classification in industry. The nature of the dataset is Multi-class and multi-hierarchical, which means that the hierarchies are in multiple levels, each level of hierarchy is multiple class in nature. One of ML’s most successful paradigms is supervised learning from which one can build a generalization model. Hence, a custom model is built, so that the model fits with the problem. Deep learning (DL), part of Artificial Intelligence (AI) , does functions that replicate the human brain's data processing capabilities in order to identify text or artifacts, translate languages, detect voice, draw conclusions and so on. Bidirectional Encoder Representations from Transformers (BERT), a Deep Learning Algorithm performs an extra-ordinary task in NLP text classification and results in high accuracy. Therefore, BERT is combined with the Custom Model developed and compared with the native algorithm to ensure the increase in accuracy rates.

Cross-Industry Process Standardization for Text Analytics

We are living in a world where everything computes, everyone and everything is connected and sharing data. Going beyond just capturing and managing data, enterprises are tapping into IoT and Artificial Intelligence (AI) to create insights and intelligence in a revolutionary way that was not possible before. For instance, by analyzing unstructured data (such as text), call centers can extract entities, concepts, themes which can enable them to get faster insights that only few years back was not feasible. Public safety and law enforcement are only few of the examples that benefit from text analytics used to strengthen crime investigation. Sentiment Analysis, Content Classification, Language Detection and Intent Detection are just some of the Text Classification applications. The overall process model of such applications considering the complexity of the unstructured data, can be definitely challenging. In response to the chaotic emerging science of unstructured data analysis, the main goal of this paper is to first contribute to the gap of no existing methodology approach for Text Analytics projects, by introducing a methodology approach based on one of the most widely accepted and used methodology approach of CRISP-DM.

Sentiment analysis and spam filtering using the YAC2 clustering algorithm with transferability

Two notable applications of text classification are sentiment analysis and spam filtering. Traditional machine learning approaches to text classification are often complex, non-transferrable, and require supervision. This paper introduces an unsupervised approach to text classification which is relatively simple and transfers between problem domains, while providing accuracy comparable or better than established alternatives. We present an integrated solution which combines a new clustering algorithm, Yet Another Clustering Algorithm (YAC2), with a domain transferrable feature engineering approach for Twitter sentiment analysis and spam filtering of YouTube comments. We evaluate the effectiveness of this integrated solution for Twitter sentiment analysis using three datasets: Starbucks, Verizon, and Southwest Airlines. YouTube spam filtering is evaluated using four datasets: Psy, LMFAO, Shakira,and Katy Perry. We compare the results with established clusteringsolutions: KNN, Spectral, and DBSCAN. Our integrated solution performs better than all the alternatives for sentiment analysis. For spam filtering, YAC2 and KNN perform within 1% of each other and far better than Spectral and DBSCAN for all datasets. Additionally, our feature engineering approach improves accuracy compared to using a traditional method, while significantly reducing model dimensionality, matrix sparsity and providing transferability across the datasets tested.

Analytics of machine learning-based algorithms for text classification

Text classification is the most vital area in natural language processing in which text data is automatically sorted into a predefined set of classes. The application of text classification is wide in commercial works like spam filtering, decision making, extracting information from raw data, and many other applications. Text classification is more significant for many enterprises since it eliminates the need for manual data classification, a more expensive and time-consuming mechanism. In this paper, a comparative analysis of text classification is done in which the efficiency of different machine learning algorithms on different datasets is analyzed and compared. Support Vector Machine (SVM), k-Nearest Neighbor (k-NN), Logistic Regression (LR), Multinomial Naïve Bayes (MNB), and Random Forest (RF) are Machine Learning based algorithms used in this work. Two different datasets are used to make a comparative analysis of these algorithms. This paper further analyzes the machine learning techniques employed for text classification on the basis of performance metrics viz accuracy, precision, recall and f1- score. The resullltsss reveals that Logistic Regression and Support Vector Machine outperforms the other models in the IMDB dataset, and kNN outperforms the other models for the SPAM dataset as per the results obtained from the proposed system.

Improving the Science of Annotation for Natural Language Processing: The Use of the Single-Case Study for Piloting Annotation Projects

Researchers need guidance on how to obtain maximum efficiency and accuracy when annotating training data for text classification applications. Further, given wide variability in the kinds of annotations researchers need to obtain, they would benefit from the ability to conduct low-cost experiments during the design phase of annotation projects. To this end, our study proposes the single-case study design as a feasible and causally-valid experimental design for determining the best procedures for a given annotation task. The key strength of the design is its ability to generate causal evidence at the individual level, identifying the impact of competing annotation techniques and interfaces for the specific annotator(s) included in an annotation project. In this paper, we demonstrate the application of the single-case study in an applied experiment and argue that future researchers should incorporate the design into the pilot stage of annotation projects so that, over time, a causally-valid body of knowledge regarding the best annotation techniques is built.

Sentiment Classification for Financial Texts Based on Deep Learning.

Sentiment classification for financial texts is of great importance for predicting stock markets and financial crises. At present, with the popularity of applications in the field of natural language processing (NLP) adopting deep learning, the application of automatic text classification and text-based sentiment classification has become more and more extensive. However, in the field of financial text-based sentiment classification, due to a lack of labeled samples, such applications are limited. A domain-adaptation-based financial text sentiment classification method is proposed in this paper, which can adopt source domain (SD) text data with sentiment labels and a large amount of unlabeled target domain (TD) financial text data as training samples for the proposed neural network. The proposed method is a cross-domain transfer-learning-based method. The domain classification subnetwork is added to the original neural network, and the domain classification loss function is also added to the original training loss function. Therefore, the network can simultaneously adapt to the target domain and then accomplish the classification task. The experiment of the proposed sentiment classification transfer learning method is carried out through an open-source dataset. The proposed method in this paper uses the reviews of Amazon Books, DVDs, electronics, and kitchen appliances as the source domain for cross-domain learning, and the classification accuracy rates can reach 65.0%, 61.2%, 61.6%, and 66.3%, respectively. Compared with nontransfer learning, the classification accuracy rate has improved by 11.0%, 7.6%, 11.4%, and 13.4%, respectively.

Categorizing Natural Language-Based Customer Satisfaction: An Implementation Method Using Support Vector Machine and Long Short-Term Memory Neural Network

Analyzing natural language-based Customer Satisfaction (CS) is a tedious process. This issue is practically true if one is to manually categorize large datasets. Fortunately, the advent of supervised machine learning techniques has paved the way toward the design of efficient categorization systems used for CS. This paper presents the feasibility of designing a text categorization model using two popular and robust algorithms – the Support Vector Machine (SVM) and Long Short-Term Memory (LSTM) Neural Network, in order to automatically categorize complaints, suggestions, feedbacks, and commendations. The study found that, in terms of training accuracy, SVM has best rating of 98.63% while LSTM has best rating of 99.32%. Such results mean that both SVM and LSTM algorithms are at par with each other in terms of training accuracy, but SVM is significantly faster than LSTM by approximately 35.47s. The training performance results of both algorithms are attributed on the limitations of the dataset size, high-dimensionality of both English and Tagalog languages, and applicability of the feature engineering techniques used. Interestingly, based on the results of actual implementation, both algorithms are found to be 100% effective in accurately predicting the correct CS categories. Hence, the extent of preference between the two algorithms boils down on the available dataset and the skill in optimizing these algorithms through feature engineering techniques and in implementing them toward actual text categorization applications.