Text Preprocessing Research Articles

BERT: A Review of Applications in Sentiment Analysis

E-commerce reviews are becoming more valued by both customers and companies. The high demand for sentiment analysis is driven by businesses relying on it as a crucial tool to improve product quality and make informed decisions in a fiercely competitive business environment. The purpose of this review paper is to explore and evaluate the applications of the BERT model, a Natural Language Processing (NLP) technique, in sentiment analysis across various fields. The model has been utilized in certain studies for various languages, restaurant businesses, agriculture, Automated Essay Scoring (AES), Twitter, and Google Play. The BERT model's fine-tuning steps involve using pre-trained BERT to perform various language understanding tasks. Text pre-processing is conducted to clean up the data and convert it to numbers before feeding it into BERT, which generates vectors for each input token. We found that BERT outperformed the norm on a range of general language understanding tasks, including sentiment analysis, paraphrase recognition, question-answering, and linguistic acceptability. The detection of neutral reviews and the presence of false reviews in the dataset are two problems that have an impact on the model's accuracy. Training is also slow because it is huge and there are many weights to update. Additional research could be conducted to improve the BERT model's accuracy by constructing a false review categorization model and providing more training to the model in recognizing neutral reviews. Doi: 10.28991/HIJ-2023-04-02-015 Full Text: PDF

IMPLEMENTASI ALGORITMA TF-IDF DAN SUPPORT VECTOR MACHINE TERHADAP ANALISIS PENDETEKSI KOMENTAR CYBERBULLYING DI MEDIA SOSIAL TIKTOK

Cyberbullying is the act of sending text, images, or videos using the internet, mobile phones, or other devices with the aim of hurting and shaming other people. Cyberbullying is often done through several social media platforms, one of which is through comments on the TikTok application. According to a report by We Are Social, TikTok has 1.4 billion monthly active users aged 18 and above globally. Indonesia currently ranks second in the world in terms of active TikTok users. As a result, the potential for cyberbullying instances will grow as the number of users grows. By using data mining, the public can create a detection system, which can perform analysis on comments in the TikTok application. The method used is Term Frequency-Inverse Document Frequency (TF-IDF) and Support Vector Machine (SVM). The stages passed are to collect comments that are labelled manually. Then, text preprocessing, tokenizing, and weighting were carried out with TF-IDF. Then, implement the Support Vector Machine algorithm to detect cyberbullying comments. This study uses 80% training data and 20% testing data. From the performance results of the Support Vector Machine algorithm, 88% overall accuracy, 88% precision, 96% recall, and 92% f1-score were obtained in detecting cyberbullying comments on social media TikTok.

СРАВНИТЕЛЬНЫЙ АНАЛИЗ МЕТОДОВ ВЕКТОРИЗАЦИИ ТЕКСТОВЫХ ДАННЫХ БОЛЬШОЙ РАЗМЕРНОСТИ

The presented publication is devoted to an overview of the problem of presenting textual informationfor the subsequent implementation of cluster analysis in the framework of processingand managing high-dimensional information. Modern requirements for analytical, search andrecommendation information systems demonstrate the weak formation of a holistic solution thatcan provide a sufficient level of speed and quality of the results obtained within the framework ofthe current information technology market. The search for a solution to the presented problementails the need to conduct an objective analysis of existing solutions for representing textual informationin vector space, in order to form a holistic view of the advantages and disadvantages ofthe analyzed approaches, as well as the formation of criteria that allow one to implement theirown approach, devoid of identified weaknesses. The presented work is analytical, and allows youto get an idea of the current state and elaboration of the identified problem within a limited subjectarea. Clustering of text data is the automatic formation of subsets, the elements of which are instancesof documents of some researched, unstructured sample of a fixed dimension. This processcan be classified as unsupervised learning, which implies the absence of an expert who personallyassigns class indices to the original sample of documents. However, the implementation of clusteranalysis of text data without any pre-processing is impossible. To do this, it is necessary to ensurestandardization and reduction of input data to a single format and form. Within the framework ofthis stage of the implementation of cluster analysis, the presented publication discusses methodsfor preprocessing text data. The novelty of the presented publication lies in the formation of thetheoretical basis of the main methods of text data vectorization, by systematizing and objectifyingthe proposed assumptions, by conducting a series of experimental studies. The main difference ofthis work from the already published scientific works is the systematization and analysis of modernsolutions, as well as the hypotheses about the relevance and effectiveness of our own hybridizedapproach designed for text data vectorization.

Text Classification Using Genetic Programming with Implementation of Map Reduce and Scraping

Classification of text documents on online media is a big data problem and requires automation. Text classification accuracy can decrease if there are many ambiguous terms between classes. Hadoop Map Reduce is a parallel processing framework for big data that has been widely used for text processing on big data. The study presented text classification using genetic programming by pre-processing text using Hadoop map-reduce and collecting data using web scraping. Genetic programming is used to perform association rule mining (ARM) before text classification to analyze big data patterns. The data used are articles from science-direct with the three keywords. This study aims to perform text classification with ARM-based data pattern analysis and data collection system through web-scraping, pre-processing using map-reduce, and text classification using genetic programming. Through web scraping, data has been collected by reducing duplicates as much as 17718. Map-reduce has tokenized and stopped-word removal with 36639 terms with 5189 unique terms and 31450 common terms. Evaluation of ARM with different amounts of multi-tree data can produce more and longer rules and better support. The multi-tree also produces more specific rules and better ARM performance than a single tree. Text classification evaluation shows that a single tree produces better accuracy (0.7042) than a decision tree (0.6892), and the lowest is a multi-tree(0.6754). The evaluation also shows that the ARM results are not in line with the classification results, where a multi-tree shows the best result (0.3904) from the decision tree (0.3588), and the lowest is a single tree (0.356).

The Automatic Classification System for Academic Performance Evaluation at the Faculty of Information Technology Atma Jaya University of Makassar

Abstract - The Faculty of Information Technology currently carries out performance evaluations at the end of each semester and involves students as sources of data evaluation. The evaluation activity took place online on the website ss.fti.uajm.ac.id. With the number of active students, the number of evaluations that need to be read and the number read by faculty stakeholders also increases. This is inversely proportional to the time that stakeholders need time to read, evaluate, and categorize comments entered by students as part of the performance evaluation. In this study, a multi-classification of student comments related to evaluations at the Faculty of Information Technology UAJM will be carried out. Text pre-processing will use the Sastrawi library which includes stopword removal, stemming, and transformation of text into TFIDF form. The results of the pre-processing text will be used as input on Naive Bayes and using three scenarios to evaluate the classifier model. The average accuracy values of the Naive Bayes algorithm for category and sentiment labels are 79% and 81%, respectively. Furthermore, the expected result of this research is to reduce the time for FTI UAJM stakeholders to read and comment/suggest faster because the evaluation results are obtained in real-time.

Perbandingan Metode Pembobotan TF-RF Dan TF-ABS Pada Kategorisasi Berita Di BDI Denpasar

BDI Denpasar is a government agency tasked with carrying out training and education for human resources of animation, crafts and art. BDI Denpasar in managing news classes in the Kabar Insan Oke service still uses conventional methods. Therefore an automatic news classification module is needed. This study was made to compare the performance level of news classification at BDI Denpasar using K-NN classification with the TF-RF and TF-ABS term weighting methods. Methods that have a high level of performance will be implemented in the news classification module. This research was carried out by collecting news documents, text preprocessing, term weighting, classification, model validation and testing. The K-NN classification uses the n_neighbhor (k), namely k=3, k=5, k=7 and k=9 using a dataset of 324 documents containing 7 classes taken from BDI Denpasar website. Based on the results of the tests performed, TF-RF method obtained a higher performance at k=5 with an accuracy of 71% with a precision of 73% and a recall of 71%. TF-ABS method with the highest performance value is found at k=9 which obtains 70% accuracy, 63% precision and 70% recall. So the method that will be implemented in the news classification module is TF-RF at k=5 with an accuracy of 71% with a precision of 73% and a recall of 71%.

Legal Document Analysis

Abstract: Text preprocessing is the most essential and foremost step for any Machine Learning model. The raw data needs to be cleaned and pre-processed to get better performance. It is the method to clean the data and makes it ready to feed the data to the model. Text classification is the heart of many software systems that involve text documents processing. The purpose of text classification is to classify the text documents automatically into two or many defined categories. In this paper ,various preprocessing and classification approaches are used such as NLP, Machine Learning, etc from patent documents.

Classification of Sentiment Analysis and Community Opinion Modeling Topics for Application of ICT in Government Operations

Utilizing information systems is very useful in the current era. Digitizing administration in the Village is beneficial in the service process to the public. This is seen as a change in service that can make it easier or more difficult for the people of Sanrobone Village to take care of administration at the village office. This study aims to analyze public opinion regarding the use of e-government, predict public opinion regarding the use of e-government, and analyze modeling topics related to the use of e-government. This research applies a text mining algorithm with a sentiment analysis method to see positive, negative, and neutral public perceptions and also uses topic modeling to get the most frequently appearing topics in the data. Stages in this study include Data Collection, Text Pre-processing, Sentiment Analysis, Topic Modelling, Classification, and Evaluation. The results obtained are the ten words that appear most often in the responses of the Village community: easy 122, help 96, village 80, accessed 80, letter 80, permit 77, resident 73, manage 60, service 52, and the person with 52 words. The sentiment analysis is positive, with 411 opinions, 37 negative opinions, and 152 neutral opinions. Finally, the performance of the Nave Bayes algorithm in predicting classification results is excellent, with an accuracy rate of 98 percent.

Exploring the Capabilities and Limitations of ChatGPT and Alternative Big Language Models

ChatGPT, an artificial intelligence (AI)-powered chatbot developed by OpenAI, has gained immense popularity since its public launch in November 2022. With its ability to write essays, emails, poems, and even computer code, it has become a useful tool for professionals in various fields. However, ChatGPT’s responses are not always rooted in reality and are instead generated by a Generative Adversarial Network (GAN). This paper aims to build a text classification model for a chatbot using Python. The model is trained on a dataset consisting of customer responses to a survey and their corresponding class labels. Many classifiers are trained and tested, such as naive Bayes, random forest, extra trees, and decision trees. The results show that the extra trees classifier performs the best with an accuracy of 90%. The system demonstrates the importance of text preprocessing and selecting appropriate classifiers for text classification tasks in building an effective chatbot. In this paper, we also explore the capabilities and limitations of ChatGPT and its alternatives in 2023. We present a comprehensive overview of the alternative’s performance. The work here concludes with a discussion of the future directions of large language models and their impact on society and technology.

A Scoping Literature Review of Natural Language Processing Application to Safety Occurrence Reports

Safety occurrence reports can contain valuable information on how incidents occur, revealing knowledge that can assist safety practitioners. This paper presents and discusses a literature review exploring how Natural Language Processing (NLP) has been applied to occurrence reports within safety-critical industries, informing further research on the topic and highlighting common challenges. Some of the uses of NLP include the ability for occurrence reports to be automatically classified against categories, and entities such as causes and consequences to be extracted from the text as well as the semantic searching of occurrence databases. The review revealed that machine learning models form the dominant method when applying NLP, although rule-based algorithms still provide a viable option for some entity extraction tasks. Recent advances in deep learning models such as Bidirectional Transformers for Language Understanding are now achieving a high accuracy while eliminating the need to substantially pre-process text. The construction of safety-themed datasets would be of benefit for the application of NLP to occurrence reporting, as this would allow the fine-tuning of current language models to safety tasks. An interesting approach is the use of topic modelling, which represents a shift away from the prescriptive classification taxonomies, splitting data into “topics”. Where many papers focus on the computational accuracy of models, they would also benefit from real-world trials to further inform usefulness. It is anticipated that NLP will soon become a mainstream tool used by safety practitioners to efficiently process and gain knowledge from safety-related text.

Clinical Text Classification with Word Representation Features and Machine Learning Algorithms

Clinical text classification of electronic medical records is a challenging task. Existing electronic records suffer from irrelevant text, misspellings, semantic ambiguity, and abbreviations. The approach reported in this paper elaborates on machine learning techniques to develop an intelligent framework for classification of the medical transcription dataset. The proposed approach is based on four main phases: the text preprocessing phase, word representation phase, features reduction phase and classification phase. We have used four machine learning algorithms, support vector machines, naïve bayes, logistic regression and k-nearest neighbors in combination with different word representation models. We have applied the four algorithms to the bag of words, to TF-IDF, to word2vec. Experimental results were evaluated based on precision, recall, accuracy and F1 score. The best results were obtained with the combination of the k-NN classifier, and the word represented by Word2vec achieving an accuracy of 92% to correctly classify the medical specialties based on the transcription text.

ArSentBERT: fine-tuned bidirectional encoder representations from transformers model for Arabic sentiment classification

Sentiment analysis in the Arabic language is challenging because of its linguistic complexity. Arabic is complex in words, paragraphs, and sentence structure. Moreover, most Arabic documents contain multiple dialects, writing alphabets, and styles (e.g., Franco-Arab). Nevertheless, fine-tuned bidirectional encoder representations from transformers (BERT) models can provide a reasonable prediction accuracy for Arabic sentiment classification tasks. This paper presents a fine-tuning approach for BERT models for classifying Arabic sentiments. It uses Arabic BERT pre-trained models and tokenizers and includes three stages. The first stage is text preprocessing and data cleaning. The second stage uses transfer-learning of the pre-trained models’ weights and trains all encoder layers. The third stage uses a fully connected layer and a drop-out layer for classification. We tested our fine-tuned models on five different datasets that contain reviews in Arabic with different dialects and compared the results to 11 state-of-the-art models. The experiment results show that our models provide better prediction accuracy than our competitors. We show that the choice of the pre-trained BERT model and the tokenizer type improves the accuracy of Arabic sentiment classification.

Alternative Text Pre-Processing using Chat GPT Open AI

Text Pre-Processing is the first step in Sentiment Analysis. Categorizing a sentiment in a dataset is part of the Text-Preprocessing stage to get the optimal model accuracy value. Generative Pretrained Transformer, often known as Chat GPT, is a Machine Learning model that can automatically generate realistic and meaningful text. This study aims to examine the capability of GPT Chat as an alternative in the Text-Pre-Processing stage by utilizing GPT Chat 3 from the openai.com website in the Text-Pre-Processing stage of the collected tweet data. The data used in this research is the result of crawling Twitter by inserting the keyword "Chat GPT”. This study method was carried out by measuring performance using the K-Nearest Neighbor and Naïve Bayes Algorithms to find the best performance value and compare it with the Text-Preprocessing generated by Rapidminer. It is shown that the performance accuracy produced using the K-Nearest Neighbor Algorithm is 73.57% using the Linear Sampling method. The comparison result with the Text-Preprocessing method using Rapidminer indeed shows a better accuracy of 75.33%, which means it has a narrow difference of 1.76% with the Chat GPT Text Pre-Processing method. However, both are still in the same category, which is Fair Classification. The results of this research show that Chat GPT can be an alternative in Text-Preprocessing datasets for sentiment analysis.

Klasifikasi Sentiment Ulasan Aplikasi Sausage Man Menggunakan VADER Lexicon dan Naïve Bayes Classifier

Battle Royale games are games that mix adventure and survival elements with last man standing game modes. One of the most popular battle royale games is the Sausage Man game. The number of complaints such as bugs, cheaters, and FPS which continues to decrease makes the game annoying. The solution is that developers must improve and improve game security so that users feel comfortable playing the game. There are many opinions or reviews from users regarding problems in the game, sentiment analysis will be carried out on the Sausage Man application review data on the Google play store as a process to produce categorization of opinions through reviews. The purpose of the researcher is to carry out a sentiment analysis to see positive, neutral or negative opinions from Sausage Man game users. The stages carried out in this study were data collection using web scraping, data labeling, text preprocessing, document weighting, classification, and evaluation. The results of data labeling using the VADER Lexicon obtained 1089 reviews (36.3%) for positive sentiment, 912 reviews for neutral sentiment (30.4%), and 999 reviews for negative sentiment (33.3%). Classification using the Naïve Bayes Classifier. Evaluation using the Confusion Matrix by dividing 90% training data and 10% test data produces an accuracy of 75%, 79% precision, and 75% recall. For the division of 80% training data 20% of the test data produces an accuracy of 73%, 76% precision and 73% recall. Positive sentences are found more often, but the accuracy is still below 80%.

Klasifikasi Sentimen Transformasi dan Reformasi Sepak Bola Indonesia Pada Twitter Menggunakan Algoritma Bernoulli Naïve Bayes

Federation Internationale de Football Association (FIFA) carried out Transformations and Reformations to Indonesian Football with one of them Indonesia was chosen as the Host of the U-20 World Cup in 2023. The transformations and reformations carried out cause people to often provide opinions through social media Twitter. Opinions given by the public can be positive or negative. The research uses Text Mining to classify sentiment in 2 categories with the Bernoulli Naïve Bayes algorithm. This research aims to classify positive and negative sentiments and determine the level of accuracy value of the sentiment classification results of Indonesian Football Transformation and Reformation. The research stages carried out are data collection, text preprocessing, data labeling, TF-IDF weighting, Bernoulli Naïve Bayes classification, and evaluation. Based on the research results from 4907 data there is duplicate data and only uses 2125 data which is divided into 90% training data and 10% testing data, so as to get accuracy with a high category value of 88%. The classification results show that many tweets are positive sentiments.

Movie recommendation system via fuzzy decision making based dual deep neural networks

The internet and social networks produce an increasing amount of data. There is a serious necessity for a recommendation system because exploring through the huge collection is time-consuming and difficult. In this study, a multi-modal classifier is introduced which makes use of the output from dual deep neural networks: GRU for text analysis and Faster R-CNN for image analysis. These two networks reduce overall complexity with minimal computational time while retaining accuracy. More precisely, the GRU network is utilized to process movie reviews and the Faster RCNN is used to recognize each frames of the movie trailers. Gated Recurrent Unit (GRU) is a well-known variety of RNN that computes sequential data across recurrent structures. Faster RCNN is an enhanced version of Fast RCNN, it combines with the rectangular region proposals and with the features is extract by the ResNet-101. Initially, the trailer of the movie is manually splitted into frames and these frames are pre-processed using fuzzy elliptical filter for image analysis and the movie reviews are also tokenized for text analysis. The pre-processed text is taken as an input for GRU to classify offensive and non-offensive movies and the pre-processed images are taken as an input for Faster R-CNN to classify violence and non- violence movies based on the extracted features from the movie trailer. Afterwards, the four classified outputs are given as input for fuzzy decision-making unit for recommending best movies based on the Mamdani fuzzy inference system with gauss membership functions. The performance of the dual deep neural networks was evaluated using the specific parameters like specificity, precision, recall, accuracy and F1 score measures. The proposed GRU yields accuracy range of 97.73% for reviews and FRCNN yields the accuracy range of 98.42% for movie trailer.

Implementasi Algoritma Term Frequency Inverse Document Frequency (TF-IDF) dalam Menganalisis Sentimen Masyarakat Terhadap Covid-19 Varian Omicron

The latest variant was detected on November 24, 2021, namely the Omicron variant WHO said, Omicron was one of the Covid-19 variants that had mutated, with a very fast spread rate. The Government Republic of Indonesia has officially banned all foreigners from entering Indonesia, both those who have done so travel or come from countries exposed to the Omicron variant. This study uses data that has been processed using Netlytic online website. Netlytic analyzes text and visualizes public online conversations on social media sites. text preprocessing has several stages, namely case folding, tokenizing, stopword, stemming. Data analysis is the stage to classify words into positive, negative, or neutral sentiment classes. the last step is calculating the weights using the tf-idf method. It is proven from the DF value which reaches 628 words in one document, the D/DF value is 0.39 and the log D/DF is -0.41. The TF-IDF method can be taken in outline, namely it is easy to calculate frequency and relevance occurrence of words in a document. The TF-IDF method produces output according to user specifications, but this method takes a long time for large amounts of data.

Exploring differences in level of construction innovation: an empirical analysis in China.

Innovations can overcome constraints posed by resource depletion, increasing environmental and ecological protection concerns. There is considerable amount of innovation that occurs in the construction industry. Accordingly, construction innovation is receiving increasing attention in China. However, the provincial development level of construction innovation remains unclear. To address this gap, the present study employs data-driven measurement for the level of construction innovation. A total of 25 alternative indexes were selected based on the innovation ecosystem theory. Then, text preprocessing, statistical methods, and search statistics were employed to acquire index data. The indexes weights were determined through expert scoring and a cloud model. The quantitative measurement of the level of construction innovation was finally performed. Additionally, the exploratory results were revealed with the analyses of the overall, regional, and cluster. The results revealed that overall level of construction innovation in China is not high and regional distribution is uneven. Simultaneously, the level of construction innovation is consistent with local economic strength, and it is most sensitive to innovation output in regional level. Moreover, the spatial distribution of the level of construction innovation in China was showed, which has similarity with the characteristics of geographical location. The measurement system this study represents breakthroughs over traditional methods that rely on statistics, cases, or questionnaires, which can be applied to other research fields.

Sentiment Analysis on App Reviews Using Support Vector Machine and Naïve Bayes Classification

A review is an assessment given by someone based on certain aspects, such as the delivery of stories, pictures, effects, or visuals. Users can provide reviews which help the company know the quality of the application. However, reviews cannot be used as a reference for rating, because there are still users who provide reviews that are irrelevant to the rating given. This study aims to carry out sentiment analysis in order to classify the application user review data. The sentiment classification process begins with collecting and labeling 700 data. The data then goes through a text preprocessing, word weighting with TF-IDF, and classification using the Support Vector Machine and Naïve Bayes Classifier. The results produce the highest accuracy in the comparison of training and test data of 90%:10%. Support Vector Machine algorithm is capable of providing high accuracy with RBF kernel, γ=1, and C=10. The results obtained using 10-fold cross validation give an accuracy value of 92.86%, a precision value of 92.88%, a recall value of 92.88%, a specificity value of 94.73%, and f-measure of 92.76%. Naïve Bayes Classifier method is able to provide high accuracy by using Multinomial Naïve Bayes Classifier. The results obtained using 10-fold cross validation give an accuracy value of 92.54%, a precision value of 92.55%, a recall value of 92.51%, a specificity value of 93.9%, and f-measure of 92.44%. Based from the result, it can be concluded that the classification using the Support Vector Machine is superior to the Naive Bayes Classification.

Application of text mining technology for comparative analysis of trends in the labor market

Object: The main purpose of the article is to analyze the employment of the population in the post-Soviet space in the context of world events to identify positive and negative trends in employment, as well as characteristic trajectories of development directions.Methods: Modern methods of word processing, in particular text mining, word cloud were used.Findings: The rapid development of modern technologies, Internet applications is accompanied by the generation of large amounts of data, the timely processing of which is today one of the main problems in various spheres of life - social, economic, political, and others. In solving this global problem, modern methods of processing text information, the so-called text mining technologies, come to the rescue. These tools allow to increase the efficiency of solving problems of different levels. The algorithms embedded in the text mining technology reveal the basic concepts of the text, the content and the relationship between them.The integration of modern text mining systems and the R-Studio programming language makes it possible to conduct research in the field of text analysis and processing. These systems, using statistical methods, process the rating of news documents, materials of scientific documents, blogs, tweets, emails, advertisements and other information. The main task of text analysis is to get a clear idea about the topics of interest, to extract important information. The analysis of text documents by text mining methods is carried out in several stages: information search, text preprocessing, extraction of the required information, application of text methods, analysis and interpretation of the obtained results. For the analysis of texts, articles in Russian in PDF format were selected, including information on trends in the labor market and employment for 1995–2020.Conclusions: over the past 20 years there have been significant changes in the labor market and employment. Text analysis technologies made it possible to reveal that during the study period, the labor market issues of unemployment, employment, employment transformation, the emergence of new forms of employment, social and gender problems, and others raised.