TF-IDF Method Research Articles

뉴스의 감성분석과 전문가 지식을 이용한 딥러닝 기반의 부동산 가격 예측

This study proposes a deep learning method for predicting real estate prices utilizing deep learning with real estate price time-series data, text data such as news, and experts’ opinions. This study provides a method to integrate qualitative data into quantitative data when developing a deep learning model to predict time series. We employed house sales prices and economic data to build a prediction model for real estate prices from January 2006 to June 2021. In addition, we crawled 150,000 news information related to real estate and applied the TF-IDF method to identify the meaning of the news. We integrated the sentiment analysis and the experts’ opinions into our proposed model. The experiment results show that our proposed method is valid in RMSE performance. It was also confirmed that LSTM was superior to short-term prediction in long-term prediction.

Social media data analysis framework for disaster response

This paper presents a social media data analysis framework applied to multiple datasets. The method developed uses machine learning classifiers, where filtering binary classifiers based on deep bidirectional neural networks are trained on benchmark datasets of disaster responses for earthquakes and floods and extreme flood events. The classifiers consist of learning from discrete handcrafted features and fine-tuning approaches using deep bidirectional Transformer neural networks on these disaster response datasets. With the development of the multiclass classification approach, we compare the state-of-the-art results in one of the benchmark datasets containing the largest number of disaster-related categories. The multiclass classification approaches developed in this research with support vector machines provide a precision of 0.83 and 0.79 compared to Bernoulli naïve Bayes, which are 0.59 and 0.76, and multinomial naïve Bayes, which are 0.79 and 0.91, respectively. The binary classification methods based on the MDRM dataset show a higher precision with deep learning methods (DistilBERT) than BoW and TF-IDF, while in the case of UnifiedCEHMET dataset show a high performance for accuracy with the deep learning method in terms of severity, with a precision of 0.92 compared to BoW and TF-IDF method which has a precision of 0.68 and 0.70, respectively.

Indonesian Language Hoax News Classification Basedn on Naïve Bayes

Hoax news in Indonesia causes various problems, therefore it is necessary to classify whether a news is in the hoax category or is valid. Naive Bayes is an algorithm that can perform classification but has a weakness, namely the selection of attributes that can affect accuracy so that it needs to be optimized by giving weights to attributes using the TF-IDF method. Classification using Naive Bayes and using TF-IDF as attribute weighting on a dataset of 600 data resulted in 82% accuracy, 84% precision, and 89% recall. The suggestion put forward is that it is better to use a larger number of datasets in order to produce higher accuracy.

Research on fake online review recognition method based on feature engineering

In the context of the explosive growth of online reviews in e-commerce, some merchants began to hire consumers to make fake purchases to increase sales, which caused the difficulty in identifying fake reviews. Therefore, from the perspective of consumers’ shopping behavior, a feature recognition method based on comment text, reviewer attribute and reviewer’s shopping behavior (referred to as TAB) is proposed to solve the problem of fake reviews caused by fake shopping behavior. Compared with BOW, TF-IDF, N-gram, FSP and other traditional feature extraction methods, experiments were carried out on Naive Bayes, SVM, Logistic Regression, Random Forest, CNN and other classification models. By changing the data dimension, the stability of each false comment recognition model was analyzed. The experimental results show that the TAB feature recognition method has better classification performance on logistic regression and support vector machine models, and the model is relatively stable, and the classification effect is not easily affected by dimension change.

The Construction of Intelligent Emotional Analysis and Marketing Model of B&B Tourism Consumption Under the Perspective of Behavioral Psychology.

This manuscript constructs an intelligent sentiment analysis and marketing model for bed and breakfast (B&B) consumption based on a behavioral psychology perspective. Based on the LDA theme model, the theme features and keywords of the reviews covering user feedback are explored from the text data, and the theme framework of user sentiment perception is constructed by combining previous literature on user perception in the B&B market, and the themes of user online reviews are summarized in four dimensions: practical, sensory, cognitive, and emotional components of user experience. In this manuscript, GooSeeker software was selected for data crawling and ROST CM (ROST content mining) developed by Wuhan University was used for text processing. To improve the accuracy of text classification and improve the missing data, the online comment text is divided into sentences by symbols, and the text is divided into words based on sentences, and the spatial vector model and the text feature word weighting method of TF-IDF are used for vector representation, and the polynomial Bayesian classifier is called to identify the topics of sentences. The classical Theory of Planned Behavior (TPB) was used to analyze the influencing factors of the willingness to consume experiential B&B tourism, and countermeasure suggestions for the development of B&B tourism were proposed based on the research findings In the empirical testing stage, a questionnaire on the willingness to consume experiential B&B tourism was designed, and web research was chosen to collect the data. SPSS20.0 was used to conduct reliability analysis, factor analysis, correlation analysis, and regression analysis on the data, and AMOS statistics were used to establish a structural equation model to verify the influence path of willingness to consume experiential B&B tourism. Finally, the moderating path of willingness to consume experiential B&B tourism was verified by using multi-group analysis.

SVM and Naïve Bayes Algorithm Comparison for User Sentiment Analysis on Twitter

With the emergence of the Peduli Protect application, which is used by the government to monitor the spread of Covid-19 in Indonesia, it turns out to be reaping the pros and cons of public opinion on Twitter. From this phenomenon, a research was conducted by mapping the sentiment analysis of twitter users towards the Peduli Protect application. This study aims to compare two classification algorithms that are included in the supervised learning category. The two algorithms are Support Vector Machine (SVM) and Naïve Bayes. The two algorithms are implemented in analyzing the sentiment analysis of twitter user reviews on the Peduli Protect application. The dataset used in this research is tweets of twitter users with a total of 4,782 tweets. Then, compared to how much accuracy and processing time required of the two algorithms. The stages of the method in this research are: collecting data from user tweets with a crawling technique, preprocessing text, weighting words using the TF-IDF method, classification using the SVM and Naïve Bayes algorithm, k-fols cross validation test, and drawing conclusions. The results showed that the accuracy of the SMV algorithm with the k-fold test method was 86% and the split 8020 technique resulted in an accuracy of 79%. Meanwhile, the Naïve Bayes algorithm produces an accuracy of 85% with k-fold, and an accuracy of 80% with a split 8020. From these results it can be concluded that both algorithms have the same level of accuracy, only different in processing time, where Naïve Bayes algorithm is faster with time required 0.0094 seconds.

Application of an Improved TF-IDF Method in Literary Text Classification

Literature is extremely important in the advancement of human civilization. Every day, many literary texts of various genres are produced, dating back to ancient times. An urgent concern for managers in the current literary activity is how to classify and save the expanding mass of literary text data for easy access by readers. In the realm of text classification, the TF-IDF algorithm is a widely used classification algorithm. However, there are significant issues with utilizing this approach, including a lack of distribution information inside categories, a lack of distribution information between categories, and an inability to adjust to skewed datasets. It is possible to improve classification accuracy by using the TF-IDF algorithm in this paper’s application situation by exploiting the association between feature words and the quantity of texts in which they appear, while ignoring the variation in feature word distribution across categories. With the purpose of classifying the literary texts in this study, this work proposes an improved IDF method for the problem of feature words appearing several times and having diverse meanings in different fields. The meanings of feature words in distinct domains are separated to increase the trust in the TF-IDF algorithm’s output. Using the improved TF-IDF method suggested in this research with the random forest (RF) classifier, the experimental results show that the classifier has a good classification impact, which can meet the actual work needs, based on comparative experiments on feature dimension selection, feature selection algorithm, feature weight algorithm, and classifier. It has a fair amount of historical significance.

Implementation of Support Vector Machine Algorithm for Shopee Customer Sentiment Anlysis

As the number one largest marketplace in Indonesia based on the criteria for the origin of international stores, Shopee must always improve the quality of its products and services based on reviews from users. Given the huge number of user reviews, it is not effective to identify them by reading one by one. For this reason, an automated system is needed that can read and identify reviews better. Sentiment analysis has proven to do the job. This study aims to conduct a sentiment analysis of shopee product reviews from users who use English. This study applies the Support Vector Machine algorithm to classify the Shopee user review data. To solve this problem, the research was carried out by going through several stages, namely: pre-processing the text of the dataset, performing feature extraction, after that the word weighting was carried out using the TF-IDF method, after clean data was obtained, the SVM algorithm was implemented, for further evaluation of the model. In the results of the study, it was found that the word that most represented the positive opinion of Shopee customers was "Good" with a total of 4684 words. While the word that represents the most negative opinion is "Seller" with 68 words. From the five sentiment analysis models tested, the average value of the confusion matrix is ​​obtained, which are precision=1, recall=0.97, and f1-score=0.98. From this research, it can be concluded that the SVM algorithm is proven to be applicable in conducting sentiment analysis on user reviews of Shopee products with an average accuracy rate of 97.3%.

Prioritizing user concerns in app reviews – A study of requests for new features, enhancements and bug fixes

: App developers spend exhaustive manual efforts towards the identification and prioritization of informative end-user reviews. Informative reviews are those that express end-users’ requests for new features, bug fixes and possible enhancements. Problem Statement : While prior studies have proposed approaches to convert app reviews into actionable knowledge , these are limited in utility due to being domain knowledge dependent or manually-based. : In this study, in order to facilitate app maintenance and evolution cycles, we develop two novel automated prioritization techniques to rank informative reviews, and also compare their performances. : We developed the techniques comprising of entropy, frequency, TF-IDF and sentiment scoring methods using reviews from four popular apps comprising more than 1000 informative reviews in each app. Time and accuracy metrics were then used to measure the performance of our techniques. We performed evaluations where the ranking outcomes generated by our techniques were compared to those provided by regular app users and developers using two rounds of evaluations (internal and external evaluations). : Our results show that the time required for prioritization was in the range of 17–25 s and the accuracy of prioritization was in the range of 73–90%. : These are promising outcomes when compared to prior work, where our outcomes were 4% and 185% better in terms of accuracy and time respectively. Thus, it is anticipated that our proposed techniques could support app developers in identifying and prioritizing informative reviews.

Unsupervised Learning Algorithms for Keyword Extraction in an Undergraduate Thesis

Abstract: The amount of data managed in many academic institutions has increased in recent years, particularly in all the research work done by undergraduate students, who simply use empirical techniques for keyword selection, forgetting existing technical methods to assist their students in this process. Information and communication technologies, such as the platform for integrated research and academic work with responsibility (PILAR), which records information about research projects, such as titles, summaries, and keywords in their various modalities, have gained relevance and importance in the management of these. We proved algorithms with these records of research projects that have been analysed in this study, and predictions were made for each of the nine (09) models of unsupervised machine learning algorithms that were implemented for each of the 7430 records from the dataset. The most efficient way of extracting keywords for this dataset was the TF-IDF method, obtaining 72% accuracy and [0.4786, SD 0.0501] in average extraction time for each thesis file processed by this model. Keywords: Keyword extraction, Machine learning, thesis. university students, unsupervised learning

Designing an information system for the electronic document management of a university: automatic classification of documents

To ensure the effective functioning of the university educational environment, document flow processes automation, which includes the task of documents automatic classification, is of great importance. The article considers the task of classifying university documents by machine learning methods in order to improve the quality of classification. Documents preprocessing was carried out, which made it possible to distinguish significant words in documents, due to which the accuracy of documents classification increased. Described are methods of extracting features from text TF and TF-IDF, which determine keywords by words frequency included in document. A modification of the TF-IDF method is proposed, which consists in calculating the words importance depending on their part of speech. This made it possible to improve the classification quality by highlighting only important and significant words in documents. Suggested is a classification algorithm using a method of support vectors to reduce the documents number involved in classification and a method of k-nearest neighbor for classification. The advantage of this algorithm over the described analogues is shown, which is expressed in the number of mistakenly classified documents decrease.

Free descriptive analysis of Review evaluation using tf-idf method

This paper describes the free description analysis on review comments evaluation using the tf-idf method, which is one of the machine learning methods. This study selected three types of comics YouTube video an evaluation target, performed machine learning by TF-IDF on the comment of the corresponding YouTube video as subjective free description data, extracted the feature values for each video, and visualized them using WordCloud tool. By conducting a questionnaire on these videos, subjective evaluations were obtained by SD method. Comparing the results obtained by machine learning with the results from the questionnaire, this study verifies the validity of the review evaluation using tf-idf method.

Sentiment Analysis using Term based Method for Customers’ Reviews in Amazon Product

Customers’ review in Amazon platform plays an important role for making online purchase decision making, however the reviews are snowballing in E-commerce day by day. The active sharing of customers’ experience and feedback helps to predict the products and retailers’ quality by using natural language processing. This paper will focus on experimental discussion on Amazon products reviews analysis coupled with sentiment analysis using term-based method and N-gram to achieve best findings. The investigation of sentiment analysis on amazon product gain more valuable information on related text to solve problem related services, products information and quality. The analysis begins with data pre-processing of Amazon products reviews then feature extraction with POS tagging and term-based concept. e-Commerce customer’s reviews normally classify different experience into positive, negative and neutral to judge human behavior and emotion towards the purchase products. The major findings discussed in this journal will be using four different classifier and N-grams methods by computing accuracy, precision, recall and F1-Score. TF-IDF method with N-gram shows unigram with Support Vector Machine learning with highest accuracy results for Amazon product customers’ reviews. The score reveals that Support Vector Machine for unigram achieved 82.27% for accuracy, 82% precision, 80% Re-call and 72% F1-Score.

A SURVEY OF RETRIEVAL ALGORITHMS AND THEIR PARALLELIZATION IN LARGE-SCALE SYSTEMS

This article presented a survey of two well-known algorithms, TF-IDF and BM-25 methods, for document ranking on a single CPU and parallel processes via HPC. An amazon review dataset with more than two million reviews was measured to measure the rank parameters. We set up the number of workers for the parallel processing during the experiment, which we selected as one and three. Four benchmarks evaluated the preprocess and reading time, vectorization time, TF-IDF transformation time, and overall time. Results metrics have shown a significant difference in speed.

Research on Classroom Online Teaching Model of "Learning" Wisdom Music on Wireless Network under the Background of Artificial Intelligence.

This article uses a multimodal smart music online teaching method combined with artificial intelligence to address the problem of smart music online teaching and to compensate for the shortcomings of the single modal classification method that only uses audio features for smart music online teaching. The selection of music intelligence models and classification models, as well as the analysis and processing of music characteristics, is the subjects of this article. It mainly studies how to use lyrics and how to combine audio and lyrics to intelligently classify music and teach multimodal and monomodal smart music online. In the online teaching of smart music based on lyrics, on the basis of the traditional wireless network node feature selection method, three parameters of frequency, concentration, and dispersion are introduced to adjust the statistical value of wireless network nodes, and an improved wireless network is proposed. After feature selection, the TFIDF method is used to calculate the weights, and then artificial intelligence is used to perform secondary dimensionality reduction on the lyrics. Experimental data shows that in the process of intelligently classifying lyrics, the accuracy of the traditional wireless network node feature selection method is 58.20%, and the accuracy of the improved wireless network node feature selection method is 67.21%, combined with artificial intelligence and improved wireless, the accuracy of the network node feature selection method is 69.68%. It can be seen that the third method has higher accuracy and lower dimensionality. In the online teaching of multimodal smart music based on audio and lyrics, this article improves the traditional fusion method for the problem of multimodal fusion and compares various fusion methods through experiments. The experimental results show that the improved classification effect of the fusion method is the best, reaching 84.43%, which verifies the feasibility and effectiveness of the method.

Evaluation of Naïve Bayes Classification in Arabic Short Text Classification

In the last few years, and due to the vast widespread of social media applications, texts have become more important and get more attention. Since texts, in general, are carrying a lot of information that can be extracted and analyzed. Several researchers have done significant works in text classification. Within different scripts such as English and other western languages, several challenges and obstacles have been addressed with such a field of research. Regarding the Arabic language, the process is different from other vital languages since Arabic is considered an orthographic language that depends on the word shape. It is not easy to apply the standard text preprocessing techniques since it affects the word meaning. This paper evaluates Arabic short text classification using three standard Naïve Bayes classifiers. In our method, we classify the thesis and dissertations using their titles to perform the classification process. The collected dataset is collected from different repositories by using standard scrapping techniques. Our method classifies the document based on their titles to be placed in the desired specialization. Several preprocessing techniques have been applied, such as (punctuation removal, stop words removal, and space vectorization). For feature extraction, we adopt the TF-IDF method. We implemented three types of Naïve Bayes classifiers which are (Multinomial Naïve Bayes, Complemented Naïve Bayes, and Gaussian Naïve Bayes). The study results showed that Complemented Naïve Bayes Classifier proposed the best performance with (0.84) of accuracy for the testing phase. The results of the study are promising to be applied with different short text classifications.

Automatic Term Recognition Method for Military Domain

With the development of military intelligence, higher requirements are put forward for automatic term recognition in military field. In view of the characteristics of flexible and diverse naming of military requirement documents without annotated corpus, the method of this paper uses the existing military domain core database, and matches the data set and core database by Aho-Corasic algorithm and word segmentation technology, so that the terms to be recognized in the data set can be divided into three types. The possible rules of word formation of military terms are summarized and phrases that conform to the rules of word formation are found in the documents as the term candidate set. The core library and TF-IDF method are used to calculate the value of the candidate terms, and the candidate terms whose value is greater than the threshold are selected iteratively as the real terms. The experimental results show that the F1 value of this method reaches 0.719, which is better than the traditional C-value method. Therefore, the method proposed in this paper can achieve better automatic term recognition effect for military requirement documents without annotation.

Analisis Sentimen Pergerakan Harga Saham Sebuah Perusahaan di Media Sosial Twitter

Twitter has become an essential platform for traders and stock investors worldwide, including major countries like America. Traders rely on Twitter to gather information, similar to how they use Bloomberg terminals. While Twitter provides valuable insights, it also contains negative elements such as false information. The sentiment surrounding stocks on Twitter has been growing, and this study aims to analyze the sentiment of Telkom Indonesia's stock price based on tweets. The research involved several stages. First, data was collected from Twitter and labeled manually into positive, neutral, and negative sentiments. The data then underwent pre-processing, including cleaning and dividing it into training and testing datasets using K-Fold Cross Validation. The data was further weighted using the TF-IDF method, and a training process was conducted to develop a model. The final stage involved testing the accuracy of the model. The study successfully implemented the Multinomial Naïve Bayes (MNB) method, achieving an accuracy of 89.0%. The tweet classification results revealed that out of 1000 tweets, 76.5% were classified as positive, 14.3% as negative, and 9.2% as neutral.

Topic Classification of Islamic Question and Answer Using Naïve Bayes and TF-IDF Method

Information spread through the internet is widely used by people to find anything. One of the most searched information on the internet is information related to Islamic religious knowledge. However, the large amount of information available from various sources makes it difficult for people to find the correct information. Previous researchers have researched this topic, but the dataset used only comes from one source. Therefore, in this study, a classification system for Islamic question and answer topics was built using the Naïve Bayes and TF-IDF methods. This study using 1000 question and answer article data taken from Islamic consultation websites, namely rumahfiqih.com and islamqa.info. The multi-class classification uses five categories which are manually labeled using the category classes on the website. From several test scenarios in this study, the Naïve Bayes classification method using TF-IDF (n-gram level) with a maximum feature of 1000 at a data separation ratio of 70:30 produces the highest accuracy of 81%. The 81% accuracy value was also generated by the SVM classification method, but the difference was in the SVM the highest accuracy value using TF-IDF (word level). It is expected that in the subsequent research will be used more website sources and the use of other classification and feature extraction methods with more optimal value than previous research.

Performance comparison of TF-IDF and Word2Vec models for emotion text classification

Emotion is the human feeling when communicating with other humans or reaction to everyday events. Emotion classification is needed to recognize human emotions from text. This study compare the performance of the TF-IDF and Word2Vec models to represent features in the emotional text classification. We use the support vector machine (SVM) and Multinomial Naïve Bayes (MNB) methods for classification of emotional text on commuter line and transjakarta tweet data. The emotion classification in this study has two steps. The first step classifies data that contain emotion or no emotion. The second step classifies data that contain emotions into five types of emotions i.e. happy, angry, sad, scared, and surprised. This study used three scenarios, namely SVM with TF-IDF, SVM with Word2Vec, and MNB with TF-IDF. The SVM with TF-IDF method generate the highest accuracy compared to other methods in the first dan second steps classification, then followed by the MNB with TF-IDF, and the last is SVM with Word2Vec. Then, the evaluation using precision, recall, and F1-measure results that the SVM with TF-IDF provides the best overall method. This study shows TF-IDF modeling has better performance than Word2Vec modeling and this study improves classification performance results compared to previous studies.