Term Frequency And Inverse Document Frequency Research Articles

Various Approaches for Software vulnerability Assessment and Risk Identification using soft computing Techniques

The detection of software flaws is a viable technique for enhancing the quality of technology and testing management by providing rapid detection of deficiency simulation models until the actual testing phase starts. These prediction outcomes help designers of technology effectively devote their available resources to components that are more vulnerable to deficiencies. In this research investigation, a software bug prediction model is proposed using Deep Learning (DL) approach. The Recurrent Neural Network (RNN) is used for classification of source code including numerous soft computing techniques. Numerous preprocessing and data filtration techniques have been carried out for data balancing and normalization. The Term Frequency and Inverse Document Frequency (TF-IDF) and relation features extraction techniques is used to generate Vector Space Model (VSM). The classification has been done using RNN on both training and validation dataset. The evaluation of performance of proposed work is performed using various real-time and synthetic accessible databases. It is observed from the experimental results that the proposed framework performs better when different evaluated with different datasets

Data oversampling and imbalanced datasets: an investigation of performance for machine learning and feature engineering

The classification of imbalanced datasets is a prominent task in text mining and machine learning. The number of samples in each class is not uniformly distributed; one class contains a large number of samples while the other has a small number. Overfitting of the model occurs as a result of imbalanced datasets, resulting in poor performance. In this study, we compare different oversampling techniques like synthetic minority oversampling technique (SMOTE), support vector machine SMOTE (SVM-SMOTE), Border-line SMOTE, K-means SMOTE, and adaptive synthetic (ADASYN) oversampling to address the issue of imbalanced datasets and enhance the performance of machine learning models. Preprocessing significantly enhances the quality of input data by reducing noise, redundant data, and unnecessary data. This enables the machines to identify crucial patterns that facilitate the extraction of significant and pertinent information from the preprocessed data. This study preprocesses the data using various top-level preprocessing steps. Furthermore, two imbalanced Twitter datasets are used to compare the performance of oversampling techniques with six machine learning models including random forest (RF), SVM, K-nearest neighbor (KNN), AdaBoost (ADA), logistic regression (LR), and decision tree (DT). In addition, the bag of words (BoW) and term frequency and inverse document frequency (TF-IDF) features extraction approaches are used to extract features from the tweets. The experiments indicate that SMOTE and ADASYN perform much better than other techniques thus providing higher accuracy. Additionally, overall results show that SVM with ’linear’ kernel tends to attain the highest accuracy and recall score of 99.67% and 1.00% on ADASYN oversampled datasets and 99.57% accuracy on SMOTE oversampled dataset with TF-IDF features. The SVM model using 10-fold cross-validation experiments achieved 97.40 mean accuracy with a 0.008 standard deviation. Our approach achieved 2.62% greater accuracy as compared to other current methods.

Enhanced MNB Method for SPAM E-mail/SMS Text Detection Using TF-IDF Vectorizer

Spam, whether in the form of SMS or email, poses significant threats by compromising user privacy and potentially leading to unauthorized access to personal data. In the era of smartphones, where users store sensitive information, the risk of cyber-attack through spam messages is heightened. This research addresses the pressing issue of spam SMS and email detection using a dataset comprising 5574 messages from reputable sources. The collection includes contributions from the National University of Singapore SMS Corpus, Grumble text Website, Caroline Tag’s PhD Theses, and SMS Spam Corpus v.0.1 Big. With a meticulous approach encompassing data cleaning, balancing, preprocessing, and exploratory data analysis, the research employs the TF-IDF (Term Frequency and Inverse Document Frequency) vectorizer to enhance the model’s ability to capture the importance of individual words. This foundational work sets the stage for evaluating various machine learning models, including Support Vector Machine, Multinomial Naïve Bayes, Decision Tree, Logistic Regression, Random Forest, AdaBoost, K-Nearest Neighbors, XGBoost, Gradient Boost Classifier, Bagging Classifier, and Extra Tree Classifier. Notably, the Multinomial Naïve Bayes model emerges as a standout performer with 100% accuracy and 97% precision in phishing detection. The research introduces an intuitive user interface, facilitating real-time interactivity for model assessment and offering valuable insights for cybersecurity applications. The study contributes to the advancement of robust cybersecurity systems, emphasizing precision and accuracy in spam SMS and email text detection.

A machine learning-based hybrid recommender framework for smart medical systems.

This article presents a hybrid recommender framework for smart medical systems by introducing two methods to improve service level evaluations and doctor recommendations for patients. The first method uses big data techniques and deep learning algorithms to develop a registration review system in medical institutions. This system outperforms conventional evaluation methods, thus achieving higher accuracy. The second method implements the term frequency and inverse document frequency (TF-IDF) algorithm to construct a model based on the patient's symptom vector space, incorporating score weighting, modified cosine similarity, and K-means clustering. Then, the alternating least squares (ALS) matrix decomposition and user collaborative filtering algorithm are applied to calculate patients' predicted scores for doctors and recommend top-performing doctors. Experimental results show significant improvements in metrics called precision and recall rates compared to conventional methods, making the proposed approach a practical solution for department triage and doctor recommendation in medical appointment platforms.

Non-Functional Requirements Classification Using Machine Learning Algorithms

Non-functional requirements define the quality attribute of a software application, which are necessary to identify in the early stage of software development life cycle. Researchers proposed automatic software Non-functional requirement classification using several Machine Learning (ML) algorithms with a combination of various vectorization techniques. However, using the best combination in Non-functional requirement classification still needs to be clarified. In this paper, we examined whether different combinations of feature extraction techniques and ML algorithms varied in the non-functional requirements classification performance. We also reported the best approach for classifying Non-functional requirements. We conducted the comparative analysis on a publicly available PROMISE_exp dataset containing labelled functional and Non-functional requirements. Initially, we normalized the textual requirements from the dataset; then extracted features through Bag of Words (BoW), Term Frequency and Inverse Document Frequency (TF-IDF), Hashing and Chi-Squared vectorization methods. Finally, we executed the 15 most popular ML algorithms to classify the requirements. The novelty of this work is the empirical analysis to find out the best combination of ML classifier with appropriate vectorization technique, which helps developers to detect Non-functional requirements early and take precise steps. We found that the linear support vector classifier and TF-IDF combination outperform any combinations with an F1-score of 81.5%.

An Alternative Application of Natural Language Processing to Express a Characteristic Feature of Diseases in Japanese Medical Records.

Owing to the linguistic situation, Japanese natural language processing (NLP) requires morphological analyses for word segmentation using dictionary techniques. We aimed to clarify whether it can be substituted with an open-end discovery-based NLP (OD-NLP), which does not use any dictionary techniques. Clinical texts at the first medical visit were collected for comparison of OD-NLP with word dictionary-based-NLP (WD-NLP). Topics were generated in each document using a topic model, which later corresponded to the respective diseases determined in International Statistical Classification of Diseases and Related Health Problems 10 revision. The prediction accuracy and expressivity of each disease were examined in equivalent number of entities/words after filtration with either term frequency and inverse document frequency (TF-IDF) or dominance value (DMV). In documents from 10,520 observed patients, 169,913 entities and 44,758 words were segmented using OD-NLP and WD-NLP, simultaneously. Without filtering, accuracy and recall levels were low, and there was no difference in the harmonic mean of the F-measure between NLPs. However, physicians reported OD-NLP contained more meaningful words than WD-NLP. When datasets were created in an equivalent number of entities/words with TF-IDF, F-measure in OD-NLP was higher than WD-NLP at lower thresholds. When the threshold increased, the number of datasets created decreased, resulting in increased values of F-measure, although the differences disappeared. Two datasets near the maximum threshold showing differences in F-measure were examined whether their topics were associated with diseases. The results showed that more diseases were found in OD-NLP at lower thresholds, indicating that the topics described characteristics of diseases. The superiority remained as much as that of TF-IDF when filtration was changed to DMV. The current findings prefer the use of OD-NLP to express characteristics of diseases from Japanese clinical texts and may help in the construction of document summaries and retrieval in clinical settings.

A Machine Learning-Based Technique with Intelligent WordNet Lemmatize for Twitter Sentiment Analysis

Laterally with the birth of the Internet, the fast growth of mobile strategies has democratised content production owing to the widespread usage of social media, resulting in a detonation of short informal writings. Twitter is microblogging short text and social networking services, with posted millions of quick messages. Twitter analysis addresses the topic of interpreting users’ tweets in terms of ideas, interests, and views in a range of settings and fields. This type of study can be useful for a variation of academics and applications that need knowing people’s perspectives on a given topic or event. Although sentiment examination of these texts is useful for a variety of reasons, it is typically seen as a difficult undertaking due to the fact that these messages are frequently short, informal, loud, and rich in linguistic ambiguities such as polysemy. Furthermore, most contemporary sentiment analysis algorithms are based on clean data. In this paper, we offers a machine-learning-based sentiment analysis method that extracts features from Term Frequency and Inverse Document Frequency (TF-IDF) and needs to apply deep intelligent wordnet lemmatize to improve the excellence of tweets by removing noise. We also utilise the Random Forest network to detect the emotion of a tweet. To authenticate the proposed approach performance, we conduct extensive tests on publically accessible datasets, and the findings reveal that the suggested technique significantly outperforms sentiment classification in multi-class emotion text data.

Sentiment analysis through twitter as a mechanism for assessing university satisfaction

Currently, the data generated in the university environment related to the perception of satisfaction is generated through surveys with categorical response questions defined on a Likert scale, with factors already defined to be evaluated, applied once per academic semester, which generates very biased information. This leads us to wonder why this survey is applied only once and why it only asks about some factors. The objective of the article is to demonstrate the feasibility of a proposal to determine the degree of perception of student satisfaction through the use of data science and natural language processing (NLP), supported by the social network twitter, as an element of data collection. As a result of the application of this proposal based on data science, it was possible to determine the level of student satisfaction, being 57.27%, through sentiment analysis using the Python library "NLTK"; Thus, it was also possible to extract texts linked to the relevant factors of teaching performance to achieve student satisfaction, through the term frequency and inverse document frequency (TF-IDF) approach, these being those linked to the use of tools of simulation in the virtual learning process.

An improved anti‐phishing model utilizing TF‐IDF and AdaBoost

SummaryA behavior over the internet, which results into false utilization of the resources is termed as phishing and the techniques to identify them are termed as anti‐phishing frames and algorithms. This research work introduces a new relative similarity approach utilizing similarity index co‐relation by applying Cosine similarity. In order to incorporate the co‐relation, term frequency and inverse document frequency (TF‐IDF) is also calculated. AdaBoost is employed to improve the classification accuracy of the proposed work by means of enhancing the performance of weak learner to classify website into legitimate and phishing one. The simulation analysis has shown that the proposed work outperformed the existing works in terms of accuracy, TPR, and precision of detecting phishing sites to offer secure browsing. Overall model evaluation against 50,000 URL instances have demonstrated an average precision, TPR, F‐measure, accuracy and execution time of 98.64%, 96.52%, 97.56%, 98.014%, and 3.485 s, respectively for phishing detection. This proves the success of the proposed model in shielding phishing attack.

Analysis of sentiments on the onset of Covid-19 using Machine Learning Techniques

The novel coronavirus (Covid-19) pandemic has struck the whole world and is one of the most striking topics on social media platforms. Sentiment outbreak on social media enduring various thoughts, opinions, and emotions about the Covid-19 disease, expressing views they are feeling presently. Analyzing sentiments helps to yield better results. Gathering data from different blogging sites like Facebook, Twitter, Weibo, YouTube, Instagram, etc., and Twitter is the largest repository. Videos, text, and audio were also collected from repositories. Sentiment analysis uses opinion mining to acquire the sentiments of its users and categorizes them accordingly as positive, negative, and neutral. Analytical and machine learning classification is implemented to 3586 tweets collected in different time frames. In this paper, sentiment analysis was performed on tweets accumulated during the Covid-19 pandemic, Coronavirus disease. Tweets are collected from the Twitter database using Hydrator a web-based application. Data-preprocessing removes all the noise, outliers from the raw data. With Natural Language Toolkit (NLTK), text classification for sentiment analysis and calculate the score subjective polarity, counts, and sentiment distribution. N-gram is used in textual mining -and Natural Language Processing for a continuous sequence of words in a text or document applying uni-gram, bi-gram, and tri-gram for statistical computation. Term frequency and Inverse document frequency (TF-IDF) is a feature extraction technique that converts textual data into numeric form. Vectorize data feed to our model to obtain insights from linguistic data. Linear SVC, MultinomialNB, GBM, and Random Forest classifier with Tfidf classification model applied to our proposed model. Linear Support Vector classification performs better than the other two classifiers. Results depict that RF performs better.

Research-Based on Telecommunication in Mobile Service Provider's Performance using Enhanced Naive Bayes Classifier

In recent years, mobile service providers have rapidly expanded across all countries. Considering unpredictable development trends, mobile service providers are essential to knowledge-based service businesses. Performance may be improved by creating and disseminating new information through innovation activities based on the usage of business intelligence. This research examined the performance of mobile service providers across all countries utilizing an enhanced Naive Bayes classifier based on telecommunication. In comparison to quantitative variables, the naive Bayes performs quite well. In the beginning, data is collected and the normalization technique is used for data preprocessing. Feature extraction is carried out using “Term Frequency and Inverse Document Frequency (TF-IDF)”. “Decision Tree algorithm” is used for data analysis. Then the feature is selected using a two-stage Markov blanket algorithm. Enhanced Naïve Bayes Classifier is the proposed algorithm for telecommunication analysis and at last, the performance of the system is analyzed. This proposed algorithm is used to compare the mobile service provider's performances with existing algorithms. The proposed method measures the following metrics as Throughput, Packet loss, Packet duplication, and User quality of experience. The proposed algorithm is more effective and produces better results.

Early Detection of Seasonal Outbreaks from Twitter Data Using Machine Learning Approaches

Seasonal outbreaks have several different periods that occur primarily during winter in temperate regions, while influenza may occur throughout the year in tropical regions, triggering outbreaks more irregularly. Similarly, dengue occurs in the star of the rainy season in early May and reaches its peak in late June. Dengue and flu brought an impact on various countries in the years 2017–2019 and streaming Twitter data reveals the status of dengue and flu outbreaks in the most affected regions. This research work presents that Social Media Analysis (SMA) can be used as a detector of the epidemic outbreak and to understand the sentiment of social media users regarding various diseases. Providing awareness about seasonal outbreaks through SMA is an effective approach for researchers and healthcare responders to detect the early outbreaks. The proposed model aims to find the sentiment about the disease in tweets, and the seasonal outbreaks-related tweets are classified into two classes as disease positive and disease negative. This work proposes a machine-learning-based approach to detect dengue and flu outbreaks in social media platform Twitter, using four machine learning algorithms: Random Forest (RF), K-Nearest Neighbor (KNN), Support Vector Machine (SVM), and Decision Tree (DT), with the help of Term Frequency and Inverse Document Frequency (TF-IDF). For experimental analysis, two datasets (dengue and flu) are analyzed individually. The experimental results show that the RF classifier has outperformed the comparison models in terms of improved accuracy, precision, recall, F1-measure, and Receiver Operating Characteristic (ROC) curve. The proposed work offers favorable performance with total precision, accuracy, recall, and F1-measure ranging from 84% to 88% for conventional machine learning techniques.

Comparison of machine learning algorithms for content based personality resolution of tweets

The content of social media (SM) is expanding quickly with individuals sharing their feelings in a variety of ways, all of which depict their personalities to varying degrees. This study endeavored to build a system that could predict an individual's personality through SM conversation. Four BIG5 personality items (i.e. Extraversion (EXT), Consciousness (CON), Agreeable (AGR) and Openness to Experiences (OPN) equivalent to the Myers–Briggs Type Indicator (MBTI)) were predicted using six supervised machine learning (SML) algorithms. In order to handle unstructured and unbalanced SM conversations, three feature extraction methods (i.e. term frequency and inverse document frequency (TF-IDF), the bag of words (BOW) and the global vector for word representation (GloVe)) were used. The TF-IDF method of feature extraction produces 2–9% higher accuracy than word2vec representation. GloVe is advocated as a better feature extractor because it maintains the spatial information of words.

Incremental learning for text categorization using rough set boundary based optimized Support Vector Neural Network

PurposeText mining has been used for various knowledge discovery based applications, and thus, a lot of research has been contributed towards it. Latest trending research in the text mining is adopting the incremental learning data, as it is economical while dealing with large volume of information.Design/methodology/approachThe primary intention of this research is to design and develop a technique for incremental text categorization using optimized Support Vector Neural Network (SVNN). The proposed technique involves four major steps, such as pre-processing, feature selection, classification and feature extraction. Initially, the data is pre-processed based on stop word removal and stemming. Then, the feature extraction is done by extracting semantic word-based features and Term Frequency and Inverse Document Frequency (TF-IDF). From the extracted features, the important features are selected using Bhattacharya distance measure and the features are subjected as the input to the proposed classifier. The proposed classifier performs incremental learning using SVNN, wherein the weights are bounded in a limit using rough set theory. Moreover, for the optimal selection of weights in SVNN, Moth Search (MS) algorithm is used. Thus, the proposed classifier, named Rough set MS-SVNN, performs the text categorization for the incremental data, given as the input.FindingsFor the experimentation, the 20 News group dataset, and the Reuters dataset are used. Simulation results indicate that the proposed Rough set based MS-SVNN has achieved 0.7743, 0.7774 and 0.7745 for the precision, recall and F-measure, respectively.Originality/valueIn this paper, an online incremental learner is developed for the text categorization. The text categorization is done by developing the Rough set MS-SVNN classifier, which classifies the incoming texts based on the boundary condition evaluated by the Rough set theory, and the optimal weights from the MS. The proposed online text categorization scheme has the basic steps, like pre-processing, feature extraction, feature selection and classification. The pre-processing is carried out to identify the unique words from the dataset, and the features like semantic word-based features and TF-IDF are obtained from the keyword set. Feature selection is done by setting a minimum Bhattacharya distance measure, and the selected features are provided to the proposed Rough set MS-SVNN for the classification.

From Words to Numbers: Getting Started with Text Analysis for Applied Social Scientists

Objectives: With texts as unstructured data everywhere, text analysis or natural language processing (NLP) is a rapidly growing academic field that has great potential for novel research among many applied social scientists and practitioners. This paper presents a practical introduction to NLP using Python as a useful tool for text analysis. Methods: Starting with installation of Python and an external library for NLP, this paper describes a step-by-step process of data preparation, transformation, and summarization for text data using examples. The example texts were obtained from a transcribed business meeting record of a multinational company based in Helsinki. Results: From the initial unstructured text data having numerous irrelevant elements, the data preparation procedures of tokenization, removing stop words, stemming, and lemmatization result in a set of words useful for main analyses. The next step of transforming the words to numbers was conducted using a bag-of-words method by assigning a unique value to each word in a matrix. As the last step, the matrix is computed for frequency summarization using TF-IDF (Term Frequency and Inverse Document Frequency). Conclusions: Unlike structured data, many unstructured text data are not generated for the purpose of data analysis. With numeric data reproduced by the process presented in this paper, communication researchers can perform various statistical methods or use machine learning algorithms. Beyond the scope of this paper, it is strongly recommended to study statistics and computational linguistics as well as have a working knowledge on R and/or Python for advanced text analysis.

Contextual Word Embeddings and Topic Modeling in Healthy Dieting and Obesity.

An alarming proportion of the US population is overweight. Obesity increases the risk of illnesses such as diabetes and cardiovascular diseases. In this paper, we propose the Contextual Word Embeddings (ContWEB) framework that aims to build contextual word embeddings on the relationship between obesity and healthy eating from the crowd domain (Twitter) and the expert domain (PubMed). For this purpose, our work is based on a pipeline model that consists of a chain of processing elements as follows: (1) to use term frequency and inverse document frequency (TF-IDF) and Word2Vec in the data collected from the crowd and expert domains; (2) to apply natural language processing (NLP) algorithms to the corpus; (3) to construct social word embeddings by sentiment analysis; (4) to discover the contextual word embeddings using co-occurrence and conditional probability; (5) to find an optimal number of topics in a topic modeling with the obesity and healthy dieting corpus; (6) to extract latent features extracted using Latent Dirichlet Allocation (LDA). The ContWEB framework has been implemented on Apache Spark and TensorFlow platforms. We have evaluated the ContWEB framework in terms of the effectiveness in contextual word embeddings constructed from the crowd and the expert domains. We conclude that the ContWEB framework would be useful in enhancing the decision-making process for healthy eating and obesity prevention.

Word and Character Information Aware Neural Model for Emotional Analysis

Background: Social media texts are often highly unstructured in accordance with the presence of hashtags, emojis and URLs occurring in abundance. Thus, a sentiment or emotion analysis on these kinds of texts becomes very difficult. The difficulty increases even more when such texts are in local languages like Arabic. Methods: This work utilizes novel deep learning architectures in the form of character-level Convolutional Neural Network (CNN) module and the word-level Recurrent Neural Network (RNN) module to produce a hybrid architecture that makes use of the character level analysis and the word level analysis to obtain state-of-the-art results on a totally new Arabic Emotions dataset. Results: The proposed method works the best among the traditional bag-of-words and Term Frequency and Inverse Document Frequency methods for emotion analysis. It also outperforms the state-of-the-art deep learning methods which are known to perform very well in an English corpus. Conclusion: The proposed deep end-to-end architecture utilizes the character level information from a text through the Character CNN Module and the word level information from a text through the Word-Level RNN Module.

An Analysis of Sindhi Annotated Corpus using Supervised Machine Learning Methods

The linguistic corpus of Sindhi language is significant for computational linguistics process, machine learning process, language features identification and analysis, semantic and sentiment analysis, information retrieval and so on. There is little computational linguistics work done on Sindhi text whereas, English, Arabic, Urdu and some other languages are fully resourced computationally. The grammar and morphemes of these languages are analyzed properly using dissimilar machine learning methods. The development and research work regarding computational linguistics are in progress on Sindhi language at this time. This study is planned to develop the Sindhi annotated corpus using universal POS (Part of Speech) tag set and Sindhi POS tag set for the purpose of language features and variation analysis. The features are extracted using TF-IDF (Term Frequency and Inverse Document Frequency) technique. The supervised machine learning model is developed to assess the annotated corpus to know the grammatical annotation of Sindhi language. The model is trained with 80% of annotated corpus and tested with 20% of test set. The cross-validation technique with 10-folds is utilized to evaluate and validate the model. The results of model show the better performance of model as well as confirm the proper annotation to Sindhi corpus. This study described a number of research gaps to work more on topic modeling, language variation, sentiment and semantic analysis of Sindhi language.

An Emotion-Aware Personalized Music Recommendation System Using a Convolutional Neural Networks Approach

Recommending music based on a user’s music preference is a way to improve user listening experience. Finding the correlation between the user data (e.g., location, time of the day, music listening history, emotion, etc.) and the music is a challenging task. In this paper, we propose an emotion-aware personalized music recommendation system (EPMRS) to extract the correlation between the user data and the music. To achieve this correlation, we combine the outputs of two approaches: the deep convolutional neural networks (DCNN) approach and the weighted feature extraction (WFE) approach. The DCNN approach is used to extract the latent features from music data (e.g., audio signals and corresponding metadata) for classification. In the WFE approach, we generate the implicit user rating for music to extract the correlation between the user data and the music data. In the WFE approach, we use the term-frequency and inverse document frequency (TF-IDF) approach to generate the implicit user ratings for the music. Later, the EPMRS recommends songs to the user based on calculated implicit user rating for the music. We use the million songs dataset (MSD) to train the EPMRS. For performance comparison, we take the content similarity music recommendation system (CSMRS) as well as the personalized music recommendation system based on electroencephalography feedback (PMRSE) as the baseline systems. Experimental results show that the EPMRS produces better accuracy of music recommendations than the CSMRS and the PMRSE. Moreover, we build the Android and iOS APPs to get realistic data of user experience on the EPMRS. The collected feedback from anonymous users also show that the EPMRS sufficiently reflect their preference on music.

Chinese microblog rumor detection based on deep sequence context

SummaryRumor is one of the main problems in social media, which often shows deeply and rapidly undesirable affection on the society. Although many rumor detection models consider content features and social features, all of them are based on the word independence assumption, which lacks the sequence context. Thus, if we use some words that often appear in rumors, our posts will be recognized as a rumor. To solve this problem, we propose a deep sequence context model (DSCM) for Chinese microblog rumor detection. This model considers two important factors of rumors: falsity and influence. Firstly, to learn falsity, we abolish the word independence assumption and use long short‐term memory (LSTM) units to capture bi‐direction sequence context information in content. Secondly, to learn influence, we combine the deep sequence context information with social features to learn the connection between content and social features. In our experiment, our results show that our approach outperforms several state‐of‐the‐art machine learning approaches, including term frequency and inverse document frequency (TFIDF), LSTM, and gated recurrent unit (GRU) in rumor detection.