Term Weighting Schemes Research Articles

A generic multi-level framework for building term-weighting schemes in text classification

Abstract Term weighting is essential for text classification tasks, and thus various supervised term-weighting (STW) methods have been designed and presented in recent years, such as TF (term frequency)-IG (information gain), TF-MI (mutual information), TF-RF (relevance frequency), and TF-IDF (inverse document frequency)-ICSDF (inverse class space density frequency). Unlike other schemes, TF-IDF-ICSDF considers not only the local factor (i.e. TF) and the category factor (i.e. ICSDF) but also the global factor (i.e. IDF) in the weighting process. Hence, a natural question is whether IDF is really useful for improving the classification performance of STW schemes. To explore this issue, a generic multi-level framework composed of term-level, text-level, and category-level is first established, which corresponds to local factor, global factor, and category factor, respectively. Based on the generic multi-level framework, a new two-level STW method, TF-ICSDF, can be generated by removing the IDF from the TF-IDF-ICSDF scheme. Conversely, we also integrated the IDF with other two-level STW schemes (e.g. TF-IG, TF-MI, TF-RF) to obtain several three-level STW schemes. We verified the general classification performance of our proposed STW schemes on three open benchmark datasets. The results manifest that performance can usually be boosted if IDF is incorporated into the STW schemes, indicating that weighting terms utilizing the IDF factor could provide better text representation. Therefore, the generic multi-level framework and STW schemes we proposed are effective.

Multi-objective genetic programming strategies for topic-based search with a focus on diversity and global recall

Topic-based search systems retrieve items by contextualizing the information seeking process on a topic of interest to the user. A key issue in topic-based search of text resources is how to automatically generate multiple queries that reflect the topic of interest in such a way that precision, recall, and diversity are achieved. The problem of generating topic-based queries can be effectively addressed by Multi-Objective Evolutionary Algorithms, which have shown promising results. However, two common problems with such an approach are loss of diversity and low global recall when combining results from multiple queries. This work proposes a family of Multi-Objective Genetic Programming strategies based on objective functions that attempt to maximize precision and recall while minimizing the similarity among the retrieved results. To this end, we define three novel objective functions based on result set similarity and on the information theoretic notion of entropy. Extensive experiments allow us to conclude that while the proposed strategies significantly improve precision after a few generations, only some of them are able to maintain or improve global recall. A comparative analysis against previous strategies based on Multi-Objective Evolutionary Algorithms, indicates that the proposed approach is superior in terms of precision and global recall. Furthermore, when compared to query-term-selection methods based on existing state-of-the-art term-weighting schemes, the presented Multi-Objective Genetic Programming strategies demonstrate significantly higher levels of precision, recall, and F1-score, while maintaining competitive global recall. Finally, we identify the strengths and limitations of the strategies and conclude that the choice of objectives to be maximized or minimized should be guided by the application at hand.

TF-TDA: A Novel Supervised Term Weighting Scheme for Sentiment Analysis

In text classification tasks, such as sentiment analysis (SA), feature representation and weighting schemes play a crucial role in classification performance. Traditional term weighting schemes depend on the term frequency within the entire document collection; therefore, they are called unsupervised term weighting (UTW) schemes. One of the most popular UTW schemes is term frequency–inverse document frequency (TF-IDF); however, this is not sufficient for SA tasks. Newer weighting schemes have been developed to take advantage of the membership of documents in their categories. These are called supervised term weighting (STW) schemes; however, most of them weigh the extracted features without considering the characteristics of some noisy features and data imbalances. Therefore, in this study, a novel STW approach was proposed, known as term frequency–term discrimination ability (TF-TDA). TF-TDA mainly presents the extracted features with different degrees of discrimination by categorizing them into several groups. Subsequently, each group is weighted based on its contribution. The proposed method was examined over four SA datasets using naive Bayes (NB) and support vector machine (SVM) models. The experimental results proved the superiority of TF-TDA over two baseline term weighting approaches, with improvements ranging from 0.52% to 3.99% in the F1 score. The statistical test results verified the significant improvement obtained by TF-TDA in most cases, where the p-value ranged from 0.0000597 to 0.0455.

Parallel multi-head attention and term-weighted question embedding for medical visual question answering.

The goal of medical visual question answering (Med-VQA) is to correctly answer a clinical question posed by a medical image. Medical images are fundamentally different from images in the general domain. As a result, using general domain Visual Question Answering (VQA) models to the medical domain is impossible. Furthermore, the large-scale data required by VQA models is rarely available in the medical arena. Existing approaches of medical visual question answering often rely on transfer learning with external data to generate good image feature representation and use cross-modal fusion of visual and language features to acclimate to the lack of labelled data. This research provides a new parallel multi-head attention framework (MaMVQA) for dealing with Med-VQA without the use of external data. The proposed framework addresses image feature extraction using the unsupervised Denoising Auto-Encoder (DAE) and language feature extraction using term-weighted question embedding. In addition, we present qf-MI, a unique supervised term-weighting (STW) scheme based on the concept of mutual information (MI) between the word and the corresponding class label. Extensive experimental findings on the VQA-RAD public medical VQA benchmark show that the proposed methodology outperforms previous state-of-the-art methods in terms of accuracy while requiring no external data to train the model. Remarkably, the presented MaMVQA model achieved significantly increased accuracy in predicting answers to both close-ended (78.68%) and open-ended (55.31%) questions. Also, an extensive set of ablations are studied to demonstrate the significance of individual components of the system.

Automatic Diagnosis of COVID-19 Patients from Unstructured Data Based on a Novel Weighting Scheme

The extraction of features from unstructured clinical data of Covid-19 patients is critical for guiding clinical decision-making and diagnosing this viral disease. Furthermore, an early and accurate diagnosis of COVID-19 can reduce the burden on healthcare systems. In this paper, an improved Term Weighting technique combined with Parts-Of-Speech (POS) Tagging is proposed to reduce dimensions for automatic and effective classification of clinical text related to Covid-19 disease. Term Frequency-Inverse Document Frequency (TF-IDF) is the most often used term weighting scheme (TWS). However, TF-IDF has several developments to improve its drawbacks, in particular, it is not efficient enough to classify text by assigning effective weights to the terms in unstructured data. In this research, we proposed a modification term weighting scheme: RTF-C-IEF and compare the proposed model with four extraction methods: TF, TF-IDF, TF-IHF, and TF-IEF. The experiment was conducted on two new datasets for COVID-19 patients. The first dataset was collected from government hospitals in Iraq with 3053 clinical records, and the second dataset with 1446 clinical reports, was collected from several different websites. Based on the experimental results using several popular classifiers applied to the datasets of Covid-19, we observe that the proposed scheme RTF-C-IEF achieves is a consistent performer with the best scores in most of the experiments. Further, the modified RTF-C-IEF proposed in the study outperformed the original scheme and other employed term weighting methods in most experiments. Thus, the proper selection of term weighting scheme among the different methods improves the performance of the classifier and helps to find the informative term.

Automated Categorization of Research Papers with MONO Supervised Term Weighting in RECApp

Natural Language Processing, specifically text classification or text categorization, has become a trend in computer science. Commonly, text classification is used to categorize large amounts of data to allocate less time to retrieve information. Students, as well as research advisers and panelists, take extra effort and time in classifying research documents. To solve this problem, the researchers used state-of-the-art supervised term weighting schemes, namely: TF-MONO and SQRTF-MONO and its application to machine learning algorithms: K-Nearest Neighbor, Linear Support Vector, Naive Bayes Classifiers, creating a total of six classifier models to ascertain which of them performs optimally in classifying research documents while utilizing Optical Character Recognition for text extraction. The results showed that among all classification models trained, SQRTF-MONO and Linear SVC outperformed all other models with an F1 score of 0.94 both in the abstract and the background of the study datasets. In conclusion, the developed classification model and application prototype can be a tool to help researchers, advisers, and panelists to lessen the time spent in classifying research documents.

USTW Vs. STW: A Comparative Analysis for Exam Question Classification based on Bloom’s Taxonomy

Bloom’s Taxonomy (BT) is widely used in educational institutions to produce high-quality exam papers to evaluate students’ knowledge at different cognitive levels. However, manual question labeling takes a long time, and not all evaluators are familiar with BT. The researchers worked to automate the exam question classification process based on BT as a solution. Enhancement in term weighting is one of the ways to increase classification accuracy while working with text data. However, all the past work on the term weighting in exam question classification focused on unsupervised term weighting (USTW) schemes. The supervised term weighting (STW) schemes showed effectiveness in text classification but were not addressed in past studies of exam question classification. As a result, this study focused on the effectiveness of STW in classifying exam questions using BT. Hence, this research performed a comparative analysis between the USTW schemes and STW for exam question classification. The STW schemes used in this study are TF-ICF, TF-IDF-ICF, and TF-IDF-ICSDF, whereas the USTW schemes used for comparison are TF-IDF, ETF-IDF, and TFPOS-IDF. This study used Support Vector Machines (SVM), Na¨ıve Bayes (NB), and Multilayer Perceptron (MLP) to train the model. Accuracy and F1 score were used in this study to evaluate the classification result. The experiment result showed that overall, the STW scheme TF-ICF outperformed all the other schemes, followed by the USTW scheme ETF-IDF. Both the ETF-IDF and TFPOS-IDF outperformed standard TFIDF. The outcome of this study indicates the future research direction where the combination of STW and USTW schemes may increase the Accuracy of BT-based exam question classification.

Evaluation of Average Term Occurrences Weighting Technique for Arabic Textual Information Retrieval

Information retrieval of documents is an important process in the current time, and the vector space retrieval model uses a term weighting scheme as a basic method for matching queries with documents. Term frequency-Inverse document frequency is a widely used and famous term weighting scheme, and many studies proved its effectiveness in information retrieval. However, this term weighting scheme has some drawbacks like retrieving irrelevant documents, which sometimes reduces effectiveness. From this point, a new term weighting scheme called Term Frequency with Average Term Occurrence was proposed and experienced in the English language to minimize retrieving unnecessary documents. In this paper, an information retrieval system is built for the Arabic language, and Open-Source Arabic Corpora was used to complete experiments. Calculations were made using two schemes which are traditional Term frequency-inverse Document Frequency and proposed Term Frequency with Average Term Occurrence. After that, comparisons of results were made using evaluation measures. With all obtained queries, four case studies with two approaches (stop word removal and stemming) are implemented. In English experiments, stop word removal was applied with another discriminative approach, which calculates the centroid of documents. After the analysis of the results, it was found that the proposed scheme is applicable on Arabic text and applied approaches enhance IR effectiveness if they are both implemented. Furthermore, it was found that stop word removal has a favorable effect on both schemes which was also proved in English experiments.

Binned Term Count: An Alternative to Term Frequency for Text Categorization

In text categorization, a well-known problem related to document length is that larger term counts in longer documents cause classification algorithms to become biased. The effect of document length can be eliminated by normalizing term counts, thus reducing the bias towards longer documents. This gives us term frequency (TF), which in conjunction with inverse document frequency (IDF) became the most commonly used term weighting scheme to capture the importance of a term in a document and corpus. However, normalization may cause term frequency of a term in a related document to become equal or smaller than its term frequency in an unrelated document, thus perturbing a term’s strength from its true worth. In this paper, we solve this problem by introducing a non-linear mapping of term frequency. This alternative to TF is called binned term count (BTC). The newly proposed term frequency factor trims large term counts before normalization, thus moderating the normalization effect on large documents. To investigate the effectiveness of BTC, we compare it against the original TF and its more recently proposed alternative named modified term frequency (MTF). In our experiments, each of these term frequency factors (BTC, TF, and MTF) is combined with four well-known collection frequency factors (IDF), RF, IGM, and MONO and the performance of each of the resulting term weighting schemes is evaluated on three standard datasets (Reuters (R8-21578), 20-Newsgroups, and WebKB) using support vector machines and K-nearest neighbor classifiers. To determine whether BTC is statistically better than TF and MTF, we have applied the paired two-sided t-test on the macro F1 results. Overall, BTC is found to be 52% statistically significant than TF and MTF. Furthermore, the highest macro F1 value on the three datasets was achieved by BTC-based term weighting schemes.

Automated Radiographic Report Generation Purely on Transformer: A Multicriteria Supervised Approach.

Automated radiographic report generation is challenging in at least two aspects. First, medical images are very similar to each other and the visual differences of clinic importance are often fine-grained. Second, the disease-related words may be submerged by many similar sentences describing the common content of the images, causing the abnormal to be misinterpreted as the normal in the worst case. To tackle these challenges, this paper proposes a pure transformer-based framework to jointly enforce better visual-textual alignment, multi-label diagnostic classification, and word importance weighting, to facilitate report generation. To the best of our knowledge, this is the first pure transformer-based framework for medical report generation, which enjoys the capacity of transformer in learning long range dependencies for both image regions and sentence words. Specifically, for the first challenge, we design a novel mechanism to embed an auxiliary image-text matching objective into the transformer's encoder-decoder structure, so that better correlated image and text features could be learned to help a report to discriminate similar images. For the second challenge, we integrate an additional multi-label classification task into our framework to guide the model in making correct diagnostic predictions. Also, a term-weighting scheme is proposed to reflect the importance of words for training so that our model would not miss key discriminative information. Our work achieves promising performance over the state-of-the-arts on two benchmark datasets, including the largest dataset MIMIC-CXR.

Comparative study of term-weighting schemes for environmental big data using machine learning

Widely-used term-weighting schemes and machine learning (ML) classifiers with default parameter settings were assessed for their performance when applied to environmental big data analysis. Five term-weighting schemes [term frequency (TF), TF–inverse document frequency (TF-IDF), Best Match 25 (BM25), TF–inverse gravity moment (TF-IGM), and TF–IDF–inverse class frequency (TF-IDF-ICF)] and five different ML classifiers [support vector machine (SVM), Naive Bayes (NB), logistic regression (LR), random forest (RF), and extreme gradient boosting (XGBoost)] were tested. The optimal text-classification scheme and classifier were TF-IDF-ICF and LR, respectively. Based on evaluation criteria, their combination resulted in the best performance of all scheme and classifier combinations for the full environmental data analysis. Category classification performance differed according to the environmental section (climate, air, water, or waste/garbage), with the best performance being achieved for climate, and the poorest for water. This demonstrated the importance of selecting term-weighting schemes and ML classifiers in human-generated environmental big data analysis.

An Assessment of Nature-Inspired Algorithms for Text Feature Selection

This paper provides a comprehensive assessment of feature selection (FS) methods that are originated from nature-inspired (NI) meta-heuristics, where two well-known filter-based FS methods are also included for comparison. The performances of the considered methods are compared on two different high-dimensional and real-world text datasets against the accuracy, the number of selected features, and computation time. This study differs from existing studies in terms of the extent of experimental analyses performed under different circumstances where classifier, feature model, and term weighting scheme are different. The results of the extensive experiments indicate that NI algorithms produce slightly different results than filter-based methods for the problem of the text FS. However, filter-based methods often provide better results by using a lower number of features and computation times.

An Enhanced Term Weighted Question Embedding for Visual Question Answering

Visual Question Answering (VQA) is a multi-modal AI-complete task of answering natural language questions about images. Literature solved VQA with a three-phase pipeline: image and question featurisation, multi-modal feature fusion and answer generation or prediction. Most of the works have given attention to the second phase, where multi-modal features get combined ignoring the effect of individual input features. This work investigates VQA’s natural language question embedding phase by proposing a new question featurisation framework based on Supervised Term Weighting (STW) schemes. In addition, two new STW schemes integrating text semantics, qf.cos and tf.rf.sim, have been introduced to boost the framework’s performance. A series of tests on the DAQUAR VQA dataset is used to compare the new system to conventional pre-trained word embedding. Over the past few years, STW schemes have been commonly used in text classification research. In light of this, tests are carried out to verify the effectiveness of the two newly proposed STW schemes in the general text classification task.

Detecting Emotion in Indonesian Tweets: A Term-Weighting Scheme Study

Background: Term-weighting plays a key role in detecting emotion in texts. Studies in term-weighting schemes aim to improve short text classification by distinguishing terms accurately. Objective: This study aims to formulate the best term-weighting schemes and discover the relationship between n-gram combinations and different classification algorithms in detecting emotion in Twitter texts. Methods: The data used was the Indonesian Twitter Emotion Dataset, with features generated through different n-gram combinations. Two approaches assign weights to the features. Tests were carried out using ten-fold cross-validation on three classification algorithms. The performance of the model was measured using accuracy and F1 score. Results: The term-weighting schemes with the highest performance are Term Frequency-Inverse Category Frequency (TF-ICF) and Term Frequency-Relevance Frequency (TF-RF). The scheme with a supervised approach performed better than the unsupervised one. However, we did not find a consistent advantage as some of the experiments found that Term Frequency-Inverse Document Frequency (TF-IDF) also performed exceptionally well. The traditional TF-IDF method remains worth considering as a term-weighting scheme. Conclusion: This study provides recommendations for emotion detection in texts. Future studies can benefit from dealing with imbalances in the dataset to provide better performance. Keywords: Emotion Detection, Feature Engineering, Term-Weighting, Text Mining

ETFPOS-IDF: A Novel Term Weighting Scheme for Examination Question Classification Based on Bloom’s Taxonomy

Numerous earlier studies focused on the term weighting scheme to increase examination question classification accuracy based on Bloom’s Taxonomy (BT). While determining the cognitive level of the examination question, all the terms present in the question are not equally significant. Verbs are the most important parts of speech while assigning weights to the terms. However, two types of verbs may be present in the questions: BT and supporting. BT verbs have a higher impact on determining the cognitive level of a question than supporting verbs. Nevertheless, the proposed schemes of past studies assigned equal weight to both types of verbs. Therefore, this study aims to introduce the term weighting scheme ETFPOS-IDF, which assigns BT a higher weight than supporting verbs. The BT verbs were identified based on their position in the questions. Three datasets and three classifiers: Support Vector Machine, Artificial Neural Network, and Random Forest, were used in this study. Two evaluation metrics: accuracy and F1 score, were used to evaluate the performance of the proposed model. The experiment results showed that the proposed ETFPOS-IDF outperformed all the schemes introduced by earlier studies in examination question classification and achieved 0.749 in accuracy and 0.746 in F1 score. The finding of this study demonstrates that distinguishing between different verb types is significant in reducing the misclassification of examination questions. This research contributed by introducing a novel term weighting scheme in classifying examination questions based on BT. Future work may involve identifying the optimal weight for both types of verbs, evaluating the proposed scheme with a larger dataset, and comparing the performance with deep learning.

Text Mining Approaches for Dependent Bug Report Assembly and Severity Prediction

In general, most existing bug report studies focus only on solving a single specific issue. Considering of multiple issues at one is required for a more complete and comprehensive process of bug fixing. We took up this challenge and proposed a method to analyze two issues of bug reports based on text mining techniques. Firstly, dependent bug reports are assembled into an individual cluster and then the bug reports in each cluster are analyzed for their severity. The method of dependent bug report assembly is experimented with threshold-based similarity analysis. Cosine similarity and BM25 are compared with term frequency (tf) weighting to obtain the most appropriate method. Meanwhile, four classification algorithms namely Random Forest (RF), Support Vector Machines (SVM) with the RBF kernel function, Multinomial Naïve Bayes (MNB), and k-Nearest Neighbor (k-NN) are utilized to model the bug severity predictor with four term weighting schemes, i.e., tf, term frequency-inverse document frequency (tf-idf), term frequency-inverse class frequency (tf-icf), and term frequency-inverse gravity moment (tf-igm). After the experimentation process, BM25 was found to be the most appropriate for dependent bug report assemblage, while for severity prediction using tf-icf weighting on the RF method yielded the best performance value.

Extractive Multi-document Text Summarization Leveraging Hybrid Semantic Similarity Measures

Because of the massive amount of textual information accessible today, automated extraction text summarization is one of the most extensively used ways to organize the information. The summarization mechanisms help to extract the important topics of data from a given set of documents. Extractive summarization is one method for providing a representative summary of a text by choosing the most pertinent sentences from the original text. Extractive multi-document text summarization systems' primary goal is to decrease the quantity of textual information in a document collection by concentrating on the most crucial subjects and removing irrelevant material. In the previous research, there are several methods such as term-weighting schemes and similarity metrics used for constructing an automated summary system. There are few studies that look at the performance of combining various Semantic similarity and word weighting techniques in automatic text summarization. We evaluated numerous semantic similarity metrics in extractive multi-document text summarization in this research. In the extractive multi-document text summarization discussed in this research, we looked at numerous semantic similarity metrics. ROUGE metrics have been used to evaluate the model performance in experiments using DUC datasets. Even more, the combination formed by different semantic similarity measures obtained the highest results in comparison with the other models.

A new document representation based on global policy for supervised term weighting schemes in text categorization.

There are two main factors involved in documents classification, document representation method and classification algorithm. In this study, we focus on document representation method and demonstrate that the choice of representation methods has impacts on quality of classification results. We propose a document representation strategy for supervised text classification named document representation based on global policy (DRGP), which can obtain an appropriate document representation according to the distribution of terms. The main idea of DRGP is to construct the optimization function through the importance of terms to different categories. In the experiments, we investigate the effects of DRGP on the 20 Newsgroups, Reuters21578 datasets, and using the SVM as classifier. The results show that the DRGP outperforms other text representation strategy schemes, such as Document Max, Document Two Max and global policy.

Parallel dynamic topic modeling via evolving topic adjustment and term weighting scheme

The parallel Hierarchical Dirichlet Process (pHDP) is an efficient topic model which explores the equivalence of the generation process between Hierarchical Dirichlet Process (HDP) and Gamma-Gamma-Poisson Process (G2PP), in order to achieve parallelism at the topic level. Unfortunately, pHDP loses the non-parametric feature of HDP, i.e., the number of topics in pHDP is predetermined and fixed. Furthermore, under the bootstrap structure of pHDP, the topic-indiscriminate words are of high probabilities to be assigned to different topics, resulting in poor qualities of the extracted topics. To achieve parallelism without sacrificing the non-parametric feature of HDP, in addition to improve the quality of extracted topics, we propose a parallel dynamic topic model by developing an adjustment mechanism of evolving topics and reducing the sampling probabilities of topic-indiscriminate words. Both supervised and unsupervised experiments on benchmark datasets show the competitive performance of our model.

Revealing traces of depression through personal statements analysis in social media

Depression is a common and very important health issue with serious effects in the daily life of people. Recently, several researchers have explored the analysis of user-generated data in social media to detect and diagnose signs of this mental disorder in individuals. In this regard, we tackled the depression detection task in social media considering the idea that terms located in phrases exposing personal statements (i.e., phrases characterized by the use of singular first person pronouns) have a special value for revealing signs of depression. First, we assessed the value of the personal statements for depression detection in social media. Second, we adapted an automatic approach that emphasizes the personal statements by means of a feature selection method and a term weighting scheme. Finally, we addressed the task in hand as an early detection problem, where the aim is to detect traces of depression with as much anticipation as possible. For evaluating these ideas, benchmark Reddit data for depression detection was used. The obtained results indicate that the personal statements have high relevance for revealing traces of depression. Furthermore, the results on early scenarios demonstrated that the proposed approach achieves high competitiveness compared with state-of-the-art methods, while maintaining its simplicity and interpretability.