Normalized Term Frequency Research Articles

An automation query expansion strategy for information retrieval by using fuzzy based grasshopper optimization algorithm on medical datasets

SummaryDue to the technical words employed, which are primarily recognized by medical specialists, information retrieval in the medical area is sometimes described as sophisticated. Because of this, users frequently have trouble coming up with queries utilizing these medical phrases. However, this problem may be readily fixed by an information retrieval system that finds the pertinent terms that fit the user's query and automatically creates a ranking document using these keywords. To enhance the IR performance, the Automatic Query expansion method is applied by appending additional query terms for the medical domain. We propose a novel fuzzy‐based Grasshopper Optimization Algorithm (GOA) based on automatic query expansion. This work is mainly focused on filtering the most relevant augmented query by utilizing the synchronization score of IR evidence like normalized term frequency, inverse document frequency, and normalization of document length. The main aim of this work is to identify the medical terms that appropriately match the user's queries. The GOA algorithm ranks the terms based on relevance and then identifies the terms with the maximum synchronization value. The documents formed using the optimal expanded query are classified into three types, namely totally relevant, moderately relevant, and marginally relevant. Besides, the comparison of the proposed work is carried out for different performance metrics like Mean‐Average Precision, F‐measure, Precision‐recall, and Precision rank are evaluated and analyzed by using TREC‐COVID, TREC Genomics 2007, and MEDLARs medical datasets for the proposed and some of the state‐of‐art works. For a total of 60 queries, the proposed model offers an F1‐Score of 0.964, 0.959, and 0.968 for the MEDLARS, TREC Genomics, and TREC COVID19 datasets, respectively. The E1‐score and Mean Reciprocal Rate (MRR) of the proposed model is 0.8 and 0.9 when evaluated using the TREC COVID19 dataset. Performance analyses show that the proposed approach outperforms the other automatic keyword expansion approaches in the medical domain.

Truncated Models for Probabilistic Weighted Retrieval

Existing probabilistic retrieval models do not restrict the domain of the random variables that they deal with. In this article, we show that the upper bound of the normalized term frequency ( tf ) from the relevant documents is much smaller than the upper bound of the normalized tf from the whole collection. As a result, the existing models suffer from two major problems: (i) the domain mismatch causes data modeling error, (ii) since the outliers have very large magnitude and the retrieval models follow tf hypothesis, the combination of these two factors tends to overestimate the relevance score. In an attempt to address these problems, we propose novel weighted probabilistic models based on truncated distributions. We evaluate our models on a set of large document collections. Significant performance improvement over six existing probabilistic models is demonstrated.

Multinomial Naïve Bayes using similarity based conditional probability

The exponential growth of Internet through sharing text content necessitates the analysis to convert them into useful information. The research areas such as Web mining, Opinion mining and Text mining focus on studies namely content mining, statistical analysis, prediction, and classification. Mult inomial Naïve Bayes (MNB), the state of art of Bayesian classifier is the fastest and simplest text classifier. The objective of the proposed study is to enhance the classification by substituting the conditional probability of existing MNB with probability based frequency computation. A new combination that consists of Pointwise Mutual Information (PMI) and different normalized Term Frequency (TF) is used for computing the conditional probability. The new combinations provide weight to the words based on the information gain carried by the words related to the document that belongs to a class. The robustness of Similarity based Enhanced Conditional Probability MNB (SECP-MNB) is reflected in classification accuracy measurement.

Detection and Recognition of Atomic Evasions Against Network Intrusion Detection/Prevention Systems

Network evasions can bypass network intrusion detection/prevention systems to deliver exploits, attacks, or malware to victims without being detected. This paper presents a novel method for the detection and recognition of atomic network evasions by the classification of a transmission control protocol (TCP) stream's packet behavior. The syntax for the conversion of TCP streams to codeword streams is proposed to facilitate the extraction of statistical features while preserving the evasion behavior attributes of original network flows. We developed a feature extraction method of employing the normalized term frequencies of codewords to characterize intra and inter packet attribute patterns hidden in actual TCP streams. A TCP stream is then transformed to a fixed length numeric feature vector. Supervised multi-class classifiers are built on the extracted feature vectors to differentiate different types of evasions from normal streams. The quantitative evaluations on an evasion dataset consisting of normal network flows and eight types of atomic evasion flows demonstrated that the proposed approach achieved an encouraging performance with an accuracy of 98.95%.

Identification of Trending Topics Using Periodically Collected Twitter Data

Social media is an interactive personal tool to articulate an individual's cognizance. This project involves one such micro blogging platform, Twitter. Trends can simply be defined as the frequently mentioned topics throughout the stream of user activities. Mining twitter data for identifying trending topics provides an overview of the topics and issues that are currently popular within the online community. Therefore, the most effective and suitable methodology should be implemented to identify the short term high intensity discussion topic. The trigrams or higher order n-grams are used to determine the trending topic. Twitter Streaming API is used to collect data from the Twitter accounts using API keys and the formatted tweets are stored in a non SQL database. Subsequent steps include data cleansing followed by stemming. The processed data is subjected to trend prediction algorithms like DB Scan, Frequent Pattern Mining, Trees(fuzzy/inductive/decision), Soft frequent pattern mining and empirical statistics such as Frequency metric, TF-IDF, Normalized term frequency and Entropy based on the key parameters to identify the most trending event within a period of time. Thus, the trending topics can be detected with a reasonably close approximation to the expected outcome. This can be used in detecting and predicting events for an early warning system (or) prediction tools and also artificially intelligent services like web search system or recognition systems.

Prediction of cause of death from forensic autopsy reports using text classification techniques: A comparative study

ObjectivesAutomatic text classification techniques are useful for classifying plaintext medical documents. This study aims to automatically predict the cause of death from free text forensic autopsy reports by comparing various schemes for feature extraction, term weighing or feature value representation, text classification, and feature reduction. MethodsFor experiments, the autopsy reports belonging to eight different causes of death were collected, preprocessed and converted into 43 master feature vectors using various schemes for feature extraction, representation, and reduction. The six different text classification techniques were applied on these 43 master feature vectors to construct a classification model that can predict the cause of death. Finally, classification model performance was evaluated using four performance measures i.e. overall accuracy, macro precision, macro-F-measure, and macro recall. ResultsFrom experiments, it was found that that unigram features obtained the highest performance compared to bigram, trigram, and hybrid-gram features. Furthermore, in feature representation schemes, term frequency, and term frequency with inverse document frequency obtained similar and better results when compared with binary frequency, and normalized term frequency with inverse document frequency. Furthermore, the chi-square feature reduction approach outperformed Pearson correlation, and information gain approaches. Finally, in text classification algorithms, support vector machine classifier outperforms random forest, Naive Bayes, k-nearest neighbor, decision tree, and ensemble-voted classifier. ConclusionOur results and comparisons hold practical importance and serve as references for future works. Moreover, the comparison outputs will act as state-of-art techniques to compare future proposals with existing automated text classification techniques.

Exploring personalized searches using tag-based user profiles and resource profiles in folksonomy

With the increase in resource-sharing websites such as YouTube and Flickr, many shared resources have arisen on the Web. Personalized searches have become more important and challenging since users demand higher retrieval quality. To achieve this goal, personalized searches need to take users’ personalized profiles and information needs into consideration. Collaborative tagging (also known as folksonomy) systems allow users to annotate resources with their own tags, which provides a simple but powerful way for organizing, retrieving and sharing different types of social resources. In this article, we examine the limitations of previous tag-based personalized searches. To handle these limitations, we propose a new method to model user profiles and resource profiles in collaborative tagging systems. We use a normalized term frequency to indicate the preference degree of a user on a tag. A novel search method using such profiles of users and resources is proposed to facilitate the desired personalization in resource searches. In our framework, instead of the keyword matching or similarity measurement used in previous works, the relevance measurement between a resource and a user query (termed the query relevance) is treated as a fuzzy satisfaction problem of a user’s query requirements. We implement a prototype system called the Folksonomy-based Multimedia Retrieval System (FMRS). Experiments using the FMRS data set and the MovieLens data set show that our proposed method outperforms baseline methods.

Quantification of Portrayal Concepts using tf-idf Weighting

Term frequencies and inverse document frequencies have been successfully applied in determining weighting for document rankings. However these have been more successful in text mining and in extraction techniques used in the web. Concept mining has become increasingly popular in the research and application areas of Computer Science. This paper attempts to demonstrate the limited usage of term frequency and inverse document frequency for the application of weighting calculations for ranking documents that are based on concept quantifications. The case study considered for experiment in this paper, is based on concept terms of David Merrill’s First Principles of Instruction (FPI). Merrill’s FPI applies cognitive structures explicitly for analyzing instructional materials. Therefore it is justified that the terms categorized under each cognitive structure (or portrayal) of FPI can be taken as respective concept of that portrayal. As question papers are representative of cognitive structures in a more clear and logical way, four question papers on ‘C Language’ have been considered for the experimental study, that are detailed in this paper. Manual method has been adopted for the computation of quantities of portrayals in selected documents for the purpose of comparative study. As manual method is accurate, the values (results) are considered as benchmark values. These benchmark values are considered for comparing with normalized term frequencies that are derived (experimented) from automated extractions from the same selected documents. The study is however limited to four documents only. Conclusions are drawn from this experimental study, which will be of immense use to concept mining researchers as well as for instructional designers.

A Novel Method of Significant Words Identification in Text Summarization

Text summarization is a process that reduces the size of the text document and extracts significant sentences from a text document. We present a novel technique for text summarization. The originality of technique lies on exploiting local and global properties of words and identifying significant words. The local property of word can be considered as the sum of normalized term frequency multiplied by its weight and normalized number of sentences containing that word multiplied by its weight. If local score of a word is less than local score threshold, we remove that word. Global property can be thought of as maximum semantic similarity between a word and title words. Also we introduce an iterative algorithm to identify significant words. This algorithm converges to the fixed number of significant words after some iterations and the number of iterations strongly depends on the text document. We used a two-layered backpropagation neural network with three neurons in the hidden layer to calculate weights. The results show that this technique has better performance than MS-word 2007, baseline and Gistsumm summarizers.

An Intelligent System for Automated Binary Knowledge Document Classification and Content Analysis

Many companies rely on patent engineers to search patent documents and offer recommendations and advice to R&D engineers. Given the increasing number of patent documents filed each year, new means to effectively and efficiently identify and manage technology specific patent documents are required. This research applies a back-propagation artificial neural network (BPANN), a hierarchical ontology technique, and a normalized term frequency (NTF) method to develop an intelligent system for binary knowledge document classification and content analysis. The intelligent system minimizes inappropriate patent document classification and reduces the effort required to search and screen patents for analysis. Finally, this paper uses the design of light emitting diode (LED) lamps as a case study to illustrate and verify the efficiency of automated binary knowledge document classification and content analysis.

Improved biomedical document retrieval system with PubMed term statistics and expansions

Large biomedical abstract databases such as MEDLINE enable users to search for large bodies of biomedical knowledge quickly. In this study, we describe a new framework to improve the performance of MEDLINE document retrieval. We first analysed and built a normalized term frequency distributions for 1.8 million terms by sampling from 1,500,000 MEDLINE abstracts. Then, we developed a statistical model to identify significantly observed terms ('gists') in a document as additional document keywords to help improve document retrieval precisions. To improve document recalls, we integrated several biological ontologies that can expand user queries with semantically compatible terms. The framework was implemented in Oracle 10g.

On setting the hyper-parameters of term frequency normalization for information retrieval

The setting of the term frequency normalization hyper-parameter suffers from the query dependence and collection dependence problems, which remarkably hurt the robustness of the retrieval performance. Our study in this article investigates three term frequency normalization methods, namely normalization 2, BM25's normalization and the Dirichlet Priors normalization. We tackle the query dependence problem by modifying the query term weight using a Divergence From Randomness term weighting model, and tackle the collection dependence problem by measuring the correlation of the normalized term frequency with the document length. Our research hypotheses for the two problems, as well as an automatic hyper-parameter setting methodology, are extensively validated and evaluated on four Text REtrieval Conference (TREC) collections.

Topic Document Model Approach for Naive Bayes Text Classification

The multinomial naive Bayes model has been widely used for probabilistic text classification. However, the parameter estimation for this model sometimes generates inappropriate probabilities. In this paper, we propose a topic document model for the multinomial naive Bayes text classification, where the parameters are estimated from normalized term frequencies of each training document. Experiments are conducted on Reuters 21578 and 20 Newsgroup collections, and our proposed approach obtained a significant improvement in performance compared to the traditional multinomial naive Bayes.