Compactness — A useful feature for generating search index

Abstract

Generating meaningful or relevant keywords for information retrieval using Data Mining techniques is a highly relevant field. Term Discrimination Values (TDVs) are better measures compared to frequency or term weights to select the keywords. Terms with high TDVs will generate good keywords. Hamdouchi, P. Willet and Carolyn J Crouch have developed various algorithms to generate TDVs. In earlier days frequency or weighted frequency was used to compute TDVs. But these simple or weighted frequencies are not enough for retrieving relevant documents. Here we use some new features, connected with the distribution of terms within the document, called distributional features, to compute the TDVs. Distributional features such as First Appearance, Last Appearance, Compactness on number of parts, distance between first and last occurrence and on variance of positions of term occurrences etc. are pointers to the importance of the term in a document. Experiments have shown that combination of various features give much improved results than individual features in the case of Text Categorization. Through this work we also could prove that it is correct in the case of generating keywords. An additional overhead in storage and time is compensated by this efficient output. This work will add a narrow light towards text document search in education for both teaching and research.