Suffix Stripping Research Articles

Evaluating the Effect of Preprocessing Tools for Marathi Text Retrieval

The dramatic growth of the e-content available on the Internet in non-English languages facilitates the researchers to develop tools and techniques for automated processing of these languages. Retrieving meaningful information from this massive data is a challenging task, hence information retrieval of non-English languages is gaining more focus since last decade. The use of pre-processing tools like: stemmers, stop-word removal, lemmatizers, etc. has proven highly effective for the task of Information Retrieval for many languages like: English, Arabic, Hindi, etc. The goal of this work is to propose a simple stemmer for Marathi language using suffix stripping mechanism and evaluates the impact of it along with stop-word removal for Marathi text retrieval process. The result shows that significant improvement is obtained in the terms of precision, r-precision, precision@10, and recall due to the use of proposed suffix stripper and stop-word removal tool for Marathi text retrieval.

RBIPA: An Algorithm for Iterative Stemming of Tamil Language Texts

RBIPA: An Algorithm for Iterative Stemming of Tamil Language Texts

Noun Identification for Tamil Language using Morphophonemic Rules

In any language, words are considered as the basic or the smallest element with a distinctive meaning. Words can be categorized into several types, depending upon their use and functions. Basically how a word changes its form to express itself in grammatical notation defines its type. The process of categorizing a word to a particular type depending upon its grammatical notation is termed as Part of Speech tagging. In this paper, an attempt has been made to identify part of speech tagging for words in Tamil language, particular to noun inflections. An algorithm has been proposed for Noun Identification for Tamil Language using Morphophonemic Rules (NIMR). A Rule based suffix stripping approach has been adopted for this implementation. The approach proposed here identifies the root word by applying various morphophonemic rules particular to suffixes. It removes the various inflections based on the set of grammatical rules available for Tamil Language and tags the word identified as a Noun. It is proposed to explore the traditional way of categorizing words in Tamil language, avoiding the influence of English grammars.

Suffix Stripping Problem as an Optimization Problem

AbstractStemming or suffix stripping is the problem of removing suffixes from words to get the root word. Word endings can be removed by developing stripping rules dependent on the morphological knowledge of a specific language; obviously such approach cannot flourish in multilingual environment. Statistical approaches survive in multilingual environment but they require significant amount of computing. We define stemming as an

Stemming resource-poor Indian languages

Stemming is a basic method for morphological normalization of natural language texts. In this study, we focus on the problem of stemming several resource-poor languages from Eastern India, viz., Assamese, Bengali, Bishnupriya Manipuri and Bodo. While Assamese, Bengali and Bishnupriya Manipuri are Indo-Aryan, Bodo is a Tibeto-Burman language. We design a rule-based approach to remove suffixes from words. To reduce over-stemming and under-stemming errors, we introduce a dictionary of frequent words. We observe that, for these languages a dominant amount of suffixes are single letters creating problems during suffix stripping. As a result, we introduce an HMM-based hybrid approach to classify the mis-matched last character. For each word, the stem is extracted by calculating the most probable path in four HMM states. At each step we measure the stemming accuracy for each language. We obtain 94% accuracy for Assamese and Bengali and 87%, and 82% for Bishnupriya Manipuri and Bodo, respectively, using the hybrid approach. We compare our work with Morfessor [Creutz and Lagus 2005]. As of now, there is no reported work on stemming for Bishnupriya Manipuri and Bodo. Our results on Assamese and Bengali show significant improvement over prior published work [Sarkar and Bandyopadhyay 2008; Sharma et al. 2002, 2003].

A simple algorithm for the problem of suffix stripping

Suffix stripping is a problem of removing morphological suffixes from a word to get the stem. We present suffix stripping as an unconstrained optimization problem. Free from linguistic or morphological knowledge, a simple algorithm is being developed. Superiority of the algorithm over an established technique for English language is being demonstrated.

Morphological Analysis for Manipuri Nominal Category Words with Finite State Techniques

paper presents the design and the implementation of a morphological analyzer for Manipuri nominal category words. A method for the analysis of nominal category Manipuri words with a suffix stripping approach in a right to left direction without using any lexicon has been proposed. Manipuri being an agglutinative language and its rule-based nature while morpheme concatenation allows the morphotactics of the different available word forms to be modeled with finite state machines (FSMs). Also the very feature of the word classes which possess the characteristics that they can only be attached with affixes meant for that class only make it possible to analyze a nominal word without a lexicon. This paper discusses the morphological features of Manipuri nominal category, identifying the affixes for this class, and the steps of the new methodology to develop the FSM for nominal category to represent the morphotactics of the language, converting the FSM from non-deterministic finite automata (NFA) to deterministic finite automata (DFA) and thereby cooperating the analysis.

Tag Ontology Automatic Building for Semantic Searching of Services: a Case Study on Mashup Services

The explosion of services like web services, APIs, Mashups, etc., makes how to find the right one you need a tough problem. Tags, as a kind of metadata have been widely used to annotate services. In this paper, we propose to use an ontology automatically built from tags to improve the performance of service searching. We use the famous Mashup directory, Programmable.com, to illustrate our approach: First, all metadata especially tags of mashups and APIs at Programmable.com are crawled and preprocessed by the suffix stripping algorithm to lower the noise. Second, a Mashup-Mashup Network (MMN) is constructed to represent the Mashups and their relationships. And the community detection technique in complex network theory is introduced to mine the community structures (the potential domains of these Mashups) in MMN. Based on the Mashup communities, we further group the tags into corresponding domains. Finally the tag ontology is built with reference to WordNet, and embedded in our own developed s oftware s ervice r egistry and r epository (S2R2), to improve the performance of service searching. Comprehensive experiments are also conduct to testify the effectiveness of the proposed approach.

Design and Development of a Stemmer for Punjabi

Stemming is the process of removing the affixes from inflected words, without doing complete morphological analysis. A stemming Algorithm is a procedure to reduce all words with the same stem to a common form [20]. It is useful in many areas of computational linguistics and information-retrieval work. This technique is used by the various search engines to find the best solution for a problem. The algorithm is a basic building block for the stemmer. Stemmer is basically used in information retrieval system to improve the performance .The paper present a stemmer for Punjabi, which uses a brute force algorithm. We also use a suffix stripping technique in our paper. Similar techniques can be used to make stemmer for other languages such as Hindi, Bengali and Marathi. The result of stemmer is good and it can be effective in information retrieval system. This stemmer also reduces the problem of over-stemming and under-stemming.

An algorithm for suffix stripping

PurposeThe automatic removal of suffixes from words in English is of particular interest in the field of information retrieval. This work was originally published in Program in 1980 and is republished as part of a series of articles commemorating the 40th anniversary of the journal.Design/methodology/approachAn algorithm for suffix stripping is described, which has been implemented as a short, fast program in BCPL.FindingsAlthough simple, it performs slightly better than a much more elaborate system with which it has been compared. It effectively works by treating complex suffixes as compounds made up of simple suffixes, and removing the simple suffixes in a number of steps. In each step the removal of the suffix is made to depend upon the form of the remaining stem, which usually involves a measure of its syllable length.Originality/valueThe piece provides a useful historical document on information retrieval.

Neural correlates of morphological decomposition in a morphologically rich language: An fMRI study

By employing visual lexical decision and functional MRI, we studied the neural correlates of morphological decomposition in a highly inflected language (Finnish) where most inflected noun forms elicit a consistent processing cost during word recognition. This behavioral effect could reflect suffix stripping at the visual word form level and/or subsequent meaning integration at the semantic-syntactic level. The first alternative predicts increased activation for inflected vs. monomorphemic words in the left occipitotemporal cortex while the second alternative predicts left inferior frontal gyrus and/or left posterior temporal activation increases. The results show significant activation effects in the latter areas. This provides support for the second alternative, i.e., that the morphological processing cost stems from the semantic-syntactic level.

Automatic lemmatization of Persian words*

This study presents a rather novel method for suffix and prefix stripping of Persian words. The method presented is a language independent one and mostly relies on a specially arranged corpus composed of a list of roots, word-forms, prefixes, and suffixes which has been manually compiled. Applying an algorithm, the morphemes of each word including root, prefix, and suffixes are separated from each other specifying their place and order in the given word. The degree of accuracy and precision of this program is dependent on the representative characteristics and richness of the corpus. The present method for which Persian words have been tested represents an accuracy of 87.24% in Persian. Although the main application of this algorithm is in the field of information retrieval, it can be used in a machine translation system from Persian into any other language. In this case a stem dictionary for morphological analysis should be used.

Suffix stripping with modern Greek

The problem of morphological analysis may be considered as minor for languages like English compared to languages with a rich inflectional system like Greek. The plural inflection of the English noun which, apart few exceptions, is very simple (add ‐s) while in modern Greek there are 41 different inflectional suffixes. Storage of all the inflected forms of lexical items appearing in a text would cause an explosion in terms of storage space and retrieval time. There are different forms of the written Greek language: such forms include classical Greek, formal language and modern Greek. In this work we examine modern Greek texts.

Stemming of French words based on grammatical categories

Automatic indexing systems use suffix stripping algorithms to cluster various words derived from a common root under the same stem. Currently, removing affixes to either a context-free or context-sensitive operation, where the context refers to the remaining stem. In this article, we propose a suffixing algorithm which uses grammatical categories to enhance the stemming process. This approach supports the use of foreign languages. In our case, the language is French, and a morphological analysis is required for removing inflectional suffixes or morphosyntactic variants of a lemma. After this analysis, we implement a suffix stripping algorithm which uses a dictionary and the grammatical categories to remove derivational suffixes. Our approach always returns a linguistically correct lemma, but not necessarily the “right” one. Based on our tests, this solution is an attractive one, with a mean error rate of 16%. We finish by explaining why we cannot expect significantly better results with this approach. © 1993 John Wiley & Sons, Inc.

Comparing and combining the effectiveness of latent semantic indexing and the ordinary vector space model for information retrieval

A retrieval system was built to find individuals with appropriate expertise within a large research establishment on the basis of their authored documents. The expert-locating system uses a new method for automatic indexing and retrieval based on singular value decomposition, a matrix decomposition technique related to factor analysis. Organizational groups, represented by the documents they write, and the terms contained in these documents, are fit simultaneously into a 100-dimensional “semantic” space. User queries are positioned in the semantic space, and the most similar groups are returned to the user. Here we compared the standard vector-space model with this new technique and found that combining the two methods improved performance over either alone. We also examined the effects of various experimental variables on the system's retrieval accuracy. In particular, the effects of: term weighting functions in the semantic space construction and in query construction, suffix stripping, and using lexical units larger than a single word were studied.

An algorithm for suffix stripping

The automatic removal of suffixes from words in English is of particular interest in the field of information retrieval. An algorithm for suffix stripping is described, which has been implemented as a short, fast program in BCPL. Although simple, it performs slightly better than a much more elaborate system with which it has been compared. It effectively works by treating complex suffixes as compounds made up of simple suffixes, and removing the simple suffixes in a number of steps. In each step the removal of the suffix is made to depend upon the form of the remaining stem, which usually involves a measure of its syllable length.

KEDMA—Linguistic Tools for Retrieval Systems

In a full-text natural-language retrieval system, frequent need for automatic hngulst~c analysis arises, e.g for keyword expansion in a search process, content analysis, or automatic construction of concordances The avadablhty of sophisticated hngulstic tools, which is highly desirable for languages such as Enghsh, is quite imperative for, say, Semmc languages, whose complex morphological structure renders simple-minded and approximate soluuons such as suffix stripping totally useless. Sophisticated tools were designed and constructed via the fusion of grammatical analysis and grammatical synthesis, resulting in a set of global files which provide in some sense a complete grammatical and lexlcal description of the language These files induce a set of local files which adapt to the database at hand and permit flexible on-hne morphological analysis.