Development of a method for determining the keywords in the slavic language texts based on the technology of web mining

Abstract

The authors accomplished the task of development of algorithmic support of processes of the content monitoring for solving the problem of defining the keywords of a Slavic language text based on Web Mining technology. Substantiation of peculiarities of its use for defining keywords and subject heading of the text content was considered. Web Mining technology allows us to take advantage of the text content monitoring method based on the Porter’s stemmer to solve the problem on determining the keywords. Stemming modification is based on the well-known classification of morpheme and word formation structure of derivatives of the Ukrainian language, revealing patterns of affixes combination, modeling the structural organization of verbs and suffixed nouns. Algorithms of morphonological modifications in the process of verb word changing and adjective word changing and word formation in the Ukrainian language were used. Decomposition of the method of determining keywords of the text content was performed. Its features include adaptation of morphological and syntactic analysis of lexical units to peculiarities of Ukrainian words/text structures. Algorithm support of its main structural components was developed. Its features include convolution and analysis of a nominal/verb group and construction of appropriate trees of analysis for each sentence, taking into account the features of their structures as elements of the Slavic language texts. The formal approach to the implementation of stemming of a Ukrainian language text was proposed. It is aimed at automatic detection of notional keywords of a Ukrainian text due to the proposed formal approach to implementation of stemming for the Ukrainian language content. Theoretically, the ways of enhancing efficiency of the keywords search, in particular their density in the text, were found. They are based on an analysis of not the words themselves (nouns, a set of nouns, adjectives with nouns, other parts of speech are ignored), but rather of word stems in Slavic language texts. The rules of stem separations in texts consider not only the isolation of inflexions, but also suffixes, as well as registering the letter alternation during declension of nouns and adjectives. Based on the developed software, we received the results of experimental testing of the proposed content monitoring method for defining keywords in Slavic language scientific texts of technical area based on the Web Mining technology. It was found that for the selected experimental base of 100 works, the best results according to density criterion are achieved by the method of article analysis without compulsory initial information and a list of literature. This is attained through training the system and by checking the refined blocked words and refined thematic dictionary. It was also discovered that for technical scientific texts of the experimental base, the best results are reached by the method of article analysis without beginning (title, authors, UDC, abstracts in two languages, author’s keywords in two languages, work place of authors) and without a list of literature with the check of specified blocked words and refined thematic dictionary – for it the average value of keywords density in the text reaches 0.34, which is by 81 % higher than the correspondent value of density of the original text, which makes 0.19. By numerous data of statistical analysis, it was proved that setting parameters of the system increases the number of defined keywords almost by 2 times without decreasing the indicator of accuracy and reliability. Testing of the proposed method for determining keywords from other categories of texts, such as scientific humanitarian, fiction, journalistic, require further experimental research.