DETERMINING THE SEMANTIC PROXIMITY OF NEWS MESSAGES BASED ON TITLES ANALYSIS

Abstract

The paper is devoted to the peculiarities of using data from unstructured sources, such as social networks, online media, etc. for the development of management decisions. The issue of analyzing such sources in the process of developing and making managerial decisions in the socio-economic sphere is considered. The difficulties of processing unstructured data are described, such as the impossibility of fully automatic evaluation of the semantics of data, the presence of a large amount of duplicate information. Various approaches to determining ratings and various metrics of sources and messages are described. The problem of identifying duplicate messages is considered using the example of online media both by full texts and by titles. Under the duplicate of a news message in this context refers a repetition of a significant amount of information from it in another article. If the text of a news message (article) is not repeated verbatim, it is impossible to determine the degree of duplication of the meaning of the article without involving human expertise. There are various metrics for assessing the similarity (semantic proximity) of textual information that can help in this matter. Some of them are described in the article. An adaptation of the Word Mover Distance method for the Russian language is proposed and the Word2Vec model is trained for its use. A hybrid approach to identifying and eliminating duplicate messages as part of the preprocessing of unstructured data in the context of managerial decision-making is proposed. According to the results of the experiments, depending on the chosen method, it was possible to automatically determine based on the publication time and the analysis of titles from 43 to 74 % duplicates.