The problem of semantic contradictions in large volumes of data

Abstract

The problem of eliminating the redundancy of semantically close textual information based on latent semantic analysis and one of the fuzzy inference algorithms is considered. A description of latent semantic analysis as a method of detecting the semantic proximity of documents is given. A variant of fuzzy inference rules for solving the task of eliminating the redundancy of semantically close information is formulated. It is proposed to evaluate the degree of impact of the contradiction elimination module on the operational efficiency of information systems. In order to assess the efficiency of the functioning of such systems, two approaches to the implementation of the function of eliminating contradictions are proposed: 1. Initially, the request is executed in its pure form without the participation of the contradiction elimination module. If as a result of the query we receive no more than one fact, then the result is passed to the upper level. When receiving more than one fact, there is a contradiction, the result is transferred to the contradiction elimination module. 2. All obtained results are transferred immediately to the contradiction elimination module, regardless of the number of obtained facts. It is obvious that the application of different approaches will affect the operational efficiency of the information system in different ways. In the first approach, the degree of influence of the heating module will depend on the number of contradictions. The article proposes to evaluate the degree of influence of the module of elimination of contradictions on the efficiency of the functioning of information systems. Redundancy in large volumes of data based on latent semantic analysis and fuzzy inference can, in our opinion, significantly reduce the amount of data stored.