Abstract
A hybrid method for the detection of near duplicates in tables is proposed. This method allows the identification of similarities between text and numeric data of tables separately, and then it generalized the results obtained. For the text data, sequences of words are formed in the canonized form, from which, based on the method of locality-sensitive hashing, the bit sequences are constructed. A similarity between data in this case is determined by the Hamming distance at the assigned threshold value. The identification of similarities between numeric data in tables is implemented based on the method of the nearest neighbours with assigned metric distances. The method makes it possible to identify near duplicates, present in data in the input table, relative to a set of tables, which are selected from the scientific publications and dissertations and theses papers. It should be noted that the method is designed for finding near duplicates in tables that contain only text and numeric data. In the case of availability in the content of examined tables of pictures and formulas, these objects are examined separately by using specific methods. The method proposed might be implemented in the systems that are intended for running intelligent analysis of information represented by text and tables to identify similarities and detect near-duplicates, in particular, antiplagiarism-systems.
Highlights
A table is understood as an arrangement of various types of data in rows and columns, or in the form of a more complex structure
The problem of finding near duplicates in numerical data is related to the identification of similarities in the time series based on the comparison to the sample using the method of nearest neighbors with assigned metric [13, 14]
As a result of the research, we described and formalized a hybrid method for the detection of near-duplicates based on the method of locality-sensitive hashing and the nearest neighbor method
Summary
A table is understood as an arrangement of various types of data in rows and columns, or in the form of a more complex structure. The usual term “column” is often interpreted as field, attribute or property that has a specified title or a name. This name may a priori consist of a word or a sequence of words, be presented by numeric values, a formula (formulas), or a date. If the original table contains results of numerical experiment, in the borrowing these numeric data may be deliberately changed. All this considerably complicates comparing tables for the identification of near-duplicates. The method proposed in the article might be used for the development of a module of software package to detect near-duplicates in thesis and diploma papers, as well as in the scientific publications
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: Eastern-European Journal of Enterprise Technologies
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
