Abstract
The paper deals with the detection of fuzzy duplicates of documents in large amounts of information with low computational costs. The existing methods give either low search completeness at low computational costs, or acceptable completeness at very large computational costs. It is proposed to use combined method of detecting fuzzy duplicates. At the beginning of the whole set of documents with the help of signatures similar texts are searched and then, using context analysis methods, a detailed comparison of the texts found in this way is carried out. The method first performs an approximate search for similar documents using description words signature with a small match threshold. A detailed search for matches in previously found documents is performed using the shingles method.
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: IOP Conference Series: Materials Science and Engineering
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
