Abstract
The paper deals with the detection of fuzzy duplicates of documents in large amounts of information with low computational costs. The existing methods give either low search completeness at low computational costs, or acceptable completeness at very large computational costs. It is proposed to use combined method of detecting fuzzy duplicates. At the beginning of the whole set of documents with the help of signatures similar texts are searched and then, using context analysis methods, a detailed comparison of the texts found in this way is carried out. The method first performs an approximate search for similar documents using description words signature with a small match threshold. A detailed search for matches in previously found documents is performed using the shingles method.
Published Version
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call