Document Similarity Calculation Research Articles

Research on document similarity calculation and detection based on deep learning

In view of the complexity of traditional document similarity calculation methods and the problem of large error, a document similarity calculation and detection method based on deep learning is proposed. The effective subtree matching method is used to calculate the similarity of document feature sequences, and the frequency of feature items is obtained. In order to reduce the complexity of document similarity detection, a deep learning method is used to classify documents. The keywords in the document are extracted, and the similarity calculation and detection results are obtained by calculating the similarity between keywords. The experimental results show that the calculation error of the proposed method is low and the results are reliable, which fully shows that the method is effective and feasible.

Using Text's Terms and Syntactical Properties for Document Similarity

This paper reports on experiments performed to investigate the use of syntactical structures of sentences combined with sentences' terms for document similarity calculation. The document's sentences were first converted into ordered Part of Speech (POS) tags that were then fed into the Longest Common Subsequence (LCS) algorithm to determine the size and count of the LCSs found when comparing the document sentence by sentence. As a first stage, these syntactical features of the text were used as a structural representation of the document’s text. However, the produced strings of tags not only work as text representative but also provide for text size reduction. This improves the processing efficiency of comparing the document's representative strings using the LCS. A score is generated by computing an accumulative value based on the number of the LCSs found. In the second stage, documents that score well in the first stage are subjected to further comparison using the actual words of the sentences (content) in a sentence by sentence fashion. An overall final is generated as a measure of similarity using the common words (accumulated for the whole document) and the total number of LCSs from the first step. Experiments were done on two different corpora. Results obtained have showed the utility of the proposed procedure in calculating similarities between written documents. The overall discrimination power was maintained while the size of the documents was reduced using only a representative of the document based on the tagged string.

Document Similarity Search Algorithm Based On Hierarchy Model

Searching for similar documents from huge amounts of documents is an important and time consuming problem. Although the numerous precise models have been developed for the task, the traditional search algorithms are unable to meet the needs of users for quick search. Herein, a new document similarity calculation and search method with high efficiency is proposed. The calculation of the similarity is based on the total probability model and the efficient search is achieved via level n nodes and paths of citation graph. A special approach from the branch and bound limits the search scope and provide decision algorithm. With the increase in the number of documents, the efficiency of the proposed algorithm is dramatically promoted.

GPU 병렬성을 이용한 문서 유사도 계산 성능 개선

In the information retrieval systems like vector model implementation and document clustering, document similarity calculation takes a great part on the overall performance of the system. In this paper, GPU parallelism has been explored to enhance the processing speed of document similarity calculation in a CUDA framework. The proposed method increased the similarity calculation speed almost 15 times better compared to the typical CPU-based framework. It is 5.2 and 3.4 times better than the methods by using CUBLAS and Thrust, respectively.

Cross-lingual Named Entity Recognition

Named Entity Recognition and Classification (NERC) is a known and well-explored text analysis application that has been applied to various languages. We are presenting an automatic, highly multilingual news analysis system that fully integrates NERC for locations, persons and organisations with document clustering, multi-label categorisation, name attribute extraction, name variant merging and the calculation of social networks. The proposed application goes beyond the state-of-the-art by automatically merging the information found in news written in ten different languages, and by using the aggregated name information to automatically link related news documents across languages for all 45 language pair combinations. While state-of-the-art approaches for cross-lingual name variant merging and document similarity calculation require bilingual resources, the methods proposed here are mostly language-independent and require a minimal amount of monolingual language-specific effort. The development of resources for additional languages is therefore kept to a minimum and new languages can be plugged into the system effortlessly. The presented online news analysis application is fully functional and has, at the end of the year 2006, reached average usage statistics of 600,000 hits per day.

Efficient phrase-based document indexing for Web document clustering

Document clustering techniques mostly rely on single term analysis of the document data set, such as the vector space model. To achieve more accurate document clustering, more informative features including phrases and their weights are particularly important in such scenarios. Document clustering is particularly useful in many applications such as automatic categorization of documents, grouping search engine results, building a taxonomy of documents, and others. This article presents two key parts of successful document clustering. The first part is a novel phrase-based document index model, the document index graph, which allows for incremental construction of a phrase-based index of the document set with an emphasis on efficiency, rather than relying on single-term indexes only. It provides efficient phrase matching that is used to judge the similarity between documents. The model is flexible in that it could revert to a compact representation of the vector space model if we choose not to index phrases. The second part is an incremental document clustering algorithm based on maximizing the tightness of clusters by carefully watching the pair-wise document similarity distribution inside clusters. The combination of these two components creates an underlying model for robust and accurate document similarity calculation that leads to much improved results in Web document clustering over traditional methods.