Similarity Search Algorithm Research Articles

An experimental evaluation of simrank-based similarity search algorithms

Given a graph, SimRank is one of the most popular measures of the similarity between two vertices. We focus on efficiently calculating SimRank, which has been studied intensively over the last decade. This has led to many algorithms that efficiently calculate or approximate SimRank being proposed by researchers. Despite these abundant research efforts, there is no systematic comparison of these algorithms. In this paper, we conduct a study to compare these algorithms to understand their pros and cons. We first introduce a taxonomy for different algorithms that calculate SimRank and classify each algorithm into one of the following three classes, namely, iterative-, non-iterative-, and random walk-based method. We implement ten algorithms published from 2002 to 2015, and compare them using synthetic and real-world graphs. To ensure the fairness of our study, our implementations use the same data structure and execution framework, and we try our best to optimize each of these algorithms. Our study reveals that none of these algorithms dominates the others: algorithms based on iterative method often have higher accuracy while algorithms based on random walk can be more scalable. One noniterative algorithm has good effectiveness and efficiency on graphs with medium size. Thus, depending on the requirements of different applications, the optimal choice of algorithms differs. This paper provides an empirical guideline for making such choices.

Research and Implementation of Finding Duplicate Science Project Based on Dimension Filtering of Q-gram Index

Because of the huge number and the dimensions sparseness of the scientific research text set, a new similarity search algorithm for the scientific research text set is proposed, which is based on the weights of the q-gram. The algorithm can greatly reduce the number of dimensions, and then quickly find the similar text. Experiments show that the time consumption and the space consumption are decreased largely for the huge text set, and it has a high accuracy rate.

Aggressive pruning strategy for time series retrieval using a multi‐resolution representation based on vector quantization coupled with discrete wavelet transform

AbstractTime series representation methods are widely used to handle time series data by projecting them onto low‐dimensional spaces where queries are processed. Multi‐resolution representation methods speed up the similarity search process by using pre‐computed distances, which are calculated and stored at the indexing stage and then used at the query stage, together with filters in the form of exclusion conditions. In this paper, we present a new multi‐resolution representation method that combines the Haar wavelet‐based multi‐resolution method with vector quantization to maximize the pruning power of the similarity search algorithm. The new method is validated through extensive experiments on different datasets from several time series repositories. The results obtained prove the efficiency of the new method.

Forward backward similarity search in knowledge networks

Similarity search is a fundamental problem in social and knowledge networks like GitHub, DBLP, Wikipedia, etc. Existing network similarity measures are limited because they only consider similarity from the perspective of the query node. However, due to the complicated topology of real-world networks, ignoring the preferences of target nodes often results in odd or unintuitive performance. In this work, we propose a dual perspective similarity metric called Forward Backward Similarity (FBS) that efficiently computes topological similarity from the perspective of both the query node and the perspective of candidate nodes. The effectiveness of our method is evaluated by traditional quantitative ranking metrics and large-scale human judgement on four large real world networks. The proposed method matches human preference and outperforms other similarity search algorithms on community overlap and link prediction. Finally, we demonstrate top-5 rankings for five famous researchers on an academic collaboration network to illustrate how our approach captures semantics more intuitively than other approaches.

Real-Valued Embeddings and Sketches for Fast Distance and Similarity Estimation

This survey article considers methods and algorithms for fast estimation of data distance/similarity measures from formed real-valued vectors of small dimension. The methods do not use learning and mainly use random projection and sampling. Initial data are mainly high-dimensional vectors with different measures of distance (Euclidean, Manhattan, statistical, etc.) and similarity (dot product, etc.). Vector representations of non-vector data are also considered. The resultant vectors can also be used in similarity search algorithms, machine learning, etc.

TSTMP: target selection for structural genomics of human transmembrane proteins.

The TSTMP database is designed to help the target selection of human transmembrane proteins for structural genomics projects and structure modeling studies. Currently, there are only 60 known 3D structures among the polytopic human transmembrane proteins and about a further 600 could be modeled using existing structures. Although there are a great number of human transmembrane protein structures left to be determined, surprisingly only a small fraction of these proteins have ‘selected’ (or above) status according to the current version the TargetDB/TargetTrack database. This figure is even worse regarding those transmembrane proteins that would contribute the most to the structural coverage of the human transmembrane proteome. The database was built by sorting out proteins from the human transmembrane proteome with known structure and searching for suitable model structures for the remaining proteins by combining the results of a state-of-the-art transmembrane specific fold recognition algorithm and a sequence similarity search algorithm. Proteins were searched for homologues among the human transmembrane proteins in order to select targets whose successful structure determination would lead to the best structural coverage of the human transmembrane proteome. The pipeline constructed for creating the TSTMP database guarantees to keep the database up-to-date. The database is available at http://tstmp.enzim.ttk.mta.hu.

Chemical Structure Similarity Search for Ligand-based Virtual Screening: Methods and Computational Resources.

For many years the assumption that "Chemical compounds with similar structures may have similar activities" has been a foundation for lead identification. The similarity can be computed based upon topological, steric, electronic, and/or physical properties. The chemical structure similarity search differs from the chemical substructure search in that the former requires assessment of the properties of each compound and thus no filter can be applied for skipping structures before they are assessed to accelerate the computation. The latter can be accelerated by pre-screening compounds and omitting those that miss one (or more) specified fragments from the query. Moreover, three-dimensional similarity search requires superimposing many conformation pairs for each compound in the library. This makes 3-D similarity search algorithms time-consuming, and in general requires high performance computing (HPC) resources. This review will summarize recent progress in the techniques for HPC-supported two and three-dimensional chemical structure similarity search algorithms, and their applications in ligand-based virtual screening.

An efficiency study of a pivot-based algorithm for similarity search on a heterogeneous platform

Graphics processing units have definitely consolidated a position of privilege in the acceleration of general purpose applications. Search algorithms in large databases are a clear example of applications that benefit from computing platforms based on these devices. To obtain an efficient implementation of a given code using these platforms, it is very important to take into account their features. However, the characteristics of the application and certain overheads still introduced by these platforms, make it not always advisable to use these devices to obtain significant time reductions. In this paper, we show how different properties of current graphics processing units are exploited for improving a version of the general metric structure similarity search algorithm introduced by the authors, and compare it with a multithreaded version of the same algorithm using conventional processors. The analysis of the results provides us relevant data to determine the most appropriate computing platform.

A novel locality-sensitive hashing algorithm for similarity searches on large-scale hyperspectral data

ABSTRACTSimilarity search is a fundamental process in many hyperspectral remote sensing applications. In this article, we investigated the locality-sensitive hashing (LSH) algorithm for approximate similarity search on hyperspectral remote sensing data and proposed a new method that uses a regulated random hyperplane projection hash function family to index data and an adjacent graph-probing method for similarity queries. This method improves the performance of LSH over non-uniformly distributed hyperspectral data sets. Comparative experiments with three benchmark hyperspectral data sets showed that the proposed method is at least two times faster than the basic LSH and two other improved methods whilst achieving the same search accuracy. Moreover, results of the proposed method are proven equivalent to the accurate similarity search for real applications.

Improving business process retrieval using categorization and multimodal search

Enterprises use repositories of Business Processes to standardize and adapt their operations in order to reuse them for new functional requirements. However, a disorganized growth of these repositories have hampered the search of Business Processes which is fundamental for reusing them. In this paper an approach for organizing and searching Business Processes is proposed, which is composed of two phases: First, an automatic and semantic categorization phase to classify Business Processes based on their functionality, and second a multimodal search phase in order to rank Business Processes based on structural and textual features. The proposed approach was tested in an evaluation over a closed repository collaboratively built by 20 expert evaluators. Initially, evaluators were asked to rate categories assigned by our approach to each Business Process in order to assess our results against the user perspective. Later, evaluators were asked to compare six queries against the repository in order to obtain a set of relevant Business Processes for each query. With these results, precision, recall and F-measure were calculated to evaluate the relevance and ranking concordance of the proposed approach against state-of-the-art algorithms for Business Process similarity search. Additionally, we applied the Friedman and the Wilcoxon signed rank tests over the results obtained for each query over precision and F-measure in order to evaluate the statistically significance of these results. The results obtained demonstrated the effectiveness of the proposed approach for categorizing and retrieving Business Processes.

SW#db: GPU-Accelerated Exact Sequence Similarity Database Search.

In recent years we have witnessed a growth in sequencing yield, the number of samples sequenced, and as a result–the growth of publicly maintained sequence databases. The increase of data present all around has put high requirements on protein similarity search algorithms with two ever-opposite goals: how to keep the running times acceptable while maintaining a high-enough level of sensitivity. The most time consuming step of similarity search are the local alignments between query and database sequences. This step is usually performed using exact local alignment algorithms such as Smith-Waterman. Due to its quadratic time complexity, alignments of a query to the whole database are usually too slow. Therefore, the majority of the protein similarity search methods prior to doing the exact local alignment apply heuristics to reduce the number of possible candidate sequences in the database. However, there is still a need for the alignment of a query sequence to a reduced database. In this paper we present the SW#db tool and a library for fast exact similarity search. Although its running times, as a standalone tool, are comparable to the running times of BLAST, it is primarily intended to be used for exact local alignment phase in which the database of sequences has already been reduced. It uses both GPU and CPU parallelization and was 4–5 times faster than SSEARCH, 6–25 times faster than CUDASW++ and more than 20 times faster than SSW at the time of writing, using multiple queries on Swiss-prot and Uniref90 databases

Effective and Efficient Algorithms for Flexible Aggregate Similarity Search in High Dimensional Spaces

Numerous applications in different fields, such as spatial databases, multimedia databases, data mining, and recommender systems, may benefit from efficient and effective aggregate similarity search, also known as aggregate nearest neighbor (AggNN) search. Given a group of query objects $Q$ , the goal of AggNN is to retrieve the $k$ most similar objects from the database, where the underlying similarity measure is defined as an aggregation (usually sum or max ) of the distances between the retrieved objects and every query object in $Q$ . Recently, the problem was generalized so as to retrieve the $k$ objects which are most similar to a fixed proportion of the elements of $Q$ . This variant of aggregate similarity search is referred to as “flexible AggNN”, or FANN. In this work, we propose two approximation algorithms, one for the sum variant of FANN, and the other for the max variant. Extensive experiments are provided showing that, relative to state-of-the-art approaches (both exact and approximate), our algorithms produce query results with good accuracy, while at the same time being very efficient.

String similarity search and join: a survey

String similarity search and join are two important operations in data cleaning and integration, which extend traditional exact search and exact join operations in databases by tolerating the errors and inconsistencies in the data. They have many real-world applications, such as spell checking, duplicate detection, entity resolution, and webpage clustering. Although these two problems have been extensively studied in the recent decade, there is no thorough survey. In this paper, we present a comprehensive survey on string similarity search and join. We first give the problem definitions and introduce widely-used similarity functions to quantify the similarity. We then present an extensive set of algorithms for string similarity search and join. We also discuss their variants, including approximate entity extraction, type-ahead search, and approximate substring matching. Finally, we provide some open datasets and summarize some research challenges and open problems.

Similarity identification algorithm of hard-structured data based on semantic networks

The main disadvantage of any similarity search algorithm is its purpose. Existing algorithms are focused purely on the text as a continuous element of structural data and do not take into account the context of information that is presented in the text. This makes it impossible to use algorithms for text with a specific context. The only application of such algorithms is the texts that can be elements of a more complex object of comparison. The paper presents a similarity search algorithm of hard-structured data using semantic networks. The built semantic network takes into account all the features of the bibliographic description of the publication and is endowed with methods of comparison of its separate parts. Application of the algorithm for similarity search of bibliographic descriptions in the information-analytical system ScienceLP was investigated. The research results have confirmed the usefulness of the developed algorithm for effective relevant search. For the versatility of the software implementation of the algorithm, reflection-oriented programming approach was used. Such an approach allows to identify almost any object, no matter whether it is built-in or user data type. This allows the algorithm to be independent of the type of the compared object and its internal structure.

Ultrafast protein structure-based virtual screening with Panther.

Molecular docking is by far the most common method used in protein structure-based virtual screening. This paper presents Panther, a novel ultrafast multipurpose docking tool. In Panther, a simple shape-electrostatic model of the ligand-binding area of the protein is created by utilizing the protein crystal structure. The features of the possible ligands are then compared to the model by using a similarity search algorithm. On average, one ligand can be processed in a few minutes by using classical docking methods, whereas using Panther processing takes <1 s. The presented Panther protocol can be used in several applications, such as speeding up the early phases of drug discovery projects, reducing the number of failures in the clinical phase of the drug development process, and estimating the environmental toxicity of chemicals. Panther-code is available in our web pages (http://www.jyu.fi/panther) free of charge after registration.

PhenoMeter: A Metabolome Database Search Tool Using Statistical Similarity Matching of Metabolic Phenotypes for High-Confidence Detection of Functional Links.

This article describes PhenoMeter (PM), a new type of metabolomics database search that accepts metabolite response patterns as queries and searches the MetaPhen database of reference patterns for responses that are statistically significantly similar or inverse for the purposes of detecting functional links. To identify a similarity measure that would detect functional links as reliably as possible, we compared the performance of four statistics in correctly top-matching metabolic phenotypes of Arabidopsis thaliana metabolism mutants affected in different steps of the photorespiration metabolic pathway to reference phenotypes of mutants affected in the same enzymes by independent mutations. The best performing statistic, the PM score, was a function of both Pearson correlation and Fisher’s Exact Test of directional overlap. This statistic outperformed Pearson correlation, biweight midcorrelation and Fisher’s Exact Test used alone. To demonstrate general applicability, we show that the PM reliably retrieved the most closely functionally linked response in the database when queried with responses to a wide variety of environmental and genetic perturbations. Attempts to match metabolic phenotypes between independent studies were met with varying success and possible reasons for this are discussed. Overall, our results suggest that integration of pattern-based search tools into metabolomics databases will aid functional annotation of newly recorded metabolic phenotypes analogously to the way sequence similarity search algorithms have aided the functional annotation of genes and proteins. PM is freely available at MetabolomeExpress (https://www.metabolome-express.org/phenometer.php).

Similarity recognition using context-based pattern for cyber-society

To measure the similarity of contexts in smart devices, comparison is made of the user-defined contexts and another context which is defined in a server or in a network device that a user has (Segev and Toch IEEE Trans Serv Comput 2(3):210---222, 2009). While it processes to compare, if they find some users who have similarities with them, they surely may be interested in the users, because they know that they can share their information without the time wasting to search, finally getting what they want. However, according to the characteristics of the registered contexts, they are classified into two types, a rank-definitional context and a rank-undefined context. Also, the users usually want to use the two types to get quickly what they want at the same time. It means that another algorithm may be needed to get the similarity depending on the contexts, because the existing similarity search algorithms usually perform the similarity process without the contexts' characteristics analysis. They assume that all contexts have the same features when they process. Now, the existing methods that find the similarity usually have an accuracy problem. Low accuracy gives invisible services to users. This paper has suggested named context-based pattern measurement method including weight defines for higher accuracy. As a result, it would be able to get accuracy similarity by applying to the proposed algorithms about 69.072 % without weight and also 95.322 % accuracy in case it has a specific weight.

類似検索アルゴリズムを上手に利用する

類似検索アルゴリズムを上手に利用する

POI(Point Of Interest) 데이터 검색에서 문자열 유사도 측정 정확도 향상 기법

With the development of smart transportation, people are likely to find their paths by using navigation and map application. However, the existing retrieval system cannot output the correct retrieval result due to the inaccurate query. In order to remedy this problem, set-based POI search algorithm was proposed. Subsequently, additionally a method for measuring POI name similarity and POI search algorithm supporting classifying duplicate characters were proposed. These algorithms tried to compensate the insufficient part of the compensate set-based POI search algorithm. In this paper, accuracy improvement methods for measuring string similarity in POI data retrieval system are proposed. By formulization, similarity measurement scheme is systematized and generalized with the development of transportation. As a result, it improves the accuracy of the retrieval result. From the experimental results, we can observe that our accuracy improvement methods show better performance than the previous algorithms.

Languages of lossless seeds

Several algorithms for similarity search employ seeding techniques to quickly discard very dissimilar regions. In this paper, we study theoretical properties of lossless seeds, i.e., spaced seeds having full sensitivity. We prove that lossless seeds coincide with languages of certain sofic subshifts, hence they can be recognized by finite automata. Moreover, we show that these subshifts are fully given by the number of allowed errors k and the seed margin l. We also show that for a fixed k, optimal seeds must asymptotically satisfy l ~ m^(k/(k+1)).