Top-k Query Processing Research Articles

Waves: a fast multi-tier top-k query processing algorithm

In this paper, we present Waves, a novel document-at-a-time algorithm for fast computing of top-k query results in search systems. The Waves algorithm uses multi-tier indexes for processing queries. It performs successive tentative evaluations of results which we call waves. Each wave traverses the index, starting from a specific tier level i. Each wave i may insert only those documents that occur in that tier level into the answer. After processing a wave, the algorithm checks whether the answer achieved might be changed by successive waves or not. A new wave is started only if it has a chance of changing the top-k scores. We show through experiments that such lazy query processing strategy results in smaller query processing times when compared to previous approaches proposed in the literature. We present experiments to compare Waves’ performance to the state-of-the-art document-at-a-time query processing methods that preserve top-k results and show scenarios where the method can be a good alternative algorithm for computing top-k results.

An Indexing Method to Construct Unbalanced Layers for High-Dimensional Data in Mobile Environments

A top-k query processing is widely used in many applications and mobile environments. An index is used for efficient query processing and layer-based indexing methods are representative to perform the top-k query processing efficiently. However, the existing methods have a problem of high index building time for multidimensional and large data; thus, it is difficult to use them. In this paper, we proposed a new concept of constructing layer-based index, which is called unbalanced layer (UB-Layer). The existing methods construct a layer as a balanced layer with outermost data and wrap the rest of the input data. However, UB-Layer constructs a layer as an unbalanced layer that does not wrap the rest of the data. To construct UB-Layer, we fist divide the dimension of the input data into divided-dimensional data and compute the convex hull in each divided-dimensional data. And then, we combine divided-convex hull to build UB-Layer. We also propose UB-SelectAttribute algorithm for dividing the dimension with major attributes. We demonstrate the superiority of the proposed methods by the performance experiments.

Adaptive convex skyline: a threshold-based project partitioned layer-based index for efficient-processing top-k queries in entrepreneurship applications

Many entrepreneurship applications use data as the core concept of their business to better understand the needs of their customers. However, as the size of databases used by these entrepreneurship applications grows and as more users access data through various interactive interfaces, obtaining the result for a top-k query may take long time if the query matches millions of the tuples in the database. Traditionally, layer-based indexing methods are representative for processing top-k queries efficiently. These methods form tuples into a list of layers where the ith layer holds the tuples that can be the top-i answer. Layer-based indexing methods enable us to obtain top-k answers by accessing at most k layers. Most of these methods achieve high accuracy of query answer at the expense of enlarged index construction time. However, we can adjust between accuracy and index construction time to achieve an optimal performance. Thus in this paper, we propose a method, called the adaptive convex skyline (AdaptCS) for efficient-processing top-k queries in entrepreneurship applications. AdaptCS first prunes the data with a virtual threshold point and finds skyline points over the pruned data. Here, by adjusting virtual threshold we are able to achieve optimal performance. Then, AdaptCS divides the skyline into m subregions with projection partitioning method and constructs the convex hull m times for each subregion with virtual objects. Lastly, AdaptCS combines the objects obtained by computing the convex hull. The experimental results show that the proposed method outperforms the existing methods.

Efficient indexing structure to handle durable queries through web crawling

This paper studies efficient processing of durable top-k queries on historical time series databases. Durable top-k queries, obtained as an extension of snapshot top-k queries during a certain time period, play a key role in finding objects with durable quality and predicting the status of these objects for successive time intervals by updating the query interval at all timestamps. Web crawling and indexing are tremendously significant in recent times, especially in terms of achieving efficient durable top-k queries from vast quantum of web documents. Existing algorithms that have been employed throw up results that are less than applicable to analyzers. This paper chiefly focuses on web crawling and indexing query terms under their respective categories and updating rank changes at every time interval. Links are crawled using the modified depth-first search (MDFS) algorithm, accessed, and metadata such as the title, keywords, and descriptions extracted. To handle query indexing, novel indexing techniques are proposed to yield efficient results. This study is invaluable for analysts working on large data obtained as a result of crawling and indexing, effectively decreasing their workload.

A Three-phase Large Scale Skyline Service Selection Framework in Clouds

For the large scale services with high-dimensional QoS attributes and distributed environment, traditional service selection approaches are faced with unprecedented challenges in terms of efficiency and performance of QoS. To address these challenges, we propose a three-phase large scale Skyline service selection framework for service composition in clouds. This framework adopts distributed parallel Skyline computation with MapReduce to prune redundant candidate services, and employs parallel multiobjective optimization algorithm based on MapReduce to select Skyline services from the tremendous amount of Skyline services warehouse for composing single service into a set of more powerful Skyline composite services, then applies Top-k query processing technology or multiple attribute decision making support method to select k Skyline composite services from the set of Skyline composite services. Through theoretical analysis, the framework can efficiently solve the service selection problem with large scale services, high-dimensional QoS in cloud computing environment, and quickly generate better composite services with the global optimal QoS.

Finding Top-k Answers in Node Proximity Search using Distribution State Transition Graph

Considerable attention has been given to processing graph data in recent years. An efficient method for computing the node proximity is one of the most challenging problems for many applications such as recommendation systems and social networks. Regarding large-scale, mutable datasets and user queries, top-k query processing has gained significant interest. This paper presents a novel method to find top-k answers in a node proximity search based on the well-known measure, Personalized PageRank (PPR). First, we introduce a distribution state transition graph (DSTG) to depict iterative steps for solving the PPR equation. Second, we propose a weight distribution model of a DSTG to capture the states of intermediate PPR scores and their distribution. Using a DSTG, we can selectively follow and compare multiple random paths with different lengths to find the most promising nodes. Moreover, we prove that the results of our method are equivalent to the PPR results. Comparative performance studies using two real datasets clearly show that our method is practical and accurate.

Subscription-based data aggregation techniques for top-k monitoring queries

With the increase of data generation in distributed fashions such as peer-to-peer systems and sensor networks, top-k query processing which is a way to aggregate only a small set of data that sufficiently satisfies many users' preferences, becomes a substantial issue. When data are periodically updated in each epoch e.g., weather information, without any techniques, a naive solution is to aggregate all data and their updates to ensure the completeness of final answers, however, it is too costly in terms of data transfer especially for data aggregator nodes (intermediate nodes). In this paper, we propose a top-k monitoring query processing method in 2-tier distributed systems based on a publish-subscribe scheme. A set of top-k subscriptions specifying summary scope of users' interests is informed to aggregators to limit the number of transferred data records for each epoch. In addition, instead of issuing subscriptions of all queries, our method identifies a small set of minimal subscriptions as well as utilizes some adaptive heuristic rules to efficiently maintain those subscriptions resulting in lower communication overhead. Our experiments through both synthetic and real datasets show that our technique is efficient and outperforms other comparative reactive methods.

An efficient top-k query processing framework in mobile sensor networks

Mobile sensor networks consist of a number of sensor nodes which are capable of sensing, processing, communicating and moving. These mobile sensor nodes move around and explore their surrounding areas. Top-k queries are useful in many mobile sensor network applications. However, the mobility of sensor nodes incurs new challenges in addition to the problems of static sensor networks (i.e., resource constraints). Since mobile sensor nodes tend to move continuously, the network condition changes frequently and they consume considerably more energy than static sensor nodes. In this paper, we propose an efficient top-k query processing framework in a mobile sensor network environment called mSensor. To construct an efficient routing topology, we devise a mobility-aware routing method. Using the semantics of the top-k query, we develop a filter-based data collection method which can save the energy consumption and provide more accurate query results. We also devise a data compression method for disconnected sensor nodes to deal with the problem of limited memory space of sensor nodes. The performance of our proposed approach is extensively evaluated using synthetic data sets and real data sets. The results show the effectiveness of our approach.

Intelligent top k query answering using meta-data base

Several algorithms of the top-k retrieval problem have been introduced in recent years. Unfortunately, current top-k query processing techniques focus on Boolean queries, and cannot be applied to the large Data Bases (DB) seen the gigantic number of data. In this paper, we propose an intelligent approach for top-k flexible queries taking into account another degree of granularity in the process of the evaluation of the query. It consists of two steps: 1) Generate a Meta-DB formed by a set of clusters resulting of a preliminary fuzzy clustering on the data. This set represents a reduced view of the initial DB and permits to deduct the semantics of the initial DB, and 2) Generate automatically the top k results using this Meta-DB and a new operator called stratified operator. We prove that this approach is optimum sight that the evaluation of the query is not done on the set of starting data which are enormous but rather by using the Meta-DB. Moreover, this approach can be applied to the large DB and that it does not require modifying the SQL language.

Crowdsourcing for Top-K Query Processing over Uncertain Data

Querying uncertain data has become a prominent application due to the proliferation of user-generated content from social media and of data streams from sensors. When data ambiguity cannot be reduced algorithmically, crowdsourcing proves a viable approach, which consists of posting tasks to humans and harnessing their judgment for improving the confidence about data values or relationships. This paper tackles the problem of processing top- $K$ queries over uncertain data with the help of crowdsourcing for quickly converging to the realordering of relevant results. Several offline and online approaches for addressing questions to a crowd are defined and contrasted on both synthetic and real data sets, with the aim of minimizing the crowd interactions necessary to find the realordering of the result set.

Top-k Query Processing and Malicious Node Identification Based on Node Grouping in MANETs

In mobile ad hoc networks (MANETs), it is effective to retrieve data items using top-k query. However, accurate results may not be acquired in environments when malicious nodes are present. In this paper, we assume that malicious nodes attempt to replace necessary data items with unnecessary ones (we call these data replacement attacks), and propose methods for top-k query processing and malicious node identification based on node grouping in MANETs. In order to maintain the accuracy of the query result, nodes reply with k data items with the highest score along multiple routes, and the query-issuing node tries to detect attacks from the information attached to the reply messages. After detecting attacks, the query-issuing node tries to identify the malicious nodes through message exchanges with other nodes. When multiple malicious nodes are present, the query-issuing node may not be able to identify all malicious nodes at a single query. It is effective for a node to share information about the identified malicious nodes with other nodes. In our method, each node divides all nodes into groups by using the similarity of the information about the identified malicious nodes. Then, it identifies malicious nodes based on the information on the groups. We conduct simulation experiments by using a network simulator, QualNet5.2, to verify that our method achieves high accuracy of the query result and identifies malicious nodes.

Threshold-Based Distributed Continuous Top-k Query Processing for Minimizing Communication Overhead

Threshold-Based Distributed Continuous Top-k Query Processing for Minimizing Communication Overhead

Efficient and Secure Top-k Query Processing on Hybrid Sensed Data

The ubiquity of mobile devices equipped with various sensors has promoted the advent of a novel data sensing paradigm. Based on the traditional static sensing mode, the mobile sensing (sensor) nodes collaboratively collect data with the static sensor nodes. This large volume of hybrid sensed data is then sent to the storage nodes for flexible management and top-k query services. One crucial security issue is that the compromised storage node may falsify or drop some data during the query processing, which returns fake or incorrect result to the query users. In this paper, we propose an efficient and verifiable scheme (EVTopk) for secure top-k query processing on hybrid sensed data, which is suitable for the tiered hybrid sensing network where mobile nodes exist. The basic idea is to bind each data record, generated by static or mobile sensing nodes, with the corresponding location where it is sensed. Then some verification information is created sequentially, which is submitted along with the encrypted locations and hybrid sensed data for user’s verification. The security and efficiency of EVTopk are thoroughly analyzed in theory and evaluated in our experiments, respectively.

아웃소싱 암호화 데이터에 대한 효율적인 Top-k 질의 처리 알고리즘

ABSTRACT Recently top-k query processing has been extremely important along with the explosion of data produced by a variety of applications. Top-k queries return the best k results ordered by a user-provided monotone scoring function. As cloud computing service has been getting more popular than ever, a hot attention has been paid to cloud-based data outsourcing in which clients' data are stored and managed by the cloud. The cloud-based data outsourcing, though, exposes a critical secuity concern of sensitive data, resulting in the misuse of unauthorized users. Hence it is essential to encrypt sensitive data before outsourcing the data to the cloud. However, there has been little attention to efficient top-k processing on the encrypted cloud data. In this paper we propose a novel top-k processing algorithm that can efficiently process a large amount of encrypted data in the cloud. The main idea of the algorithm is to prune unpromising intermediate results at the early phase without decrypting the encrypted data by leveraging an order-preserving encrypted technique. Experiment results show that the proposed top-k processing algorithm significantly reduces the overhead of client systems from 10X to 10000X.Keywords:Cloud Computing, Encryption, Top-k Query

Uncertain top-k query processing in distributed environments

The top-k query on uncertain data set has been a very hot topic these years, and there have been many studies on uncertain top-k queries. Unfortunately, most of the existing algorithms only consider centralized processing environments, and they are not suitable for the large-scale data. In this paper, it is the first attempt to process probabilistic threshold top-k queries (an important uncertain top-k query, PT-k for short) in a distributed environment. We propose 3 efficient algorithms. The serial distributed approach adopts a new method, which only requires a few amount of calculations, to serially process PT-k queries in distributed environments. The global sorting first algorithm for PT-k query processing (GSP) is designed for improving the computation speed. In GSP, a distributed sorting operation is performed, and then we compute the candidates for PT-k queries in parallel. The query results can be computed by using a novel incremental method which can reduce the number of calculations. The local filtering first algorithm for PT-k query processing is designed for reducing the network overhead. Specifically, several filtering strategies are proposed to filter out redundant data locally, and then the incremental method in GSP is used to process the PT-k queries. Finally, the effectiveness of our proposed algorithms is verified through a series of experiments.

An Effective Uncertain Data Streams Top-K Query Algorithm

Large scale uncertain data streams are produced in many modern applications, such as RFID technology and sensor networks. Top-K query processing is one of the important techniques in the management of uncertain data streams. Existing Top-K queries processing does not consider the score and uncertainty of tuples. This paper first analyzes the un- certain data model and possible world semantic model, and then defines new Top-K queries semantics for uncertain data streams, and finally designs and realizes an effective Top-K queries algorithm on uncertain data streams. This algorithm sorts the score of each tuple and selects the k tuples with the highest probabilities to form the set, Top-K queries results. Compared to CSQ and SCSQ algorithm, the experiments show that this algorithm is more practical and effective than the others.

Efficient cluster-based top-k query routing with data replication in MANETs

Top-k queries, which retrieve k data items with the best score, have been receiving considerable attention because they can support many real life applications. In this paper, we propose an efficient query routing scheme in mobile ad hoc networks (MANETs), namely CTR. CTR enables top-k data retrieval by only necessary nodes by employing a new clustering framework for top-k query processing. In this framework, nodes holding high rank data items become ClusterHeads (CHs), and top-k queries are transmitted between CHs via gateway nodes which belong to multiple clusters. Each CH maintains a set of hop counts between itself and high rank data items, so that it can judge whether or not to transmit a query on the fly. We further propose a query routing method, CTR$$^{2}$$2, which integrates the clustering framework of CTR and a data replication approach. CTR$$^{2}$$2 improves the performance by retrieving the top-k data from nearby nodes. Extensive experiments have demonstrated that the proposed approaches function well in terms of accuracy of the query result, traffic, and delay.

Top-k query processing in mobile-P2P networks using economic incentive schemes

This work proposes the E-Top system for the efficient processing of top-k queries in mobile ad hoc peer to peer (M-P2P) networks using economic incentive schemes. In E-Top, brokers facilitate top-k query processing in lieu of a commission. E-Top issues economic rewards to the mobile peers, which send relevant data items (i.e., those that contribute to the top-k query result), and penalizes peers otherwise, thereby optimizing the communication traffic. Peers use the payoffs (rewards/penalties) as a means of feedback to re-evaluate the scores of their items for re-ranking purposes. The main contributions of E-Top are three-fold. First, it proposes two economic incentive schemes, namely ETK and ETK+, in which peers act individually towards top-k query processing. Second, it extends ETK and ETK+ to propose a peer group-based economic incentive scheme ETG. Third, our performance evaluation shows that our schemes are indeed effective in improving the performance of top-k queries in terms of query response times and accuracy at reasonable communication traffic cost.

Efficient Verifiable Top-k Queries in Two-tiered Wireless Sensor Networks

Tiered wireless sensor network is a network model of flexibility and robustness, which consists of the traditional resource-limited sensor nodes and the resource-abundant storage nodes. In such architecture, collected data from the sensor nodes are periodically submitted to the nearby storage nodes for archive purpose. When a query is requested, storage nodes also process the query and return qualified data as the result to the base station. The role of the storage nodes leads to an attack prone situation and leaves them more vulnerable in a hostile environment. If any of them is compromised, fake data may be injected into and/or qualified data may be discarded. And the base station would receive incorrect answers incurring malfunction to applications. In this paper, an efficient verifiable top-k query processing scheme called EVTQ is proposed, which is capable of verifying the authentication and completeness of the results. Collected data items with the embedded information of ordering and adjacent relationship through a hashed message authentication coding function, which serves as a validation code, are submitted from the sensor nodes to the storage nodes. Any injected or incomplete data in the returned result from a corresponded storage node is detected by the validation code at the base station. For saving communication cost, two optimized solutions that fuse and compress validation codes are presented. Experiments on communication cost show the proposed method is more efficiency than previous works.

Top-k Query Processing Techniques in Uncertain Databases: A Review

applications involving large databases with uncertain data require various techniques to rank queries. Ranking queries (often called as top-k) are useful in answering most important query answers in various domains such as web search, managing sensor data, location tracking, data mining tasks and multimedia. In this survey paper, we describe and classify different top-k processing techniques in probabilistic databases and their implications. Keywordsdatabase, ranking queries, sliding window, possible world, top-k query.