Top-k Query Research Articles

Efficient Top-K Continuous Query Processing Over Sliding Window Model (SWM) Method on Uncertain Data Stream

Query processing using the Uncertain Data Stream (UDS) can be complex in many technological scenarios due to inconsistencies, unclear information, and interpretation latency. As a result of both the sheer amount of data generated and the rate of change, traditional processing methods are in dire need of an upgrade. UDS consists of a finite set of states known as possible worlds (PW), and enhancing data organization can lead to more accurate extraction of user preferences. The number of possible world instances in UDS grows exponentially, making achieving Top-k query processing quickly a significant challenge. Different methods are available to handle Top-k queries in various types of UDS, and their key concerns include reducing duplicate scans of the entire dataset, enhancing uncertainty computation, and focusing on processing the latest tuple item entry. It appears that there have been limited studies conducted on the issue of UDS using the Sliding Window Model (SWM). The current approach for handling continuous queries on UDS within the SWM has proven to be ineffective, resulting in complex trade-offs between maximizing probability and generating high-scoring result sets. The challenge is to find the correct result list that satisfies a Top-k query predicate with scoring and probability. This study proposes a framework for processing Top-k queries for UDS using the sliding window model to improve efficiency. The study also discusses an improved optimization method for reducing computational redundancy in the context of the sliding window model and Top-k query processing. Overall, this research will significantly contribute to the Top-k computational query processing field.

Durable reverse top-k queries on time-varying preference

Durable reverse top-k queries on time-varying preference

Relational Algorithms for Top-k Query Evaluation

The evaluation of top-k conjunctive queries, a staple in business analysis, often requires evaluating the conjunctive query prior to filtering the top-k results, leading to a significant computational overhead within Database Management Systems (DBMSs). While efficient algorithms have been proposed, their integration into DBMSs remains arduous. We introduce relational algorithms, a paradigm where each algorithmic step is expressed by a relational operator. This allows the algorithm to be represented as a set of SQL queries, enabling easy deployment across different systems that support SQL. We introduce two novel relational algorithms, level-k and product-k, specifically designed for evaluating top-k conjunctive queries and demonstrate that level-k achieves optimal running time for top-k free-connex queries. Furthermore, these algorithms enable easy translation into an oblivious algorithm for secure query evaluations. The presented algorithms are not only theoretically optimal but also exhibit eminent efficiency in practice. The experiment results show significant improvements, with our rewritten SQL outperforming the baseline by up to 6 orders of magnitude. Moreover, our secure implementations not only achieve substantial speedup compared to the baseline with secure guarantees but even surpass those baselines that have no secure guarantees.

Predictive and Near-Optimal Sampling for View Materialization in Video Databases

Scalable video query optimization has re-emerged as an attractive research topic in recent years. The OTIF system, a video database with cutting-edge efficiency, has introduced a new paradigm of utilizing view materialization to facilitate online query processing. Specifically, it stores the results of multi-object tracking queries to answer common video queries with sub-second latency. However, the cost associated with view materialization in OTIF is prohibitively high for supporting large-scale video streams. In this paper, we study efficient MOT-based view materialization in video databases. We first conduct a theoretical analysis and establish two types of optimality measures that serve as lower bounds for video frame sampling. In order to minimize the number of processed video frames, we propose a novel predictive sampling framework, namely LEAP, exhibits near-optimal sampling performance. Its efficacy relies on a data-driven motion manager that enables accurate trajectory prediction, a compact object detection model via knowledge distillation, and a robust cross-frame associator to connect moving objects in two frames with a large time gap. Extensive experiments are conducted in 7 real datasets, with 7 baselines and a comprehensive query set, including selection, aggregation and top-k queries. The results show that with comparable query accuracy to OTIF, our LEAP can reduce the number of processed video frames by up to 9× and achieve 5× speedup in query processing time. Moreover, LEAP demonstrates impressive throughput when handling large-scale video streams, as it leverages a single NVIDIA RTX 3090ti GPU to support real-time MOT-based view materialization from 160 video streams simultaneously.

Lightweight verifiable blockchain top-[formula omitted] queries

Blockchain has been exploited in many applications as a fundamental technology to construct trust and share data among multiple participants. A user with limited resources who runs a light node fetches data records stored on the blockchain by requesting a full node that maintains the complete blockchain data. As a type of broadly used query, top-k queries which ask for data records with the highest k values in a certain region can be useful in many blockchain scenarios. However, untrusted full nodes may return incorrect top-k results. In this paper, we take the first step toward addressing the problem of verifiable blockchain top-k queries without the help of a trusted third-party. First, we design a novel Sorted Merkle tree-based authenticated data structure and based on it we propose a new index named inverted SMT index that supports verifiable top-k query with specific keyword conditions. Second, we propose a fundamental strawman design to support verifiable blockchain top-k queries with arbitrary query regions. We further introduce two improved schemes, named segmentation design and sliding-window design, that overcome the efficiency obstacle of the strawman design. Finally, we confirm the efficiency of our solutions through an extensive experimental evaluation.

Query Optimization in Distributed Database Based on Improved Artificial Bee Colony Algorithm

Query optimization is one of the key factors affecting the performance of database systems that aim to enact the query execution plan with minimum cost. Particularly in distributed database systems, due to the multiple copies of the data that are stored in different data nodes, resulting in the dramatic increase in the feasible query execution plans for a query statement. Because of the increasing volume of stored data, the cluster size of distributed databases also increases, resulting in poor performance of current query optimization algorithms. In this case, a dynamic perturbation-based artificial bee colony algorithm is proposed to solve the query optimization problem in distributed database systems. The improved artificial bee colony algorithm improves the global search capability by combining the selection, crossover, and mutation operators of the genetic algorithm to overcome the problem of falling into the local optimal solution easily. At the same time, the dynamic perturbation factor is introduced so that the algorithm parameters can be dynamically varied along with the process of iteration as well as the convergence degree of the whole population to improve the convergence efficiency of the algorithm. Finally, comparative experiments conducted to assess the average execution cost of Top-k query plans generated by the algorithms and the convergence speed of algorithms under the conditions of query statements in six different dimension sets. The results demonstrate that the Top-k query plans generated by the proposed method have a lower execution cost and a faster convergence speed, which can effectively improve the query efficiency. However, this method requires more execution time.

Cluster-based multi-hop wake-up control for top-k query in wireless sensor networks

Cluster-based multi-hop wake-up control for top-k query in wireless sensor networks

Efficient Approximation Framework for Attribute Recommendation

Trend analysis is a fundamental type of analytical query in online analytical processing (OLAP) systems. In trend analysis, a key step is to identify k valuable attributes whose distributions in two subsets under different predicates significantly differ for further investigation, where the difference is measured by metric functions. However, the exact solution that involves scanning all records is prohibitively expensive, particularly when handling large datasets in the era of big data. To minimize unnecessary data access, the existing state-of-the-art solution TopKAttr adopts sampling to avoid the expensive data scan. However, their solution still has two main drawbacks. Firstly, their solution is tailored only for two limited metric functions: the Earth Mover distance and Euclidean distance, and cannot be generalized to more complicated metric functions. Besides, their solution still aims to return the exact top-k answers via the sampling method, which still causes high running costs as shown in our experiment. Motivated by these limitations, we propose a general approximation framework for attribute recommendation that efficiently returns the top-k attributes with theoretical guarantees while supporting an extensive range of metric functions, such as the Kolmogorov-Smirnov test (KS-test), Chebyshev distance, the Earth Mover distance, Euclidean distance, and with the potential to more metrics. The key to our framework is a new bound estimation strategy that can be applied to a wide spectrum of metrics, as we listed above. Based on our estimation framework, we further devise an efficient approximation algorithm with theoretical guarantees to answer the top-k queries, which is widely used in attribute recommendation. Extensive experiments on four real large datasets show that our framework gains up to an order of magnitude speed-up and consistently high accuracy compared to TopKAttr, providing a promising alternative for attribute recommendation in OLAP systems.

PPT-LBS: Privacy-preserving top-k query scheme for outsourced data of location-based services

Location-based service (LBS) is enjoying a great popularity with the fast growth of mobile Internet. As the volume of data increases dramatically, an increasing number of location service providers (LSPs) are moving LBS data to cloud platforms for benefit of affordability and stability. However, while cloud server provides convenience and stability, it also leads to data security and user privacy leakage. Aiming at the problems of insufficient privacy protection and inefficient query in the existing LBS data outsourcing schemes, this paper presents a novel privacy-preserving top-k query for outsourcing situations. Firstly, to ensure data security of LSP and privacy of the user, the enhanced asymmetric scalar-product preserving encryption and public key searchable encryption have been adopted to encrypt outsourced data and LBS query, which can effectively lower the computational cost and realize the privacy protection search. Secondly, an efficient and secure index structure is constructed by using a coded quadtree and the bloom filter, so that the cloud server can quickly locate the user’s query region to improve retrieval efficiency. Finally, the formal security analysis is given under the random oracle model, and the performance is evaluated by experiments which demonstrates that our scheme is preferable to existing schemes.

DoveDB: A Declarative and Low-Latency Video Database

Concerning the usability and efficiency to manage video data generated from large-scale cameras, we demonstrate DoveDB, a declarative and low-latency video database. We devise a more comprehensive video query language called VMQL to improve the expressiveness of previous SQL-like languages, which are augmented with functionalities for model-oriented management and deployment. We also propose a light-weight ingestion scheme to extract tracklets of all the moving objects and build semantic indexes to facilitate efficient query processing. For user interaction, we construct a simulation environment with 120 cameras deployed in a road network and demonstrate three interesting scenarios. Using VMQL, users are allowed to 1) train a visual model using SQL-like statement and deploy it on dozens of target cameras simultaneously for online inference; 2) submit multi-object tracking (MOT) requests on target cameras, store the ingested results and build semantic indexes; and 3) issue an aggregation or top- k query on the ingested cameras and obtain the response within milliseconds. A preliminary video introduction of DoveDB is available at https://www.youtube.com/watch?v=N139dEyvAJk

A unified model of data uncertainty and data relation uncertainty

Data uncertainty (DU), one of the most interesting topics in information science, inevitably leads to data relation uncertainty (DRU). Moreover, DU greatly affects the results of information processing problems such as sorting and top-k queries with uncertain data. However, studies have not clarified the mechanism, mode, and degree of impact of DU on DRU. Hence, we established a unified mathematical model to reveal the precise relation between DU and DRU. The experimental results showed this model’s high accuracy under various conditions and its relative error and absolute error of less than 0.2%. This model can be used to quantitatively estimate DRU according to a given DU and vice versa. Unexpectedly, the experiments showed that under given conditions, high DU will not necessarily lead to high DRU. This characteristic is highly significant to the design of data sorting and query algorithms with uncertain data.

RkHit: Representative Query with Uncertain Preference

A top-k query retrieves the k tuples with highest scores according to a user preference, defined as a scoring function. It is difficult for a user to precisely specify the scoring function. Instead, obtaining the distribution on scoring functions, i.e., the preference distribution, has been extensively explored in many fields. Motivated by this, we introduce the uniform (r,k)-hit (UrkHit) problem. Given a preference distribution, UrkHit aims to select a representative set of r tuples to maximize the probability of containing a tuple attractive to the user. We say a tuple attracts a user, if it is a top-k tuple for the scoring function adopted by the user. Further, we generalize UrkHit and propose the (r,k)-hit (rkHit) problem with an additional penalty function to model the user satisfaction with the tuple ranked i-th. rkHit aims to maximize the expected user satisfaction with the representative set. In 2D space, we design an exact algorithm 2DH for rkHit, indicating rkHit is in P for d=2. We show that rkHit is NP-hard when d\ge3. In 3D space, assuming a uniform preference distribution, we propose a (1-1/e)-approximation algorithm 3DH based on space partitioning. In addition, we propose an approximate algorithm MDH suitable for any dimension and distribution, which creatively combines the ideas of sampling and clustering. It relaxes the approximation guarantee slightly. Comprehensive experiments demonstrate the efficiency and effectiveness of our algorithms.

Comparative Studies on Intelligent Swarming Network (iSWAN) Geno-Generative Algorithm and Top-K Query Processing Algorithm

This paper proposed an enhanced Top-k query processing in a real time distributed database system. The system employs a Particle Swarm Optimizer (PSO) based Geno-Generative iSWAN Model technique that enhances and allows multi-task concurrent query processing in a real time co-simulation data acquisition platform and as part of refinement to an existing Top-k query processing Technique. In this paper, the proposed system is compared for efficiency with the Top-K Query Algorithm, which is emerging as an alternative to more conventional technique for real time query processing in distributed databases. Dynamic simulations were performed with a real time small testbed comprising of physical and non-physical devices to test and evaluate the performance and efficiency of the two systems. Considering the estimated and expected temperatures, the result of simulation study proves that the Intelligent Swarming Network (iSWAN) Geno-Generative Model is more preferred over Top-K Query Algorithm due to its 70% accuracy over the Top-K Model, which reported a lower accuracy level of 40%.

Why Not Yet: Fixing a Top-k Ranking that is Not Fair to Individuals

This work considers why-not questions in the context of top-k queries and score-based ranking functions. Following the popular linear scalarization approach for multi-objective optimization, we study rankings based on the weighted sum of multiple scores. A given weight choice may be controversial or perceived as unfair to certain individuals or organizations, triggering the question why some entity of interest has not yet shown up in the top-k. We introduce various notions of such why-not-yet queries and formally define them as satisfiability or optimization problems, whose goal is to propose alternative ranking functions that address the placement of the entities of interest. While some why-not-yet problems have linear constraints, others require quantifiers, disjunction, and negation. We propose several optimizations, ranging from a monotonic-core construction that approximates the complex constraints with a conjunction of linear ones, to various techniques that let the user control the tradeoff between running time and approximation quality. Experiments with real and synthetic data demonstrate the practicality and scalability of our technique, showing its superiority compared to the state of the art (SOA).

A Cost-Driven Top-K Queries Optimization Approach on Federated RDF Systems

RDF (Resource Description Framework) is a model widely used to construct knowledge bases, while SPARQL (SPARQL Protocol and RDF Query Language) is the standardized structured query language to manipulate RDF data. Recently, many data providers have published their RDF datasets in their own autonomous sites and provided SPARQL query interfaces, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RDF sources</i> . In order to integrate multiple RDF sources, researchers put forward the federated RDF system to support the federated SPARQL queries. However, existing studies can only support efficient basic queries but not top-k queries. Toward this end, we propose a cost-driven top-k queries optimization approach in federated RDF systems, which can support both top-k queries for single variable ordering and expression ordering. Firstly, we propose an optimized query decomposition method to decompose the federated query into multiple subqueries. Secondly, while considering the top-k operator, we propose a cost model to evaluate the query cost and join cost of subqueries. The optimal query plan can be obtained by the costed-based query plan generation algorithm. Finally, combined with the characteristics of top-k queries, an incremental query plan execution strategy is developed to minimize the total query cost. Experimental results show that the proposed method is effective, efficient and scalable.

Many are Better than One: Algorithm Selection for Faster Top-K Retrieval

Large-scale search engines have become a fundamental tool to efficiently access information on the Web. Typically, users expect answers in sub-second time frames, which demands highly efficient algorithms to traverse the data structures to return the top-k results. Despite different top-k algorithms that avoid processing all postings for all query terms, finding one algorithm that performs the fastest on any query is not always possible. The fastest average algorithm does not necessarily perform the best on all queries when evaluated on a per-query basis. To overcome this challenge, we propose to combine different state-of-the-art disjunctive top-k query processing algorithms to minimize the execution time by selecting the most promising one for each query. We model the selection step as a classification problem in a machine-learning setup. We conduct extensive experimentation and compare the results against state-of-the-art baselines using standard document collections and query sets. On ClueWeb12, our proposal shows a speed-up of up to 1.20x for non-blocked index organizations and 1.19x for block-based ones. Moreover, tail latencies are reduced showing proportional improvements on average, but a resulting dramatic decrease in latency variance. Given these findings, the proposed approach can be easily applied to existing search infrastructures to speed up query processing and reduce resource consumption, positively impacting providers’ operative costs.

Efficient Approximation of Certain and Possible Answers for Ranking and Window Queries over Uncertain Data

Uncertainty arises naturally in many application domains due to, e.g., data entry errors and ambiguity in data cleaning. Prior work in incomplete and probabilistic databases has investigated the semantics and efficient evaluation of ranking and top-k queries over uncertain data. However, most approaches deal with top-k and ranking in isolation and do represent uncertain input data and query results using separate, incompatible data models. We present an efficient approach for under- and over-approximating results of ranking, top-k, and window queries over uncertain data. Our approach integrates well with existing techniques for querying uncertain data, is efficient, and is to the best of our knowledge the first to support windowed aggregation. We design algorithms for physical operators for uncertain sorting and windowed aggregation, and implement them in PostgreSQL. We evaluated our approach on synthetic and real world datasets, demonstrating that it outperforms all competitors, and often produces more accurate results.

Top-K Query for Large Dataset of Restaurant Review Based-on Hadoop MapReduce Framework

In order to develop post-COVID-19 culinary tourism, high-quality facilities and services are essential. Information technology can contribute through a top-k query-based decision-making system. This study implements top-k queries on a distributed Hadoop MapReduce system to evaluate its capability in managing data and selecting culinary tourism potential. Research findings indicate that for the European restaurant data from TripAdvisor with 5 dimensions, both single-node and multi-node (3 nodes) executions exhibit comparable execution times across various data quantities. Conversely, for the European restaurant data from TripAdvisor with 14 dimensions, the use of multi-node (3 nodes) tends to result in longer execution times compared to the single-node approach for larger data quantities. Furthermore, the utilization of multi-node (3 nodes) proves to be more efficient in processing synthetic data with 5 dimensions as the data quantity increases, demonstrating a significant difference in execution times compared to the single-node approach. The study also reveals that across different dimensions, the multi-node (3 nodes) approach generally outperforms the single-node approach in terms of speed. Regarding node variations, processing 20 million data points with 5 dimensions using 6 nodes yields the optimal method with the shortest execution time. By leveraging information technology and a top-k query-based decision-making system, the development of culinary tourism potential can be conducted more efficiently and effectively. The performance of MapReduce in processing culinary tourism potential data can be optimized by employing multi-node execution for large datasets and single-node execution for relatively smaller datasets.

A Classification Method of Reader Borrowing Data Information in Modern Library Based on Top-k Query Algorithm

A Classification Method of Reader Borrowing Data Information in Modern Library Based on Top-k Query Algorithm

Durable queries over non-synchronized temporal data

Temporal data are ubiquitous nowadays and efficient management of temporal data is of key importance. A temporal data typically describes the evolution of an object over time. One of the most useful queries over temporal data are the durable top-k queries. Given a time window, a durable top-k query finds the objects that are frequently among the best. Existing solutions to durable top-k queries assume that all temporal data are sampled at the same time points (i.e., at any time, there is a corresponding observed value for every temporal data). However, in many practical applications, temporal data are collected from multiple data sources with different sampling rates. In this light, we investigate the efficient processing of durable top-k queries over temporal data with different sampling rates. We propose an efficient sweep line algorithm to process durable top-k queries over non-synchronized temporal data. We conduct extensive experiments on two real datasets to test the performance of our proposed method. The results show that our methods outperforms the baseline solutions by a large margin.