Stream Query Research Articles

Near-Duplicate Subsequence Matching Between the Continuous Stream and Large Video Dataset

In this paper, we study the subsequence matching problem of near-duplicate video detection. In particular, we address the application of monitoring a continuous stream with a large video dataset. To achieve real-time response and high accuracy, we propose a novel framework containing two characteristics . First, the subsequence matching is transformed to a 2-D Hough space projection of pairwise frame similarities between two subsequences. We present an approximate Hough transform that replaces the 2-D Hough space with a 1-D Hough space. The near-duplicate subsequence detection can be deemed to be the voting and searching in the 1-D Hough space with a lower time complexity. Second, a coarse-to-fine matching strategy is incorporated in the proposed framework. The coarse level matching selects the candidate videos based on the time-decay hit frequency between the query stream and dataset videos. The fine level matching applies the approximate Hough transform to detect the near-duplicate subsequences coexisting within the query stream and candidate videos. Several state-of-the-art methods are implemented for comparison. Experimental results show our framework outperforms in terms of accuracy and efficiency.

Scalable real-time OLAP on cloud architectures

In contrast to queries for on-line transaction processing (OLTP) systems that typically access only a small portion of a database, OLAP queries may need to aggregate large portions of a database which often leads to performance issues. In this paper we introduce CR-OLAP, a scalable Cloud based Real-time OLAP system based on a new distributed index structure for OLAP, the distributed PDCR tree. CR-OLAP utilizes a scalable cloud infrastructure consisting of multiple commodity servers (processors). That is, with increasing database size, CR-OLAP dynamically increases the number of processors to maintain performance. Our distributed PDCR tree data structure supports multiple dimension hierarchies and efficient query processing on the elaborate dimension hierarchies which are so central to OLAP systems. It is particularly efficient for complex OLAP queries that need to aggregate large portions of the data warehouse, such as “report the total sales in all stores located in California and New York during the months February–May of all years”. We evaluated CR-OLAP on the Amazon EC2 cloud, using the TPC-DS benchmark data set. The tests demonstrate that CR-OLAP scales well with increasing number of processors, even for complex queries. For example, for an Amazon EC2 cloud instance with 16 processors, a data warehouse with 160 million tuples, and a TPC-DS OLAP query stream where each query aggregates between 60% and 95% of the database, CR-OLAP achieved a query latency of below 0.3 s which can be considered a real time response.

A Sliding-Window Method to Discover Recent Frequent Query Patterns from XML Query Streams

Providing efficient mining algorithm to discover recent frequent XML user query patterns is crucial, as many applications use XML to represent data in their disciplines over the Internet. These recent frequent XML user query patterns can be used to design an index mechanism or cached and thus enhance XML query performance. Several XML query pattern stream mining algorithms have been proposed to record user queries in the system and thus discover the recent frequent XML query patterns over a stream. By using these recent frequent XML query patterns, the query performance of XML data stream is improved. In this paper, user queries are modeled as a stream of XML queries and the recent frequent XML query patterns are thus mined over the stream. Data-stream mining differs from traditional data mining since its input of mining is data streams, while the latter focuses on mining static databases. To facilitate the one-pass mining process, novel schemes (i.e. XstreamCode and XstreamList) are devised in the mining algorithm (i.e. X2StreamMiner) in this paper. X2StreamMiner not only reduces the memory space, but also improves the mining performance. The simulation results also show that X2StreamMiner algorithm is both efficient and scalable. There are two major contributions in this paper. First, the novel schemes are proposed to encode and store the information of user queries in an XML query stream. Second, based on the two schemes, an efficient XML query stream mining algorithm, X2StreamMiner, is proposed to discover the recent frequent XML query patterns.

Distributed text search using suffix arrays

We introduce distributed-memory text-search algorithms based on suffix arrays.We analyze the proposed search algorithms to determine their theoretical speed-up.We do intensive experimentation on high-performance clusters of processors.We assess our practicality using also web search engines as a case study. Text search is a classical problem in Computer Science, with many data-intensive applications. For this problem, suffix arrays are among the most widely known and used data structures, enabling fast searches for phrases, terms, substrings and regular expressions in large texts. Potential application domains for these operations include large-scale search services, such as Web search engines, where it is necessary to efficiently process intensive-traffic streams of on-line queries. This paper proposes strategies to enable such services by means of suffix arrays. We introduce techniques for deploying suffix arrays on clusters of distributed-memory processors and then study the processing of multiple queries on the distributed data structure. Even though the cost of individual search operations in sequential (non-distributed) suffix arrays is low in practice, the problem of processing multiple queries on distributed-memory systems, so that hardware resources are used efficiently, is relevant to services aimed at achieving high query throughput at low operational costs. Our theoretical and experimental performance studies show that our proposals are suitable solutions for building efficient and scalable on-line search services based on suffix arrays.

Biperpedia

Search engines make significant efforts to recognize queries that can be answered by structured data and invest heavily in creating and maintaining high-precision databases. While these databases have a relatively wide coverage of entities, the number of attributes they model (e.g., GDP, CAPITAL, ANTHEM) is relatively small. Extending the number of attributes known to the search engine can enable it to more precisely answer queries from the long and heavy tail, extract a broader range of facts from the Web, and recover the semantics of tables on the Web. We describe Biperpedia, an ontology with 1.6M (class, attribute) pairs and 67K distinct attribute names. Biperpedia extracts attributes from the query stream, and then uses the best extractions to seed attribute extraction from text. For every attribute Biperpedia saves a set of synonyms and text patterns in which it appears, thereby enabling it to recognize the attribute in more contexts. In addition to a detailed analysis of the quality of Biperpedia, we show that it can increase the number of Web tables whose semantics we can recover by more than a factor of 4 compared with Freebase.

맵리듀스에서 집계 질의 스트림의 효율적인 처리 기법

맵리듀스는 빅데이터 분석 및 처리에 널리 사용되는 프로그래밍 모델이다. 빅데이터 분석을 위해 흔히 사용되는 질의 중 하나는 집계 질의(aggregate query)이다. 본 논문에서는 여러 사용자가 동시에 여러 집계 질의를 계속해서 요청하는 경우, 맵리듀스를 사용하여 이들 질의를 효율적으로 처리하는 방법을 제안한다. 제안 방법은 각 집계 질의를 개별적으로 처리하지 않고, 여러 집계 질의를 묶어 하나의 최적화된 맵리듀스 잡(job)으로 만들어 일괄 처리한다. 그 결과로 제안 방법은 단순 방법에 비해 시간당 처리하는 질의 수를 크게 증가시킨다. 성능 평가를 통해, 제안 방법은 단순 방법에 비해 질의 처리 속도를 크게 향상시킴을 보인다. MapReduce is a widely used programming model for analyzing and processing Big data. Aggregate queries are one of the most common types of queries used for analyzing Big data. In this paper, we propose an efficient method for processing an aggregate query stream, where many concurrent users continuously issue different aggregate queries on the same data. Instead of processing each aggregate query separately, the proposed method processes multiple aggregate queries together in a batch by a single, optimized MapReduce job. As a result, the number of queries processed per unit time increases significantly. Through various experiments, we show that the proposed method improves the performance significantly compared to a naive method.

A Review of Window Query Processing for Data Streams

In recent years, progress in hardware technology has resulted in the possibility of monitoring many events in real time. The volume of incoming data may be so large, that monitoring all individual data might be intractable. Revisiting any particular record can also be impossible in this environment. Therefore, many database schemes, such as aggregation, join, frequent pattern mining, and indexing, become more challenging in this context. This paper surveys the previous efforts to resolve these issues in processing data streams. The emphasis is on specifying and processing sliding window queries, which are supported in many stream processing engines. We also review the related work on stream query processing, including synopsis structures, plan sharing, operator scheduling, load shedding, and disorder control.

High performance stream query processing with correlation-aware partitioning

State-of-the-art optimizers produce one single optimal query plan for all stream data, in spite of such a singleton plan typically being sub-optimal or even poor for highly correlated data. Recently a new stream processing paradigm, called multi-route approach, has emerged as a promising approach for tackling this problem. Multi-route first divides data streams into several partitions and then creates a separate query plan for each combination of partitions. Unfortunately current approaches suffer from severe shortcomings, in particular, the lack of an effective partitioning strategy and the prohibitive query optimization expense. In this work we propose the first practical multi-route optimizer named <u>c</u>orrelation-aware <u>m</u>ulti-<u>r</u>oute stream query optimizer (or CMR) that solves both problems. By exploiting both intra- and inter-stream correlations of streams, CMR produces effective partitions without having to undertake repeated expensive query plan generation. The produced partitions not only are best served by distinct optimal query plans, but also leverage the partition-driven pruning opportunity. Experimental results with both synthetic and real life stream data confirm that CMR outperforms the state-of-the-art solutions up to an order of magnitude in both the query optimization time and the run-time execution performance.

Second Chance

Web search engines are known to cache the results of previously issued queries. The stored results typically contain the document summaries and some data that is used to construct the final search result page returned to the user. An alternative strategy is to store in the cache only the result document IDs, which take much less space, allowing results of more queries to be cached. These two strategies lead to an interesting trade-off between the hit rate and the average query response latency. In this work, in order to exploit this trade-off, we propose a hybrid result caching strategy where a dynamic result cache is split into two sections: an HTML cache and a docID cache. Moreover, using a realistic cost model, we evaluate the performance of different result prefetching strategies for the proposed hybrid cache and the baseline HTML-only cache. Finally, we propose a machine learning approach to predict singleton queries, which occur only once in the query stream. We show that when the proposed hybrid result caching strategy is coupled with the singleton query predictor, the hit rate is further improved.

Efficient Video Stream Monitoring for Near-Duplicate Detection and Localization in a Large-Scale Repository

In this article, we study the efficiency problem of video stream near-duplicate monitoring in a large-scale repository. Existing stream monitoring methods are mainly designed for a short video to scan over a query stream; they have difficulty being scalable for a large number of long videos. We present a simple but effective algorithm called incremental similarity update to address the problem. That is, a similarity upper bound between two videos can be calculated incrementally by leveraging the prior knowledge of the previous calculation. The similarity upper bound takes a lightweight computation to filter out unnecessary time-consuming computation for the actual similarity between two videos, making the search process more efficient. We integrate the algorithm with inverted indexing to obtain a candidate list from the repository for the given query stream. Meanwhile, the algorithm is applied to scan each candidate for locating exact near-duplicate subsequences. We implement several state-of-the-art methods for comparison in terms of accuracy, execution time, and memory consumption. Experimental results demonstrate the proposed algorithm yields comparable accuracy, compact memory size, and more efficient execution time.

Préférences contextuelles pour la personnalisation des requêtes sur flux de données

Preference queries have been largely studied for relational systems but few propo- sals exist for stream data systems. Most of the existing proposals concern the skyline, top-k or top-k dominating queries, coupled with the sliding-window operator. However, user preferences queries on data streams may be more sophisticated than skyline or top-k and may involve more expressive operations on streams. This paper improves the existing work on data stream query- answering personalization by proposing a solution to express and handle contextual preferences together with a large variety of queries including one-shot and continuous queries. It adopts a more expressive preference model supporting context-based preferences, allowing to capture a wide range of situations. We propose algorithms to implement the new preference operators on stream data and validate their performance on a real-world dataset of stock market streams.

Flexible query processor on FPGAs

In this work, we demonstrate Flexible Query Processor (FQP), an online reconfigurable event stream query processor. FQP is an FPGA-based query processor that supports select, project and join queries over event streams at line rate. While processing incoming events, FQP can accept new query expressions, a key distinguishing characteristic from related approaches employing FPGAs for acceleration. Our solution aims to address performance limitations experienced with general purpose processors needing to operate at line rate and lack of on the fly reconfigurability with custom designed hardware solutions on FPGAs.

Sketch-based geometric monitoring of distributed stream queries

Emerging large-scale monitoring applications rely on continuous tracking of complex data-analysis queries over collections of massive, physically-distributed data streams. Thus, in addition to the space- and time-efficiency requirements of conventional stream processing (at each remote monitor site), effective solutions also need to guarantee communication efficiency (over the underlying communication network). The complexity of the monitored query adds to the difficulty of the problem -- this is especially true for nonlinear queries (e.g., joins), where no obvious solutions exist for distributing the monitor condition across sites. The recently proposed geometric method offers a generic methodology for splitting an arbitrary (non-linear) global threshold-monitoring task into a collection of local site constraints; still, the approach relies on maintaining the complete stream(s) at each site, thus raising serious efficiency concerns for massive data streams. In this paper, we propose novel algorithms for efficiently tracking a broad class of complex aggregate queries in such distributed-streams settings. Our tracking schemes rely on a novel combination of the geometric method with compact sketch summaries of local data streams, and maintain approximate answers with provable error guarantees, while optimizing space and processing costs at each remote site and communication cost across the network. One of our key technical insights for the effective use of the geometric method lies in exploiting a much lower-dimensional space for monitoring the sketch-based estimation query. Due to the complex, highly nonlinear nature of these estimates, efficiently monitoring the local geometric constraints poses challenging algorithmic issues for which we propose novel solutions. Experimental results on real-life data streams verify the effectiveness of our approach.

A Novel Energy Efficient Hierarchical Multi Level Cluster Routing Protocol based on LEACH

As the use of Wireless sensor networks (WSNs) has grown enormously in the past few decades, the need of scalable & energy efficient routing and data aggregation protocol for large scale deployment has also risen. LEACH is a hierarchical clustering protocol that provides an elegant solution for such protocols. One deficiency that affects the performance of the protocol is existence of very large and very small clusters in the network at the same time. This leads to decrease in lifetime of WSNs. Sensors operate on battery of limited power, so it is a great challenging aim to design an energy efficient routing protocol, which can minimize the delay while offering highenergy efficiency and long span of network lifetime. In this paper, a new protocol is proposed. The proposed protocol is energy efficient multi level cluster-based routing protocol for continuous stream queries in WSN for large network areas. We incorporate multi hop concept into LEACH [3] by dividing clusters into levels as in TL-LEACH [8] and consider a set of cluster heads, head-set, for cluster-based routing. The head-set members are responsible for control and management of the network. On rotation basis, a head-set member receives data from the neighboring nodes and from previous level, then aggregated data is transmitted to node of next level and so on, until it reaches the distant base station. For a given number of data collecting sensor nodes, the number of control and management nodes can be systematically adjusted to reduce the energy consumption, which increases network life and multi hop transmission reduces energy consumption as distance between transmitting nodes is reduced.

Report on the first workshop on innovative querying of streams

was held on September 25-27, 2012 in the Department of Computer Science of the University of Oxford (UK). It was sponsored by the UK’s Engineering and Physical Sciences Research Council (EPSRC), as part of the project “Enforcement of Constraints on XML Streams”. Stream processing represents a thriving area of research across the algorithms, databases, networking, programming languages, and systems research communities. Within the database community, a “classical” problem is query processing on streams of discrete tuple-oriented data. One goal of the workshop considers the way recent developments add complexity to this problem:

Report on the SIGIR 2013 workshop on health search and discovery

The workshop brought together 40 researchers and practitioners from academia and industry to discuss search and discovery in the medical domain. Presentations and discussions spanned several challenging and important topics, including directions improving the accessibility of medical and health information for lay people (with associated enhancements to result ranking algorithms and search interfaces), and methods for discovering biomedical phenomena from the information that people seek online, as evidenced in query streams and other sources such as social and news media. A thread throughout the workshop was the opportunity for new methods and applications to enhance the quality of life of people suffering from medical disorders, carry out surveillance of emerging diseases and other threats, and, more generally, to improve the health and well-being of people via tools to support their health-related information behavior.

Evaluation of RDF queries via equivalence

Performance and scalability are two issues that are becoming increasingly pressing as the resource description framework (RDF) datamodel is applied to real-world applications. Because neither vertical nor flat structures of RDF storage can handle frequent schema updates and meanwhile avoid possible long-chain joins, there is no clear winner between the two typical structures. In this paper, we propose an alternative open user schema. The open user schema consists of flat tables automatically extracted from RDF query streams. A query is divided into two parts and conquered on the flat tables in the open user schema and on the vertical table stored in a backend storage. At the core of this divide and conquer architecture with open user schema, an efficient isomorphic decision algorithm is introduced to guide a query to related flat tables in the open user schema. Our proposal in essence departs from existing methods in that it can accommodate schema updates without possible long-chain joins. We implement our approach and provide empirical evaluations to demonstrate both the efficiency and effectiveness of our approach in evaluating complex RDF queries.

Fay

Fay is a flexible platform for the efficient collection, processing, and analysis of software execution traces. Fay provides dynamic tracing through use of runtime instrumentation and distributed aggregation within machines and across clusters. At the lowest level, Fay can be safely extended with new tracing primitives, including even untrusted, fully optimized machine code, and Fay can be applied to running user-mode or kernel-mode software without compromising system stability. At the highest level, Fay provides a unified, declarative means of specifying what events to trace, as well as the aggregation, processing, and analysis of those events. We have implemented the Fay tracing platform for Windows and integrated it with two powerful, expressive systems for distributed programming. Our implementation is easy to use, can be applied to unmodified production systems, and provides primitives that allow the overhead of tracing to be greatly reduced, compared to previous dynamic tracing platforms. To show the generality of Fay tracing, we reimplement, in experiments, a range of tracing strategies and several custom mechanisms from existing tracing frameworks. Fay shows that modern techniques for high-level querying and data-parallel processing of disagreggated data streams are well suited to comprehensive monitoring of software execution in distributed systems. Revisiting a lesson from the late 1960s [Deutsch and Grant 1971], Fay also demonstrates the efficiency and extensibility benefits of using safe, statically verified machine code as the basis for low-level execution tracing. Finally, Fay establishes that, by automatically deriving optimized query plans and code for safe extensions, the expressiveness and performance of high-level tracing queries can equal or even surpass that of specialized monitoring tools.

Optimizing adaptive multi-route query processing via time-partitioned indices

Adaptive multi-route query processing (AMR) is an emerging paradigm for processing stream queries in highly fluctuating environments. The content of stream data can be unpredictable. Thus, instead of selecting a fixed plan, AMR dynamically routes batches of tuples to operators in the query network based on up-to-date system statistics. The workload of query access patterns in AMR systems is ever changing. Selecting a single best index may not efficiently support all query access patterns at all times. While maintaining multiple indices to match a variety of query access patterns increases overhead and decreases throughput. Index design, while paramount for efficient query execution, is particularly challenging in AMR systems as the indices must serve the continuously evolving query access patterns. Our proposed Adaptive Multi-Route Index (AMRI) employs a bitmap time-partitioned design that serves a diverse ever changing workload of query access patterns and remains lightweight in terms of maintenance and storage requirements. We propose a high quality yet efficient assessment method modeled after hierarchical heavy hitters that exploits route relationships by modeling the frequency of the search access patterns used as nodes in a lattice. We also design assessment scheduling methods for AMRI based upon detecting changes in the search access patterns used. Our AMRI incorporates migration strategies that seek to meet the needs of both old partially serviced and new incoming search requests. Our experimental study using both synthetic and real data streams demonstrates that AMRI strikes a balance between effectively supporting dynamic stream environments while keeping the index overhead to a minimum. Using an environmental data set collected in the Intel Berkeley Research lab, our AMRI produced on average 68% more cumulative throughput than the state-of-the-art approach.

User-centric modeling and processing for ubiquitous events using semantic capability models

Recent development in sensor networks and mobile computing is gaining increasing interest from enterprises. Sensor data services can provide fine-grained information of the physical world and feed the event processing systems where high level business logic is evaluated and coarse-grained business events are derived. However current event processing systems are not user-oriented and take considerable effort to configure and implement mainly because of the granularity mismatch. In this paper, we present a intuitive way to model the complex event patterns based on semantic descriptions of sensor service capabilities. Then, we transform the event patterns into stream queries to facilitate an automatic implementation for the event processing system.