Incremental Maintenance Research Articles

Incrementalizing lattice-based program analyses in Datalog

Program analyses detect errors in code, but when code changes frequently as in an IDE, repeated re-analysis from-scratch is unnecessary: It leads to poor performance unless we give up on precision and recall. Incremental program analysis promises to deliver fast feedback without giving up on precision or recall by deriving a new analysis result from the previous one. However, Datalog and other existing frameworks for incremental program analysis are limited in expressive power: They only support the powerset lattice as representation of analysis results, whereas many practically relevant analyses require custom lattices and aggregation over lattice values. To this end, we present a novel algorithm called DRedL that supports incremental maintenance of recursive lattice-value aggregation in Datalog. The key insight of DRedL is to dynamically recognize increasing replacements of old lattice values by new ones, which allows us to avoid the expensive deletion of the old value. We integrate DRedL into the analysis framework IncA and use IncA to realize incremental implementations of strong-update points-to analysis and string analysis for Java. As our performance evaluation demonstrates, both analyses react to code changes within milliseconds.

Adaptive correlation exploitation in big data query optimization

Correlations among the data attributes are abundant and inherent in most application domains. These correlations, if managed in systematic and efficient ways, would enable various optimization opportunities. Unfortunately, the state-of-art techniques are all heavily tailored toward optimizing factors intrinsic to relational databases, e.g., predicate selectivity, random I/O accesses, and secondary indexes, which are mostly not applicable to the modern big data infrastructures, e.g., Hadoop and Spark. In this paper, we propose the EXORD$$^+$$+ system for exploiting the data's correlations in big data query optimization. EXORD$$^+$$+ supports two types of correlations; hard (which does not allow for exceptions) and soft (which allows for exceptions). We introduce a three-phase approach for managing soft correlations including: (1) validating and judging the worthiness of soft correlations, (2) selecting and preparing the soft correlations for deployment, and (3) exploiting the correlations in query optimization. EXORD$$^+$$+ introduces a novel cost-benefit model for adaptively selecting the most beneficial soft correlations given a query workload. We show the complexity of this problem (NP-Hard) and propose a heuristic to efficiently solve it in a polynomial time. Moreover, we present incremental maintenance algorithms for efficiently updating the system's state under data appends and workload changes. EXORD$$^+$$+ prototype is implemented as an extension to the Hive engine on top of Hadoop. The experimental evaluation shows the potential of EXORD$$^+$$+ in achieving more than 10x speedup while introducing minimal storage overheads.

A Semantic Approach to Data Reduction for Weighted Graphs and Complex Queries

Graphs are widely used to store complex data nowadays: social networks, recommendation engines, computer networks and bioinformatics, to name a few. With a rapidly growing amount of data on the Internet in recent years, designing scalable systems to process the huge graph data efficiently has become a critical issue. In order to store and process the graph data efficiently in memory/disk, as well as to save time for transferring the data, graph compression techniques are often used. However, most of the existing graph data compression approaches are syntactic, which means they focus on graph structure and reduce it by serialization or redundancy removal. In this paper we focus on a semantic approach, namely query-based graph data reduction, which reduces a graph by preserving only the information relevant to the queries needed by an application. We study several classical graph problems and their applications, and design a suite of graph reduction algorithms to generate reduced graphs in which an application can still compute the same solutions. In addition, we design a synthesis method that can combine existing graph reduction algorithms to generate a reduced graph for a complex graph problem that includes more than one constraint. We also discuss incremental maintenance in order to update a reduced graph without reprocessing the whole graph again when the original graph is modified. We conduct experiments to compare the reduction rate of our algorithms with different sizes and types of data.

Incremental Maintenance of Maximal Bicliques in a Dynamic Bipartite Graph

We consider incremental maintenance of maximal bicliques from a dynamic bipartite graph that changes over time due to the addition of edges. When new edges are added to the graph, we seek to enumerate the change in the set of maximal bicliques, without enumerating the set of maximal bicliques that remain unaffected. The challenge in an efficient algorithm is to enumerate the change without explicitly enumerating the set of all maximal bicliques. In this work, we present (1) Near-tight bounds on the magnitude of change in the set of maximal bicliques of a graph, due to a change in the edge set, and an (2) Incremental algorithm for enumerating the change in the set of maximal bicliques. For the case when a constant number of edges are added to the graph, our algorithm is “change-sensitive”, i.e., its time complexity is proportional to the magnitude of change in the set of maximal bicliques. To our knowledge, this is the first incremental algorithm for enumerating maximal bicliques in a dynamic graph, with a provable performance guarantee. Our algorithm is easy to implement, and experimental results show that its performance exceeds that of baseline implementations by orders of magnitude.

Lighting retrofit and maintenance models with decay and adaptive control

In lighting retrofit projects, a lamp population is subject to decay, which results in significantly deteriorated energy efficiency (EE) and reduced cost saving. Incremental retrofit and maintenance are studied to overcome the decay in the population, so that EE performance can be sustained. Current models of natural decay cannot reflect the interactive dynamics of incremental retrofit and maintenance, so a new decay model is proposed for these interventions. Using a control approach, a multiple-input and multiple-output state equation is formulated. Adaptive control laws are designed to cope with unknown parameters of the proposed model, and to achieve stable performance improvement. This new model is verified, based on empirical data, and the results of adaptive control indicate that the number of working lamps can be maintained as a required value.

Mining Historical Changes to Predict Software Evolution

Software evolution reflects the progress volume of the development process. This increasing volume is based on a sequence of incremental changes and maintenance activities. As volume increases, more resources are needed to control and handle future change requests. Developers and designers need to be able to analyse and predict how the software evolves in advance, in order to better allocate needed resources. Better and correct predictions help in estimating the required resources as cost and maintainers. Prediction can also help in setting the strategies and assumptions to solve the problems that can be encountered when evolving a project from version to the next. This paper presents a prediction model to predict the evolution of software projects. A set of hypotheses are examined to predict the evolution of specific parameters based on machine learning techniques. Data mining approaches are used to test the hypothesis on different versions of two software projects. Experimental results showed that future changes in software systems can be predicted using changed parameters of the previous version.

The incremental Fourier classifier: Leveraging the discrete Fourier transform for classifying high speed data streams

Two major performance bottlenecks with decision tree based classifiers in a data stream environment are the depth of the tree and the update overhead of maintaining leaf node statistics on an instance-wise basis to ensure that classification is consistent with the current state of the data stream. Previous research has shown that classifiers based on Fourier spectra derived from decision trees produce compact array structures that can be searched and maintained much more efficiently than deep tree based structures. However, the key issue of incrementally adapting the spectrum to changes has not been addressed. In this research we present a strategy for incremental maintenance of the Fourier spectrum to changes in concept that take place in data stream environments. Along with the incremental approach we also propose schemes for feature selection and synopsis generation that enable the coefficient array to be refreshed efficiently on a periodic basis. Our empirical evaluation on a number of widely used stream classifiers reveals that the Fourier classifier outperforms them, both in terms of classification accuracy as well as speed of classification.

Incremental maintenance of all-pairs shortest paths in relational DBMSs

Computing shortest paths is a classical graph theory problem and a central task in many applications. Although many algorithms to solve this problem have been proposed over the years, they are designed to work in the main memory and/or with static graphs, which limits their applicability to many current applications where graphs are highly dynamic, that is, subject to frequent updates. In this paper, we propose novel efficient incremental algorithms for maintaining all-pairs shortest paths and distances in dynamic graphs. We experimentally evaluate our approach on several real-world datasets, showing that it significantly outperforms current algorithms designed for the same problem.

ViewDF: Declarative incremental view maintenance for streaming data

We present ViewDF: a flexible and declarative framework for incremental maintenance of materialized views (i.e., results of continuous queries) over streaming data. The main component of the proposed framework is the View Delta Function (ViewDF), which declaratively specifies how to update a materialized view when a new batch of data arrives. We describe and experimentally evaluate a prototype system based on this idea, which allows users to write ViewDFs directly and automatically translates common classes of streaming queries into ViewDFs. Our approach generalizes existing work on incremental view maintenance and enables new optimizations for views that are common in stream analytics, including those with pattern matching and sliding windows.

SAP: Improving Continuous Top-K Queries Over Streaming Data

Continuous top- $k$ query over streaming data is a fundamental problem in database. In this paper, we focus on the sliding window scenario, where a continuous top- $k$ query returns the top- $k$ objects within each query window on the data stream. Existing algorithms support this type of queries via incrementally maintaining a subset of objects in the window and try to retrieve the answer from this subset as much as possible whenever the window slides. However, since all the existing algorithms are sensitive to query parameters and data distribution, they all suffer from expensive incremental maintenance cost. In this paper, we propose a self-adaptive partition framework to support continuous top- $k$ query. It partitions the window into sub-windows and only maintains a small number of candidates with highest scores in each sub-window. Based on this framework, we have developed several partition algorithms to cater for different object distributions and query parameters. To our best knowledge, it is the first algorithm that achieves logarithmic complexity w.r.t. $k$ for incrementally maintaining the candidate set even in the worst case scenarios.

Incremental maintenance of discovered association rules and approximate dependencies

Incremental maintenance of discovered association rules and approximate dependencies

Approximate Continuous Top-k Query over Sliding Window

Continuous top-k query over sliding window is a fundamental problem in database, which retrieves k objects with the highest scores when the window slides. Existing studies mainly adopt exact algorithms to tackle this type of queries, whose key idea is to maintain a subset of objects in the window, and try to retrieve answers from it. However, all the existing algorithms are sensitive to query parameters and data distribution. In addition, they suffer from expensive overhead for incremental maintenance, and thus cannot satisfy real-time requirement. In this paper, we define a novel query named (e, δ)-approximate continuous top-k query, which returns approximate answers for top-k query. In order to efficiently support this query, we propose an efficient framework, named PABF (Probabilistic Approximate Based Framework), to support approximate top-k query over sliding window. We firstly maintain a self-adaptive pruning value, which could filter out newly arrived objects who have a probability less than 1 − δ of being a query result. For those objects that are not filtered, we combine them together, if the score difference among them is less than a threshold. To efficiently maintain these combined results, the framework PABF also proposes a multi-phase merging algorithm. Theoretical analysis indicates that even in the worst case, we require only logarithmic complexity for maintaining each candidate.

Incremental View Maintenance for Deductive Graph Databases Using Generalized Discrimination Networks

Nowadays, graph databases are employed when relationships between entities are in the scope of database queries to avoid performance-critical join operations of relational databases. Graph queries are used to query and modify graphs stored in graph databases. Graph queries employ graph pattern matching that is NP-complete for subgraph isomorphism. Graph database views can be employed that keep ready answers in terms of precalculated graph pattern matches for often stated and complex graph queries to increase query performance. However, such graph database views must be kept consistent with the graphs stored in the graph database. In this paper, we describe how to use incremental graph pattern matching as technique for maintaining graph database views. We present an incremental maintenance algorithm for graph database views, which works for imperatively and declaratively specified graph queries. The evaluation shows that our maintenance algorithm scales when the number of nodes and edges stored in the graph database increases. Furthermore, our evaluation shows that our approach can outperform existing approaches for the incremental maintenance of graph query results.

Multi‐objective transmission reinforcement planning approach for analysing future energy scenarios in the Great Britain network

A multi-objective transmission reinforcement planning framework has been designed to evaluate the effect of applying a future energy scenario to the Great Britain transmission network. This is achieved by examining the identified non-dominated set of transmission reinforcement plans, which alleviate thermal capacity constraints, for the multi-criteria problem of five objectives: investment cost, annual constraint cost saving, annual incremental operation and maintenance cost, outage cost and annual line loss saving. The framework is flexible and utilises a systematic algorithm to generate reinforcement plans and alter the associated reinforcements should they exacerbate thermal constraints; hence a pre-determined set of reinforcements is not required to evaluate a scenario. The reinforcements considered are line addition (single circuit and double circuit) and line upgrading through reconductoring. The Strength Pareto Evolutionary Algorithm 2 is utilised to explore varying locations, configurations and capacities of network reinforcement. The solutions produced achieve similar cost savings to solutions created by the transmission network owners, showing the suitability of the approach to provide a useful trade-off analysis of the objectives and to assess the network-related thermal and economic impact of future energy scenarios. Here, the framework is applied to the 2020 generation mix of the Gone Green scenario developed by National Grid.

Efficient community identification and maintenance at multiple resolutions on distributed datastores

The topic of network community identification at multiple resolutions is of great interest in practice to learn high cohesive subnetworks about different subjects in a network. For instance, one might examine the interconnections among web pages, blogs and social content to identify pockets of influencers on subjects like ‘Big Data’, ‘smart phone’ or ‘global warming’. With dynamic changes to its graph representation and content, the incremental maintenance of a community poses significant challenges in computation. Moreover, the intensity of community engagement can be distinguished at multiple levels, resulting in a multi-resolution community representation that has to be maintained over time. In this paper, we first formalize this problem using the k-core metric projected at multiple k-values, so that multiple community resolutions are represented with multiple k-core graphs. Recognizing that large graphs and their even larger attributed content cannot be stored and managed by a single server, we then propose distributed algorithms to construct and maintain a multi-k-core graph, implemented on the scalable Big Data platform Apache HBase. Our experimental evaluation results demonstrate orders of magnitude speedup by maintaining multi-k-core incrementally over complete reconstruction. Our algorithms thus enable practitioners to create and maintain communities at multiple resolutions on multiple subjects in rich network content simultaneously.

Application and Research of Data Interface Synchronization on Web Maintenance

To adapt to the changing requirement of task data interface under the situation of far distance, multiple segment, multiple circle, multiple satellite and multi-station visibility for satellite misson in transfer orbit segment, the web incremental maintenance system based on materialized view was achieved through applying incremental maintenance principle, database technology, synchronization mechanism and maintenance proxy, and realizing the synchronization and consistency of data interface about the distributed experiment information surveillance software system. The result shows that web incremental maintenance system can ensure the real-time and consistency of data processing and transmission.

DBToaster: higher-order delta processing for dynamic, frequently fresh views

Applications ranging from algorithmic trading to scientific data analysis require real-time analytics based on views over databases receiving thousands of updates each second. Such views have to be kept fresh at millisecond latencies. At the same time, these views have to support classical SQL, rather than window semantics, to enable applications that combine current with aged or historical data. In this article, we present the DBToaster system, which keeps materialized views of standard SQL queries continuously fresh as data changes very rapidly. This is achieved by a combination of aggressive compilation techniques and DBToaster’s original recursive finite differencing technique which materializes a query and a set of its higher-order deltas as views. These views support each other’s incremental maintenance, leading to a reduced overall view maintenance cost. DBToaster supports tens of thousands of complete view refreshes per second for a wide range of queries.

Incremental Maintenance of Rough Fuzzy Set Approximations under the Variation of Object Set

The lower and upper approximations in rough set theory will change dynamically over time due to the variation of the information system. Incremental methods for updating approximations in rough set theory and its extensions have received much attention recently. Most existing incremental methods have difficulties in dealing with fuzzy decision systems which decision attributes are fuzzy. This paper introduces an incremental algorithm for updating approximations of rough fuzzy sets under the variation of the object set in fuzzy decision systems. In experiments on 6 data sets from UCI, comparisons of the incremental and non-incremental methods for updating approximations are conducted. The experimental results show that the incremental method effectively reduces the computational time.

Incremental maintenance of discovered spatial association rules

Executing spatial association rule mining repeatedly is often necessary to get interesting and effective rules.Though incremental maintenance algorithms can be introduced to improve the efficiency of association rule mining, currently there exists no such algorithm that can use spatial datasets directly. To solve this problem, the update strategy of the discovered rules was discussed. Both threshold changes and spatial datasets updates were taken into consideration, and an incremental mining algorithm called Incremental Spatial Apriori( ISA) was suggested. ISA algorithm aimed to update frequent predicate sets and association rules after the minimum support threshold decreased or new spatial layers added. This algorithm did not rely on the creation and update of spatial transaction tables; it directly used spatial layers as input data. In experiments with real-world data, the mining result extracted by ISA and Apriori-like algorithms are identical, but ISA can save 20. 0% to71. 0% time comparatively. Besides, 1 372 722 rules were successfully updated with the filtering method, costing less than0. 1 seconds. These results indicate the incremental update strategy and algorithm for spatial association rules suggested in this paper are correct, efficient and applicable.

Incremental Maintenance of the Minimum Bisimulation of Cyclic Graphs

There have been numerous recent applications of graph databases (e.g., the Semantic Web, ontology representation, social networks, XML, chemical databases, and biological databases). A fundamental structural index for data graphs, namely minimum bisimulation, has been reported useful for efficient path query processing and optimization including selectivity estimation, among many others. Data graphs are subject to change and their indexes are updated accordingly. This paper studies the incremental maintenance problem of the minimum bisimulation of a possibly cyclic data graph. While cyclic graphs are ubiquitous among the data on the web, previous work on the maintenance problem has mostly focused on acyclic graphs. To study the problem with cyclic graphs, we first show that the two existing classes of minimization algorithms - merging algorithm and partition refinement - have their strengths and weaknesses. Second, we propose a novel hybrid algorithm and its analytical model. This algorithm supports an edge insertion or deletion and two forms of batch insertions or deletions. To the best of our knowledge, this is the first maintenance algorithm that guarantees minimum bisimulation of cyclic graphs. Third, we propose to partially reuse the minimum bisimulation before an update in order to optimize maintenance performance. We present an experimental study on both synthetic and real-data graphs that verified the efficiency and effectiveness of our algorithms.