Consistency Guarantees Research Articles

Distributed Caching System with Strict Consistency Guarantee for Critical Information Infrastructure

The place of caching in modern information systems is considered. Typical models of network applications architecture construction and consistency guarantees that they provide to the end user are presented. A comparative analysis of analogues is given, which allows one to outline the features of the proposed system. The developed architecture is described. The architecture of a separate cluster node is described, as well as the architecture and implementation of software development kits for different programming languages. The specifics of designing complex client libra­ries for distributed systems are considered. The importance of consistency models is noted, their interrelationships and hierarchy are described based on the summarized information of existing research. A brief comparison of the two popular consensus algorithms, Raft and Paxos, is given in order to determine the basics of designing a common architecture. Industry leaders in caching systems emphasize performance, deliberately relaxing the consistency of the stored data. Using such systems in an environment that requires maintaining the order of events in the external world is difficult or involves the need for additional tools to maintain data consistency. This paper proposes a caching system that guarantees strict data consistency and provides a single interface for handling it, similar to industry leaders.

PALF: Replicated Write-Ahead Logging for Distributed Databases

Distributed databases have been widely researched and developed in recent years due to their scalability, availability, and consistency guarantees. The write-ahead logging (WAL) system is one of the most vital components in a database. It is still a non-trivial problem to design a replicated logging system as the foundation of a distributed database with the power of ACID transactions. This paper proposes PALF, a Paxos-backed Append-only Log File System, to address these challenges. The basic idea behind PALF is to co-design the logging system with the entire database for supporting database-specific functions and to abstract the functions as PALF primitives to power other distributed systems. Many database functions, including transaction processing, database restore, and physical standby databases, have been built based on PALF primitives. Evaluation shows that PALF greatly outperforms well-known implementations of consensus protocols and is fully competent for distributed database workloads. PALF has been deployed as a component of the OceanBase 4.0 database and has been made open-source along with it.

Transparent Migration from Datastore to Firestore

Datastore was one of Google's first cloud databases, launched initially as part of App Engine, and built over Google's internal Megastore database system. Firestore was launched in 2019, both a re-implementation of Datastore over Google's Spanner database system and a new, mobile and web-friendly Firestore API. Spanner was chosen as the storage engine of Firestore in particular for technical reasons---it provides unrestricted transaction capabilities, strong consistency guarantees, and other improvements over Megastore. To provide these improvements to all our customers, and simplify our overall system, a non-disruptive, zero-downtime migration was executed of all Datastore databases (stored in Megastore) to Firestore databases (stored in Spanner). This migration took a couple of years to design and plan, and about three to execute. This paper describes both the core engine for migrating databases, and various practical problems that were solved to make this journey successful. As of the writing of this paper, all (over one million) databases have been successfully migrated.

Cross-organizational data exchange based on consortium blockchain with consistency guarantee

Cross-organizational data exchange based on consortium blockchain with consistency guarantee

CEIU: Consistent and Efficient Incremental Update mechanism for mobile systems on flash storage

The ever-growing sizes and frequent updating of mobile applications cause high network and storage cost for updating. Hence, emerging mobile systems often employ incremental update algorithms, typically HDiffPatch, to upgrade mobile applications. However, we find that existing incremental update algorithms not only generate a significant amount of redundant data accesses, but also lacks of consistency guarantees for the whole package of application. In this paper, we present a novel Consistent and Efficient Incremental Update (CEIU) mechanism for upgrading mobile applications. Firstly, CEIU reduces the memory consumption and file access of incremental updates by reusing the indexes of blocks in the old image rather than copying the blocks. Secondly, CEIU employs a two-level journaling mechanism to ensure the consistency of the whole package and subfiles of the new image. We implement the proposed mechanism in Linux kernel based on TI-LFAT file system and evaluate it with real-world applications. The experimental results show that the proposed mechanism can reduce 30%–80% memory footprints in comparison with HDiffPatch, the state-of-the-art incremental update algorithm. It also significantly reduces the recovery time when power failure or system crash occurs.

Learning the Causal Structure of Networked Dynamical Systems under Latent Nodes and Structured Noise

This paper considers learning the hidden causal network of a linear networked dynamical system (NDS) from the time series data at some of its nodes -- partial observability. The dynamics of the NDS are driven by colored noise that generates spurious associations across pairs of nodes, rendering the problem much harder. To address the challenge of noise correlation and partial observability, we assign to each pair of nodes a feature vector computed from the time series data of observed nodes. The feature embedding is engineered to yield structural consistency: there exists an affine hyperplane that consistently partitions the set of features, separating the feature vectors corresponding to connected pairs of nodes from those corresponding to disconnected pairs. The causal inference problem is thus addressed via clustering the designed features. We demonstrate with simple baseline supervised methods the competitive performance of the proposed causal inference mechanism under broad connectivity regimes and noise correlation levels, including a real world network. Further, we devise novel technical guarantees of structural consistency for linear NDS under the considered regime.

Semantic Map-Based Visual Localization With Consistency Guarantee

Semantic Map-Based Visual Localization With Consistency Guarantee

TVDO: Tchebycheff Value-Decomposition Optimization for Multiagent Reinforcement Learning.

In cooperative multiagent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) has recently attracted more attention due to the physical demand. However, the most dilemma therein is the inconsistency between jointly-trained policies and individually executed actions. In this article, we propose a factorized Tchebycheff value-decomposition optimization (TVDO) method to overcome the trouble of inconsistency. In particular, a nonlinear Tchebycheff aggregation function is formulated to realize the global optimum by tightly constraining the upper bound of individual action-value bias, which is inspired by the Tchebycheff method of multiobjective optimization (MOO). We theoretically prove that, under no extra limitations, the factorized value decomposition with Tchebycheff aggregation satisfies the sufficiency and necessity of individual-global-max (IGM), which guarantees the consistency between the global and individual optimal action-value function. Empirically, in the climb and penalty game, we verify that TVDO precisely expresses the global-to-individual value decomposition with a guarantee of policy consistency. Meanwhile, we evaluate TVDO in the StarCraft multiagent challenge (SMAC) benchmark, and extensive experiments demonstrate that TVDO achieves a significant performance superiority over some SOTA MARL baselines.

Strong consistency guarantees for clustering high-dimensional bipartite graphs with the spectral method

Strong consistency guarantees for clustering high-dimensional bipartite graphs with the spectral method

Noisy Label Learning With Provable Consistency for a Wider Family of Losses.

Deep models have achieved state-of-the-art performance on a broad range of visual recognition tasks. Nevertheless, the generalization ability of deep models is seriously affected by noisy labels. Though deep learning packages have different losses, this is not transparent for users to choose consistent losses. This paper addresses the problem of how to use abundant loss functions designed for the traditional classification problem in the presence of label noise. We present a dynamic label learning (DLL) algorithm for noisy label learning and then prove that any surrogate loss function can be used for classification with noisy labels by using our proposed algorithm, with a consistency guarantee that the label noise does not ultimately hinder the search for the optimal classifier of the noise-free sample. In addition, we provide a depth theoretical analysis of our algorithm to verify the justifies' correctness and explain the powerful robustness. Finally, experimental results on synthetic and real datasets confirm the efficiency of our algorithm and the correctness of our justifies and show that our proposed algorithm significantly outperforms or is comparable to current state-of-the-art counterparts.

VMIFresh: Efficient and fresh caches for virtual machine introspection

Virtual machine introspection (VMI) is the process of extracting knowledge about the inner state of a virtual machine from the outside. Traditional passive introspection mechanisms have proved themselves ineffective in many application domains due to their low performance. As a remedy for this issue, caching at the level of the introspection application was introduced. However, this sacrificed the freshness of VMI and led to an inconsistent outside view.In this work, we propose a multi-purpose hybrid caching scheme with freshness and consistency guarantees that is interleaved with the guest's MMU. This scheme can easily be integrated into existing applications and frameworks such as libvmi and Volatility 3. We demonstrate its feasibility by developing a prototype for such applications. Furthermore, the experimental evaluation of our approach suggests that it even significantly exceeds the performance of previous inconsistent caches.

Blockchain-Based Distributed Computing Consistency Verification for IoT Mobile Applications

The maturation of wireless connectivity, blockchain (distributed ledger technologies), and intelligent systems has fostered a comprehensive ecosystem for the Internet of Things (IoT). However, the growing volume of data generated by IoT devices creates substantial pressure on blockchain storage and computation capabilities, impeding the further development of the IoT ecosystem. Decentralizing data storage across multiple chains and utilizing cross-chain technology for data exchange eliminates the need for expensive centralized infrastructure, lowers data transfer costs, and improves accessibility. Hence, the issue of computational and storage pressure in blockchain can be improved. Nonetheless, the data of IoT devices are constantly updating, and ensuring consistency for dynamic data across heterogeneous chains remains a significant challenge. To address the aforementioned challenge, we propose a blockchain-based distributed and lightweight data consistency verification model (BDCA), which leverages a batch verification dynamic Merkle hash tree (BV-MHT) and an advanced gamma multi-signature scheme (AGMS) to enable consistent verification of dynamic data while ensuring secure and private data transmission. The AGMS scheme is reliable and robust based on security analysis while the dependability and consistency of BDCA are verified through inductive reasoning. Experimental results indicate that BDCA outperforms CPVPA and Fortress in communication and computation overhead for data preprocessing and auditing in a similar condition, and the AGMS scheme exhibits superior performance when compared to other widely adopted multi-signature schemes such as Cosi, BLS, and RSA. Furthermore, BDCA provides up to 99% data consistency guarantees, demonstrating its practicality.

Universe Points Representation Learning for Partial Multi-Graph Matching

Many challenges from natural world can be formulated as a graph matching problem. Previous deep learning-based methods mainly consider a full two-graph matching setting. In this work, we study the more general partial matching problem with multi-graph cycle consistency guarantees. Building on a recent progress in deep learning on graphs, we propose a novel data-driven method (URL) for partial multi-graph matching, which uses an object-to-universe formulation and learns latent representations of abstract universe points. The proposed approach advances the state of the art in semantic keypoint matching problem, evaluated on Pascal VOC, CUB, and Willow datasets. Moreover, the set of controlled experiments on a synthetic graph matching dataset demonstrates the scalability of our method to graphs with large number of nodes and its robustness to high partiality.

A novel spatio-temporal clustering algorithm with applications on COVID-19 data from the United States

A new clustering algorithm for spatio-temporal data is developed. The proposed method leverages a weighted combination of a spatial haversine distance matrix and a spectral-density based temporal distance matrix between the locations. Concepts of partition around medoids algorithm and the gap statistic are utilized to develop the algorithm and to determine the optimal number of clusters. Such a non-parametric algorithm is novel as it incorporates both spatial and temporal distances of the units and it can work for time-series of possibly different lengths. Theoretical guarantee of consistency of the proposed method is provided. An elaborate simulation study is also given to demonstrate the efficacy of the algorithm. As an interesting real life application, the proposed algorithm is implemented to analyze the spatio-temporal dynamics of the time series of coronavirus (COVID-19) incidence rates observed at county-level in the United States of America. The results are demonstrated on datasets of different sizes: the entire country, the Midwest region and the state of California. Special emphasis is given on the last two cases to display how the clustering results offer interesting insights into the epidemic progression in these areas. Particularly, it sheds light on whether state-mandated restrictions impacted the entire state similarly or if there are interesting local behaviors in terms of the COVID-19 spread.

One-shot Garbage Collection for In-memory OLTP through Temporality-aware Version Storage

Most modern in-memory online transaction processing (OLTP) engines rely on multi-version concurrency control (MVCC) to provide data consistency guarantees in the presence of conflicting data accesses. MVCC improves concurrency by generating a new version of a record on every write, thus increasing the storage requirements. Existing approaches rely on garbage collection and chain consolidation to reduce the length of version chains and reclaim space by freeing unreachable versions. However, finding unreachable versions requires the traversal of long version chains, which incurs random accesses right into the critical path of transaction execution, hence limiting scalability. This paper introduces OneShotGC, a new multi-version storage design that eliminates version traversal during garbage collection, with minimal discovery and memory management overheads. OneShotGC leverages the temporal correlations across versions to opportunistically cluster them into contiguous memory blocks that can be released in one shot. We implement OneShotGC in Proteus and use YCSB and TPC-C to experimentally evaluate its performance with respect to the state-of-the-art, where we observe an improvement of up to 2x in transactional throughput.

A novel framework to improve the consistency of water quality attribution from natural and anthropogenic factors

One critical question for water security and sustainable development is how water quality responses to the changes in natural factors and human activities, especially in light of the expected exacerbation in water scarcity. Although machine learning models have shown noticeable advances in water quality attribution analysis, they have limited interpretability in explaining the feature importance with theoretical guarantees of consistency. To fill this gap, this study built a modelling framework that employed the inverse distance weighting method and the extreme gradient boosting model to simulate the water quality at grid scale, and adapted the Shapley additive explanation to interpret the contributions of the drivers to water quality over the Yangtze River basin. Different from previous studies, we calculated the contribution of features to water quality at each grid within river basin and aggregated the contribution from all the grids as the feature importance. Our analysis revealed dramatic changes in response magnitudes of water quality to drivers within river basin. Air temperature had high importance in the variability of key water quality indicators (i.e. ammonia-nitrogen, total phosphorus, and chemical oxygen demand), and dominated the changes of water quality in Yangtze River basin, especially in the upstream region. In the mid- and downstream regions, water quality was mainly affected by human activities. This study provided a modelling framework applicable to robustly identify the feature importance by explaining the contribution of features to water quality at each grid.

On the correctness of highly available systems in the presence of failures

In this paper we formally study the guarantees provided by highly available, eventually consistent replicated systems in an environment in which machine failures and network splits are possible. Our analysis accounts for possible replica recovery after a crash, and clients that are (1) stateless or stateful, (2) sticky (always connect to a concrete set of replicas) or mobile, and (3) which can timeout before receiving a response to the sent request. We show why the approaches to prove protocol correctness prevalent in the literature, which do not take into account replica or network crashes, may lead to incorrect conclusions regarding the guarantees offered by the protocol. We adapt the existing formal correctness criteria, such as basic eventual consistency, to the considered environment by defining the family of failure-aware consistency guarantees. We formally identify a set of undesired phenomena (in particular phantom operations) observed by the clients, which, as we prove, are unavoidable in highly available systems in which unrecoverable replica crashes are possible. We also introduce context preservation, a new client-side requirement for eventually consistent systems that expose concurrency to the client, i.e., allow clients to use, e.g., multi-value registers or observed-remove sets. Context preservation is incomparable with classic session guarantees.

Detecting approximate replicate components of a high-dimensional random vector with latent structure

High-dimensional feature vectors are likely to contain sets of measurements that are approximate replicates of one another. In complex applications, or automated data collection, these feature sets are not known a priori, and need to be determined. This work proposes a class of latent factor models on the observed, high-dimensional, random vector X∈Rp, for defining, identifying and estimating the index set of its approximately replicate components. The model class is parametrized by a p×K loading matrix A that contains a hidden sub-matrix whose rows can be partitioned into groups of parallel vectors. Under this model class, a set of approximate replicate components of X corresponds to a set of parallel rows in A: these entries of X are, up to scale and additive error, the same linear combination of the K latent factors; the value of K is itself unknown. The problem of finding approximate replicates in X reduces to identifying, and estimating, the location of the hidden sub-matrix within A, and of the partition H of its row index set H. Both H and H can be fully characterized in terms of a new family of criteria based on the correlation matrix of X, and their identifiability, as well as that of the unknown latent dimension K, are obtained as consequences. The constructive nature of the identifiability arguments enables computationally efficient procedures, with consistency guarantees. Furthermore, when the loading matrix A has a particular sparse structure, provided by the errors-in-variable parametrization, the difficulty of the problem is elevated. The task becomes that of separating out groups of parallel rows that are proportional to canonical basis vectors from other, possibly dense, parallel rows in A. This is met under a scale assumption, via a principled way of selecting the target row indices, guided by the successive maximization of Schur complements of appropriate covariance matrices. The resulting procedure is an enhanced version of that developed for recovering general parallel rows in A. It is also computationally efficient, consistent. It has immediate applications to latent space overlapping clustering and the estimation of loading matrices that satisfy a canonical parametrization.

Cost-Effective Strong Consistency on Scalable Geo-Diverse Data Replicas

The Raft algorithm maintains strong consistency across data replicas in Cloud. This algorithm places nodes, i.e., leader and follower, to serve read/write requests spanning geo-diverse sites. As the workload increases, Raft shall provide proportional scale-out performance. However, traditional scale-out techniques are bottlenecked in Raft with an exponentially increased performance penalty when provisioned sites exhaust local resources. To provide scalability in Raft, this paper presents a cost-effective mechanism that enables elastic auto scaling in Raft, called BW-Raft. BW-Raft extends the original Raft with the following abstractions: (1) secretary nodes that take over expensive log synchronization operations from the leader, relaxing the performance constraint on locks. (2) observer nodes that handle reads only, improving throughput for typical data intensive services. These abstractions are stateless, allowing elastic scale-out on unreliable yet cheap spot instances. In theory, we prove that BW-Raft can preserve the strong consistency guarantee from Raft at scale-out, handling 50X more nodes, compared to the original Raft. We have prototyped the BW-Raft on key-value services and evaluated it with many state-of-the-arts on Amazon EC2 and Alibaba Cloud. Our results show that within the same budget, BW-Raft incurs 5-7X less resource footprint increment than Multi-Raft. Using spot instances, BW-Raft can reduces costs by 84.5%, compared to Multi-Raft. In the real world experiments, BW-Raft improves goodput of the 95th-percentile SLO by 9X, thus, serves an alternative for distributed service scaling out with strong consistency.

NimbleChain: Speeding up Cryptocurrencies in General-purpose Permissionless Blockchains

Nakamoto’s seminal work gave rise to permissionless blockchains, as well as a wide range of proposals to mitigate their performance shortcomings. Despite substantial throughput and energy efficiency achievements, most proposals only bring modest (or marginal) gains in transaction commit latency. Consequently, commit latencies in today’s permissionless blockchain landscape remain prohibitively high. This article proposes NimbleChain, a novel algorithm that extends permissionless blockchains based on Nakamoto consensus with a fast path that delivers causal promises of commitment or simply promises . Since promises only partially order transactions, their latency is only a small fraction of the totally ordered commitment latency of Nakamoto consensus. Still, the weak consistency guarantees of promises are strong enough to correctly implement cryptocurrencies. To the best of our knowledge, NimbleChain is the first system to bring together fast, partially ordered transactions with consensus-based, totally ordered transactions in a permissionless setting. This hybrid consistency model is able to speed up cryptocurrency transactions while still supporting smart contracts, which typically have (strong) sequential consistency needs. We implement NimbleChain as an extension of Ethereum and evaluate it in a 500-node geo-distributed deployment. The results show NimbleChain can promise a cryptocurrency transactions up to an order of magnitude faster than a vanilla Ethereum implementation, with marginal overheads.