Set Reconciliation Research Articles

Set Reconciliation Using Ternary and Invertible Bloom Filters

Set reconciliation between different hosts to hold the same dataset is an important prerequisite in numerous distributed applications. Sending the entire dataset to achieve set reconciliation is inefficient if the majority of the data owned by each host is the same. Each host is required to send exclusive elements uniquely included in its set to the other to minimize communication complexity. This paper proposes an efficient algorithm for a host to identify its exclusive elements using the recursive comparison of ternary Bloom filters, each representing the signature of a subset of elements and used for filtering out subsets with identical elements. Hence, subsets with exclusive elements are identified, and elements included in the identified subsets are programmed to an invertible Bloom filter (IBF) to be sent to the other host. Thus, the number of elements programmed in the IBF is significantly reduced. Simulation results show that the proposed algorithm provides excellent performance compared with existing set reconciliation algorithms under the constraint of the same amount of data communication.

A Rate-Compatible Solution to the Set Reconciliation Problem

A Rate-Compatible Solution to the Set Reconciliation Problem

GenSync: A New Framework for Benchmarking and Optimizing Reconciliation of Data

In the set reconciliation problem, remote parties seek to reconcile similar sets of data according to an efficiency objective, such as minimizing communication or computation. Though investigated for many individual distributed applications, this problem still lacks a holistic treatment, and this is the aim of this work. Specifically, we design and analyze GenSync, a unified set reconciliation framework that incorporates several state-of-the-art set reconciliation protocols with an integrated testbed. We compare and analyze the various protocols and offer general guidelines for selecting a good protocol for a given application. Through extensive experiments, we demonstrate that the optimal choice of protocol is highly sensitive to several parameters, including network properties (e.g., bandwidth and latency) and computing power. Notably, none of our framework’s protocols are universally dominant under diverse conditions, and a poor protocol choice may lead to a 5x hit in performance. To demonstrate our framework, we measure the effects of protocol choice in reconciling memory pools of adjacent Bitcoin nodes.

MCFsyn: A Multi-Party Set Reconciliation Protocol With the Marked Cuckoo Filter

Multi-party set reconciliation is a key component in distributed and networking systems. It naturally contains two dimensions, i.e., set representation and reconciliation protocol. However, existing sketch data structures are insufficient to satisfy the new needs brought by the multi-party scenario simultaneously, including space-efficiency, mergeability, and completeness. The current reconciliation protocols, on the other hand, fail to achieve the global optimization of communication cost. To this end, in this article, we propose the marked cuckoo filter (MCF), a data structure for representing set members. Grounded on MCF, we implement the MCFsyn protocol to reconcile multiple sets. MCFsyn aggregates and distributes sets information represented by MCFs along with an underlying minimum spanning tree among the participants. The participants then identify the different elements by traversing the overall MCF which contains the information of all elements in the union set. For the identified missing elements, MCFsyn helps the participants to choose the optimal senders to fetch with the minimum communication cost. Comprehensive evaluations indicate that MCFsyn significantly outperforms existing alternatives in terms of both reconciliation accuracy and communication cost.

Space- and computationally-efficient set reconciliation via parity bitmap sketch (PBS)

Set reconciliation is a fundamental algorithmic problem that arises in many networking, system, and database applications. In this problem, two large sets A and B of objects (bitcoins, files, records, etc.) are stored respectively at two different network-connected hosts, which we name Alice and Bob respectively. Alice and Bob communicate with each other to learn A Δ B , the difference between A and B , and as a result the reconciled set A ∪ B. Current set reconciliation schemes are based on either invertible Bloom filters (IBF) or error-correction codes (ECC). The former has a low computational complexity of O(d) , where d is the cardinality of A Δ B , but has a high communication overhead that is several times larger than the theoretical minimum. The latter has a low communication overhead close to the theoretical minimum, but has a much higher computational complexity of O(d 2 ). In this work, we propose Parity Bitmap Sketch (PBS), an ECC-based set reconciliation scheme that gets the better of both worlds: PBS has both a low computational complexity of O(d) just like IBF-based solutions and a low communication overhead of roughly twice the theoretical minimum. A separate contribution of this work is a novel rigorous analytical framework that can be used for the precise calculation of various performance metrics and for the near-optimal parameter tuning of PBS.

Near-accurate Multiset Reconciliation

The mission of set reconciliation (also called set synchronization) is to identify those elements which appear only in exactly one of two given sets. In this paper, we extend the set reconciliation problem into three design rationales: (i) multiset support; (ii) near 100 percent reconciliation accuracy; and (iii) communication-friendly and time-saving. These three rationales, if realized, will lead to unprecedented benefits for the set reconciliation paradigm. Generally, prior reconciliation methods are mainly designed for simple sets and thus remain inapplicable for multisets. Methods based on probabilistic data structures, e.g., the Counting Bloom Filter (CBF), support efficient representation, and multiplicity queries. Based on these probabilistic data structures, approximate multiset reconciliation can be enabled. However, they often cannot achieve a statisfying accuracy, due to potential hash collisions. The reconciliations enabled by logs or lists incur high time-complexity and communication overhead. Therefore, existing reconciliation methods, fail to realize the three rationales simultaneously. To this end, we redesign Trie and Fenwick Tree (FT), to near-accurately represent and reconcile two types of multisets that we refer to as unsorted and sorted multisets, respectively. Moreover, to further reduce the communication overhead during the reconciliation process, we design a partial transmission strategy when exchanging two Tries or FTs. Comprehensive evaluations are conducted to quantify the performance of our proposals. The trace-driven evaluations demonstrate that Trie and FT achieve near-accurate multiset reconciliation, with 4.31 and 2.96 times faster than the CBF-based method, respectively. The simulations based on synthetic datasets further indicate that our proposals outperform the CBF-based method in terms of accuracy and communication overhead at most time.

SKA-PS: Secure key agreement protocol using physiological signals

In this paper, we propose SKA-PS, a novel Secure Key Agreement protocol using Physiological Signals, for Body Area Networks (BANs). Our protocol generates symmetric cryptographic keys using the physiological parameters derived from the physiological signals of the users, such as electrocardiogram, photoplethysmogram and blood pressure. In our construction, we reduce the problem of secure key agreement into the problem of set reconciliation by representing the physiological parameter sequences generated from the physiological signals of the BAN users with appropriate sets. When properly selected parameters are applied, biosensors of the same BAN user can agree on symmetric cryptographic keys with remarkably high true match and low false match rates (as much as 100% and 0.46% for pairwise execution, and 100% and 0.26% for group execution, respectively), and low communication, computational and storage costs. We implemented our model in an embedded system, thus the results show real implementation outcomes. Moreover, we comparatively analyze the performance of SKA-PS with two existing bio-cryptographic key agreement protocols and show that SKA-PS outperforms both in all performance metrics.

Every Timestamp Counts: Accurate Tracking of Network Latencies Using Reconcilable Difference Aggregator

User-facing services deployed in data centers must respond quickly to user actions. The measurement of network latencies is of paramount importance. Recently, a new family of compact data structures has been proposed to estimate one-way latencies. In order to achieve scalability, these new methods rely on timestamp aggregation. Unfortunately, this approach suffers from serious accuracy problems in the presence of packet loss and reordering, given that a single lost or out-of-order packet may invalidate a huge number of aggregated samples. In this paper, we unify the problem to detect lost and reordered packets within the set reconciliation framework. Although the set reconciliation approach and the data structures for aggregating packet timestamps are previously known, the combination of these two principles is novel. We present a space-efficient synopsis called reconcilable difference aggregator (RDA). RDA maximizes the percentage of useful packets for latency measurement by mapping packets to multiple banks and repairing aggregated samples that have been damaged by lost and reordered packets. RDA simultaneously obtains the average and the standard deviation of the latency. We provide a formal guarantee of the performance and derive optimized parameters. We further design and implement a user-space passive latency measurement system that addresses practical issues of integrating RDA into the network stack. Our extensive evaluation shows that compared with existing methods, our approach improves the relative error of the average latency estimation in 10–15 orders of magnitude, and the relative error of the standard deviation in 0.5–6 orders of magnitude.

Simple multi-party set reconciliation

Many distributed cloud-based services use multiple loosely consistent replicas of user information to avoid the high overhead of more tightly coupled synchronization. Periodically, the information must be synchronized, or reconciled. One can place this problem in the theoretical framework of set reconciliation: two parties $$A_1$$ and $$A_2$$ each hold a set of keys, named $$S_1$$ and $$S_2$$ respectively, and the goal is for both parties to obtain $$S_1 \cup S_2$$ . Typically, set reconciliation is interesting algorithmically when sets are large but the set difference $$|S_1-S_2|+|S_2-S_1|$$ is small. In this setting the focus is on accomplishing reconciliation efficiently in terms of communication; ideally, the communication should depend on the size of the set difference, and not on the size of the sets. In this paper, we extend recent approaches using Invertible Bloom Lookup Tables (IBLTs) for set reconciliation to the multi-party setting. There are three or more parties $$A_1,A_2,\ldots ,A_n$$ holding sets of keys $$S_1,S_2,\ldots ,S_n$$ respectively, and the goal is for all parties to obtain $$\cup _i S_i$$ . While this could be done by pairwise reconciliations, we seek more effective methods. Our general approach can function even if the number of parties is not exactly known in advance, and with some additional cost can be used to determine which other parties hold missing keys. Our methodology uses network coding techniques in conjunction with IBLTs, allowing efficiency in network utilization along with efficiency obtained by passing messages of size $$O(|\cup _i S_i - \cap _i S_i|)$$ . By connecting reconciliation with network coding, we can provide efficient reconciliation methods for a number of natural distributed settings.

Byzantine set-union consensus using efficient set reconciliation

Applications of secure multiparty computation such as certain electronic voting or auction protocols require Byzantine agreement on large sets of elements. Implementations proposed in the literature so far have relied on state machine replication and reach agreement on each individual set element in sequence.We introduce set-union consensus, a specialization of Byzantine consensus that reaches agreement over whole sets. This primitive admits an efficient and simple implementation by the composition of Eppstein’s set reconciliation protocol with Ben-Or’s ByzConsensus protocol.A free software implementation of this construction is available in GNUnet. Experimental results indicate that our approach results in an efficient protocol for very large sets, especially in the absence of Byzantine faults. We show the versatility of set-union consensus by using it to implement distributed key generation, ballot collection, and cooperative decryption for an electronic voting protocol implemented in GNUnet.This is a revised and extended version of a paper published under the same title at ARES 2016.

Comparing set reconciliation methods based on bloom filters and their variants

Set reconciliation between two nodes is widely used in network applications.The basic idea is that each member of a node pair has an object set and seeks to deliver its unique objects to the other member.The Standard Bloom Filter(SBF)and its variants,such as the Invertible Bloom Filter(IBF),are effective approaches to solving the set reconciliation problem.The SBF-based method requires each node to represent its objects using an SBF,which is exchanged with the other node.A receiving node queries the received SBF against its local objects to identify the unique objects.Finally,each node exchanges its unique objects with the other node in the node pair.For the IBFbased method,each node represents its objects using an IBF,which is then exchanged.A receiving node subtracts the received IBF from its local IBF so as to decode the different objects between the two sets.Intuitively,it would seem that the IBF-based method,with only one round of communication,entails less communication overhead than the SBF-based method,which incurs two rounds of communication.Our research results,however,indicate that neither of these two methods has an absolute advantages over the others.In this paper,we aim to provide an in-depth understanding of the two methods,by evaluating and comparing their communication overhead.We find that the best method depends on parameter settings.We demonstrate that the SBF-based method outperforms the IBF-based method in most cases.But when the number of different objects in the two sets is below a certain threshold,the IBF-based method outperforms the SBF-based method.

Optimal Las Vegas reduction from one-way set reconciliation to error correction

Suppose we have two players A and C, where player A has a string s[0..u−1] and player C has a string t[0..u−1] and none of the two players knows the other's string.Assume that s and t are both over an integer alphabet [σ]=[0,σ−1], where the first string contains n non-zero entries. We would wish to answer the following basic question. Assuming that s and t differ in at most k positions, how many bits does player A need to send to player C so that he can recover s with certainty? Further, how much time does player A need to spend to compute the sent bits and how much time does player C need to recover the string s? This problem has a certain number of applications, for example in databases, where each of the two parties possesses a set of n key-value pairs, where keys are from the universe [u] and values are from [σ] and usually n≪u.In this paper, we show a time and message-size optimal Las Vegas reduction from this problem to the problem of systematic error correction of k errors for strings of length Θ(n) over an alphabet of size 2Θ(log⁡σ+log⁡(u/n)).The additional running time incurred by the reduction is linear expected (randomized) for player A and linear worst-case (deterministic) for player C, but the correction works with certainty. When using the popular Reed–Solomon codes, the reduction gives a protocol that transmits O(k(log⁡u+log⁡σ)) bits and runs in time O(n⋅polylog(n)(log⁡u+log⁡σ)) for all values of k. The time is expected for player A (encoding time) and worst-case for player C (decoding time). The message size is optimal whenever k≤(uσ)1−Ω(1).

Peer-to-peer mobile data flow in a crop field

The adoption of mobile technology in the agriculture sector can lead to benefits such as high productivity, improvement in mechanisation, increment in revenue, and timely information accessibility by farmers. The problem, however, is that mobile devices communicate over wireless mediums that can be unreliable; and when data states are kept on distributed nodes, inconsistencies can arise due to inefficiencies in the flow propagation of the updated agronomic data. This work describes a new key-value storage synchronisation workflow using exact set reconciliation and bloom filters. The proposed flow algorithm with its two-phase architect, with the first one being approximate synchronisation and the second one being exact synchronisation, provides better performance regarding bandwidth management. In addition, the usage of redundancy detection in the synchronisation mechanism makes it possible for the proposed algorithm to operate in a peer-to-peer environment, independent of any centralised server.

A broadcast-based key agreement scheme using set reconciliation for wireless body area networks.

Information and communication technologies have thrived over the last few years. Healthcare systems have also benefited from this progression. A wireless body area network (WBAN) consists of small, low-power sensors used to monitor human physiological values remotely, which enables physicians to remotely monitor the health of patients. Communication security in WBANs is essential because it involves human physiological data. Key agreement and authentication are the primary issues in the security of WBANs. To agree upon a common key, the nodes exchange information with each other using wireless communication. This information exchange process must be secure enough or the information exchange should be minimized to a certain level so that if information leak occurs, it does not affect the overall system. Most of the existing solutions for this problem exchange too much information for the sake of key agreement; getting this information is sufficient for an attacker to reproduce the key. Set reconciliation is a technique used to reconcile two similar sets held by two different hosts with minimal communication complexity. This paper presents a broadcast-based key agreement scheme using set reconciliation for secure communication in WBANs. The proposed scheme allows the neighboring nodes to agree upon a common key with the personal server (PS), generated from the electrocardiogram (EKG) feature set of the host body. Minimal information is exchanged in a broadcast manner, and even if every node is missing a different subset, by reconciling these feature sets, the whole network will still agree upon a single common key. Because of the limited information exchange, if an attacker gets the information in any way, he/she will not be able to reproduce the key. The proposed scheme mitigates replay, selective forwarding, and denial of service attacks using a challenge-response authentication mechanism. The simulation results show that the proposed scheme has a great deal of adoptability in terms of security, communication overhead, and running time complexity, as compared to the existing EKG-based key agreement scheme.

Set Reconciliation via Counting Bloom Filters

In this paper, we study the set reconciliation problem, in which each member of a node pair has a set of objects and seeks to deliver its unique objects to the other member. How could each node compute the set difference, however, is challenging in the set reconciliation problem. To address such an issue, we propose a lightweight but efficient method that only requires the pair of nodes to represent objects using a counting Bloom filter (CBF) of size $(O(d))$ and exchange with each other, where $(d)$ denotes the total size of the set differences. A receiving node then subtracts the received CBF from its local one via minus operation proposed in this paper. The resultant CBF can approximately represent the union of the set differences and thus the set difference to each node can be identified after querying the resultant CBF. In this paper, we propose a novel estimator through which each node can accurately estimate not only the value of $(d)$ but also the size of the set difference to each node. Such an estimation result can be used to optimize the parameter setting of the CBF to achieve less false positives and false negatives. Comprehensive analysis and evaluation demonstrates that our method is more efficient than prior BF-based methods in terms of achieving the same accuracy with less communication cost. Moreover, our reconciliating method needs no prior context logs and it is very useful in networking and distributed applications.

Finite Length Analysis on Listing Failure Probability of Invertible Bloom Lookup Tables

The Invertible Bloom Lookup Tables (IBLT) is a data structure which supports insertion, deletion, retrieval and listing operations of the key-value pair. The IBLT can be used to realize efficient set reconciliation for database synchronization. The most notable feature of the IBLT is the complete listing operation of the key-value pairs based on the algorithm similar to the peeling algorithm for low-density generator-matrix (LDGM) codes. In this paper, we will present a stopping set (SS) analysis for the IBLT which reveals finite length behaviors of the listing failure probability. The key of the analysis is enumeration of the number of stopping matrices of given size. We derived a novel recursive formula useful for computationally efficient enumeration. An upper bound on the listing failure probability based on the union bound accurately captures the error floor behaviors. It will be shown that, in the error floor region, the dominant SS have size 2. We propose a simple modification on hash functions, which are called SS avoiding hash functions, for preventing occurrences of the SS of size 2.

Efficiently repairing and measuring replica consistency in distributed databases

In a distributed database, maintaining large table replicas with frequent asynchronous insertions is a challenging problem that requires carefully managing a tradeoff between consistency and availability. With that motivation in mind, we propose efficient algorithms to repair and measure replica consistency. Specifically, we adapt, extend and optimize distributed set reconciliation algorithms to efficiently compute the symmetric difference between replicated tables in a distributed relational database. Our novel algorithms enable fast synchronization of replicas being updated with small sets of new records, measuring obsolence of replicas having many insertions and deciding when to update a replica, as each table replica is being continuously updated in an asynchronous manner. We first present an algorithm to repair and measure distributed consistency on a large table continuously updated with new records at several sites when the number of insertions is small. We then present a complementary algorithm that enables fast synchronization of a summarization table based on foreign keys when the number of insertions is large, but happening on a few foreign key values. From a distributed systems perspective, in the first algorithm the large table with data is reconciled, whereas in the second case, its summarization table is reconciled. Both distributed database algorithms have linear communication complexity and cubic time complexity in the size of the symmetric difference between the respective table replicas they work on. That is, they are effective when the network speed is smaller than CPU speed at each site. A performance experimental evaluation with synthetic and real databases shows our algorithms are faster than a previous state-of-the art algorithm as well as more efficient than transferring complete tables, assuming large replicated tables and sporadic asynchronous insertions.

What's the difference?

We describe a synopsis structure, the Difference Digest, that allows two nodes to compute the elements belonging to the set difference in a single round with communication overhead proportional to the size of the difference times the logarithm of the keyspace. While set reconciliation can be done efficiently using logs, logs require overhead for every update and scale poorly when multiple users are to be reconciled. By contrast, our abstraction assumes no prior context and is useful in networking and distributed systems applications such as trading blocks in a peer-to-peer network, and synchronizing link-state databases after a partition. Our basic set-reconciliation method has a similarity with the peeling algorithm used in Tornado codes [6], which is not surprising, as there is an intimate connection between set difference and coding. Beyond set reconciliation, an essential component in our Difference Digest is a new estimator for the size of the set difference that outperforms min-wise sketches [3] for small set differences. Our experiments show that the Difference Digest is more efficient than prior approaches such as Approximate Reconciliation Trees [5] and Characteristic Polynomial Interpolation [17]. We use Difference Digests to implement a generic KeyDiff service in Linux that runs over TCP and returns the sets of keys that differ between machines.

Restorative Justice in Transitional Sierra Leone

Intense debate surrounds truth commissions as to their mission, perceived roles and outcomes. This paper seeks to contribute to the understanding of truth commissions in post-conflict settings. It examines the Truth and Reconciliation Commission (TRC) for Sierra Leone, the first truth commission to be engaged concurrently with a retributive mechanism, the Special Court for Sierra Leone for transitional justice. The study finds that the TRC provided an opening for conversation in Sierra Leonean communities to search for the meanings of truth about the conflict. In this way the communities simultaneously created an understanding of the situation and set reconciliation directions and commitment from the process of creative conversation. This notwithstanding, the TRC did not have the needed public cooperation because the people were not sure the war was over and feared that their assailants could harm them if they disclosed the truth to the TRC. The presence of the Special Court also created tensions and fears rendering the transitional environment unfriendly to the reconciliation and truth telling endeavors of the TRC. The study has implications for future truth commissions in that the timing for post-conflict reconciliation endeavors should take into consideration the state of the peace equilibrium of the societies involved. It should also be packaged for harmonious existence in a given transitional contexts, particularly where it will coexist with a retributive mechanism.

BodySec

This paper proposes a new biometric-based key distribution scheme, BodySec, for energyefficient secure communications in wireless body sensor networks. To the best of our knowledge, BodySec is the first approach that can remove the timing synchronization assumption of the biometric inter-pulse signals among different biomedical sensors. The basic design is to represent the biometrics as an unordered set of short-term slots, then transform the problem of key agreement using biometrics into a problem of set reconciliation, finally ensure the different sensors can possess the same keys with low energy consumption. The experimental results on the PhysioBank electrocardiogram database demonstrate that BodySec opens up a new feasible way for energy-efficient key distribution in the future human-centered wireless system characterized by timing asynchronism and limited power.