Delta Compression Research Articles

Delta compression correction method for covert communication in IoT

Delta compression correction method for covert communication in IoT

The Design of Fast Delta Encoding for Delta Compression Based Storage Systems

Delta encoding is a data reduction technique capable of calculating the differences (i.e., delta) among very similar files and chunks. It is widely used for various applications, such as synchronization replication, backup/archival storage, cache compression, and so on. However, delta encoding is computationally costly due to its time-consuming word-matching operations for delta calculation. Existing delta encoding approaches either run at a slow encoding speed, such as Xdelta and Zdelta, or at a low compression ratio, such as Ddelta and Edelta. In this article, we propose Gdelta, a fast delta encoding approach with a high compression ratio. The key idea behind Gdelta is the combined use of five techniques: (1) employing an improved Gear-based rolling hash to replace Adler32 hash for fast scanning overlapping words of similar chunks, (2) adopting a quick array-based indexing for word-matching, (3) applying a sampling indexing scheme to reduce the cost of traditional building full indexes for base chunks’ words, (4) skipping unmatched words to accelerate delta encoding through non-redundant areas, and (5) last but not least, after word-matching, further batch compressing the remainder to improve the compression ratio. Our evaluation results driven by seven real-world datasets suggest that Gdelta achieves encoding/decoding speedups of 3.5X∼25X over the classic Xdelta and Zdelta approaches while increasing the compression ratio by about 10%∼240%.

LeCo: Lightweight Compression via Learning Serial Correlations

Lightweight data compression is a key technique that allows column stores to exhibit superior performance for analytical queries. Despite a comprehensive study on dictionary-based encodings to approach Shannon's entropy, few prior works have systematically exploited the serial correlation in a column for compression. In this paper, we propose LeCo (i.e., Learned Compression), a framework that uses machine learning to remove the serial redundancy in a value sequence automatically to achieve an outstanding compression ratio and decompression performance. LeCo presents a general approach to this end, making existing algorithms such as Frame-of-Reference (FOR), Delta Encoding, and Run-Length Encoding (RLE) special cases under our framework. Our microbenchmark with three synthetic and eight real-world data sets shows that a prototype of LeCo achieves a Pareto improvement on both compression ratio and random access speed over the existing solutions. When integrating LeCo into widely-used applications, we observe up to 5.2× speed up in a data analytical query in the Arrow columnar execution engine, and a 16% increase in RocksDB's throughput.

A method for assessing the characteristics of the virtual machine migration process, taking into account the type of migrations and applications

Dynamic resource allocation in cloud computing is a critical aspect. It is necessary for computing resources scaling to maintain application performance, reduce costs and ensure the reliability of IT systems. Dynamic resource allocation is possible using open virtual machines (VM), but live migration is a relatively expensive and resource-intensive operation. A number of VM migration algorithms have been proposed, each with different performance characteristics depending on the state of the host system, network, and the VM workload. Recently, a large number of works have emerged that have developed models for assessing the effectiveness of of VM migration, but many of them do not provide a standard of satisfactory prediction accuracy and are built for a single migration algorithm, which limits the use of data models. This work uses a statistical approach to estimate the total migration time and VM downtime based on the approximation of probability densities by Gram-Charlier and Laguerre Series. The proposed method, in comparison with the known ones, allows one to answer the question of what is the probability of total migration time and VM downtime for a given migration type and VM application. The analysis of total migration time and VM downtime is based on the Virtual Machine Live Migration Dataset includes more than 40 000 virtual machine migration records with five different live migration algorithms: post-copy migration, pre-copy migration and its modifications: CPU throttling delta compression, and data compression. The dataset includes approximately 8000 migration records of each migration algorithm with 9 types of workloads. Analysis of the results allows us to conclude that the type of application significantly influences both the shape of the empirical distribution (histograms) of total migration time and its characteristics, therefore it is advisable to obtain an analytical expression for the distribution law of the total migration time and VM downtime taking into account these circumstances. The results of the experiments shows that this method does not depend on the migration algorithm and the type of working application. The use of the Laguerre series can be recommended as giving more reliable results compared to Gram-Charlier series. A method for estimating total migration time and VM downtime can be integrated into a management system to select the best monitoring window for servers and evaluate service-level agreement terms.

A fragmentation-aware redundancy elimination scheme for inline backup systems

Data deduplication is a widely employed technique in backup systems to enhance storage efficiency by eliminating duplicate chunks. Delta compression is a technique that complements deduplication by removing redundant data between similar chunks. However, when integrated into deduplication-based backup systems, delta compression can considerably decrease backup throughput due to the additional I/Os for fetching base chunks. Moreover, data deduplication can lead to chunk fragmentation, causing continuous chunks in a backup to become fragmented. We observe that the number of additional I/Os for fetching base chunks when delta compression is applied depends on the degree of chunk fragmentation. When chunk fragmentation is not significant, delta compression can improve storage efficiency without significantly impacting backup throughput. Based on this observation, we propose a redundancy elimination approach called Fragmentation-aware Redundancy Elimination (FaRE) for in-line backup systems.The main idea behind FaRE is the combined use of three techniques: fragmentation estimation for assessing the degree of chunk fragmentation, sequential deduplication to significantly reduce chunk fragmentation by only deduplicating against chunks with sequential layout when fragmentation is severe, and local redundancy elimination to greatly enhance storage efficiency by performing both deduplication and delta compression when chunk fragmentation is not significant. Our evaluation results demonstrate that FaRE achieves higher storage efficiency and restoration performance compared to traditional approaches that only adopt deduplication for redundancy elimination, while achieving comparable backup throughput due to the limited number of additional I/Os required.

Chunk2vec: A novel resemblance detection scheme based on Sentence‐BERT for post‐deduplication delta compression in network transmission

AbstractDelta compression, as a complementary technique for data deduplication, has gained widespread attention in network storage systems. It can eliminate redundant data between non‐duplicate but similar chunks that cannot be identified by data deduplication. The network transmission overhead between servers and clients can be greatly reduced by using data deduplication and delta compression techniques. Resemblance detection is a technique that identifies similar chunks for post‐deduplication delta compression in network storage systems. The existing resemblance detection approaches fail to detect similar chunks with arbitrary similarity by setting a similarity threshold, which can be suboptimal. In this paper, the authors propose Chunk2vec, a resemblance detection scheme for delta compression that utilizes deep learning techniques and Approximate Nearest Neighbour Search technique to detect similar chunks with any given similarity range. Chunk2vec uses a deep neural network, Sentence‐BERT, to extract an approximate feature vector for each chunk while preserving its similarity with other chunks. The experimental results on five real‐world datasets indicate that Chunk2vec improves the accuracy of resemblance detection for delta compression and achieves higher compression ratio than the state‐of‐the‐art resemblance detection technique.

Enhancing IoT Data Integrity and Effectiveness through hybrid Compression Method: A Step Towards Energy Efficiency

The expansion of the Internet of Things (IoT) has magnified the challenge of managing data generated by IoT devices, notably in meteorological applications like temperature and humidity monitoring. This research addresses the imperative of efficiently reducing IoT data volume while preserving data integrity and underscores the significant implications for energy consumption. Our approach involved a two-fold strategy, employing the DHT11 sensor and ESP32 microcontroller for data collection, followed by an exploration of various data compression algorithms: delta encoding, run-length encoding (RLE), variable-length integer encoding (VLI), and bit-packing. The strategic combination of RLE and delta encoding yielded an exceptional compression rate of 98%. Beyond data reduction, this methodology offers energy savings by minimizing data transmission times, evidenced by the swift 133-microsecond compression process. Furthermore, the seamless transmission of compressed IoT data to Azure Cloud not only reduced cloud storage costs but also optimized storage space, contributing to energy efficiency. This research illuminates the significance of data compression in mitigating the environmental impact of IoT technologies, fostering a greener, more energy-conscious future.

Applying Delta Compression to Packed Datasets for Efficient Data Reduction

Applying Delta Compression to Packed Datasets for Efficient Data Reduction

A Secure Lossless Redundancy Elimination Scheme With Semantic Awareness for Cloud-Assisted Health Systems

Due to the limited scalability and reliability of traditional storage systems, more and more medical data are stored in cloud. However, the cost of cloud-assisted health systems is skyrocketing. How to improve storage utilization and save resource cost for health systems has become urgent problems. Therefore, this study proposes a secure lossless redundancy elimination scheme with semantic awareness, called SLRE, to decrease storage overhead and improve the overall system performance. Compared with existing studies, when the locality of data is almost nonexistent or poor, SLRE can eliminate redundant data well by exploiting semantic properties of similarity and entropy rather than relying on the locality. Then, fine-grained post-deduplication delta compression based on proxy similarity discovers more similar blocks and improves storage efficiency. In addition, filtered Lempel–Ziv compression based on proxy entropy filters data with low redundancy to improve the overall performance. Importantly, in order to ensure the security of medical data after deredundancy, different auditing strategies for medical data with diverse characteristics are proposed to guarantee security and improve the auditing efficiency. The experimental results show that compared with the state-of-the-art methods, the overall compression ratio of SLRE is improved, and the system throughput and auditing efficiency are also improved.

The Design of Fast and Lightweight Resemblance Detection for Efficient Post-Deduplication Delta Compression

Post-deduplication delta compression is a data reduction technique that calculates and stores the differences of very similar but non-duplicate chunks in storage systems, which is able to achieve a very high compression ratio. However, the low throughput of widely used resemblance detection approaches (e.g., N-Transform) usually becomes the bottleneck of delta compression systems due to introducing high computational overhead. Generally, this overhead mainly consists of two parts: ① calculating the rolling hash byte by byte across data chunks and ② applying multiple transforms on all of the calculated rolling hash values. In this article, we propose Odess, a fast and lightweight resemblance detection approach, that greatly reduces the computational overhead for resemblance detection while achieving high detection accuracy and a high compression ratio. Odess first utilizes a novel Subwindow-based Parallel Rolling (SWPR) hash method using Single Instruction Multiple Data [ 1 ] (SIMD) to accelerate calculation of rolling hashes (corresponding to the first part of the overhead). Odess then uses a novel Content-Defined Sampling method to generate a much smaller proxy hash set from the whole rolling hash set and quickly applies transforms on this small hash set for resemblance detection (corresponding to the second part of the overhead). Evaluation results show that during the stage of resemblance detection, the Odess approach is ∼31.4× and ∼7.9× faster than the state-of-the-art N-Transform and Finesse (a recent variant of N-Transform [ 39 ]), respectively. When considering an end-to-end data reduction storage system, the Odess-based system’s throughput is about 3.20× and 1.41× higher than the N-Transform- and Finesse-based systems’ throughput, respectively, while maintaining the high compression ratio of N-Transform and achieving ∼1.22× higher compression ratio over Finesse.

The Case for Cross-entity Delta Encoding in Web Compression (Extended)

Delta encoding and shared dictionary compression (SDC) for accelerating Web content have been studied extensively in research over the last two decades, but have only found limited adoption in the industry so far; compression approaches that use a custom-tailored dictionary per website have all failed in practice due to lacking browser support and high overall complexity. General-purpose SDC approaches such as Brotli reduce complexity by shipping the same dictionary for all use cases, while most delta encoding approaches just consider similarities between versions of the same entity (but not between different entities). In this study, we investigate how much of the potential benefits of SDC and delta encoding are left on the table by these two simplifications. As our first contribution, we describe the idea of cross-entity delta encoding that uses cached assets from the immediate browser history for content encoding instead of a precompiled shared dictionary; this avoids the need to create a custom dictionary, but enables highly customized and efficient compression. Second, we present an experimental evaluation of compression efficiency to hold cross-entity delta encoding against state-of-the-art Web compression algorithms. We consciously compare algorithms some of which are not yet available in browsers to understand their potential value before investing resources to build them. Our results indicate that cross-entity delta encoding is over 50% more efficient for text-based resources than compression industry standards. We hope our findings motivate further research and development on this topic.
 The extended version of our previously published paper [10] includes an additional section on the deltas of HTML files, a more detailed description of our approach (including a new visualization for the different dictionary strategies), a deeper discussion of compression efficiency, and details on additional future and ongoing work.

A multi‐feature‐based intelligent redundancy elimination scheme for cloud‐assisted health systems

AbstractRedundancy elimination techniques are extensively investigated to reduce storage overheads for cloud‐assisted health systems. Deduplication eliminates the redundancy of duplicate blocks by storing one physical instance referenced by multiple duplicates. Delta compression is usually regarded as a complementary technique to deduplication to further remove the redundancy of similar blocks, but our observations indicate that this is disobedient when data have sparse duplicate blocks. In addition, there are many overlapped deltas in the resemblance detection process of post‐deduplication delta compression, which hinders the efficiency of delta compression and the index phase of resemblance detection inquires abundant non‐similar blocks, resulting in inefficient system throughput. Therefore, a multi‐feature‐based redundancy elimination scheme, called MFRE, is proposed to solve these problems. The similarity feature and temporal locality feature are excavated to assist redundancy elimination where the similarity feature well expresses the duplicate attribute. Then, similarity‐based dynamic post‐deduplication delta compression and temporal locality‐based dynamic delta compression discover more similar base blocks to minimise overlapped deltas and improve compression ratios. Moreover, the clustering method based on block‐relationship and the feature index strategy based on bloom filters reduce IO overheads and improve system throughput. Experiments demonstrate that the proposed method, compared to the state‐of‐the‐art method, improves the compression ratio and system throughput by 9.68% and 50%, respectively.

TDD LoRa and Delta Encoding in Low-Power Networks of Environmental Sensor Arrays for Temperature and Deformation Monitoring

Densely distributed sensor networks can revolutionize environmental observations by providing real-time data with an unprecedented spatiotemporal resolution. However, field deployments often pose unique challenges in terms of power provisions and wireless connectivity. We present a framework for wirelessly connected distributed sensor arrays for near-surface temperature and/or deformation monitoring. Our research focuses on a novel time division duplex implementation of the LoRa protocol, enabling battery powered base stations and avoiding collisions within the network. In order to minimize transmissions and improve battery life throughout the network, we propose a dedicated delta encoding algorithm that utilizes the spatial and temporal similarity in the acquired data sets. We implemented the developed technologies in a AA battery powered hardware platform that can be used as a wireless data logger or base station, and we conducted an assessment of the power consumption. Without data compression, the projected battery life for a data logger is 4.74 years, and a wireless base stations can last several weeks or months depending on the amount of network traffic. The delta encoding algorithm can further improve this battery life with a factor of up to 3.50. Our results demonstrate the viability of the proposed methods for low-power environmental wireless sensor networks.

Design of a Quantization-Based DNN Delta Compression Framework for Model Snapshots and Federated Learning

Deep neural networks (DNNs) have achieved remarkable success in many fields. However, large-scale DNNs also bring storage costs when storing snapshots for preventing clusters’ frequent failures or incur significant communication overheads when transmitting DNNs in the Federated Learning (FL). Recently, several approaches, such as Delta-DNN and LC-Checkpoint, aim to reduce the size of DNNs’ snapshot storage by compressing the difference between two neighboring versions of the DNNs (a.k.a., delta). However, we observe that existing approaches, applying traditional global lossy quantization techniques in DNN's delta compression, can not fully exploit the data similarity since the parameters’ value ranges vary among layers. To fully explore the similarity of the delta model and improve the compression ratio, we propose a quantization-based local-sensitive delta compression approach, named QD-Compressor, by developing a layer-based local-sensitive quantization scheme and error feedback mechanism. Specifically, the quantizers and number of quantization bits are adaptive among layers based on the value distribution and weighted entropy of the delta's parameters. To avoid quantization error degrading the performance of the restored model, an alternative error feedback mechanism is designed to dynamically correct the quantization error during the training process. Experiments on multiple popular DNNs and datasets show that QD-Compressor obtains a higher 7×-40× compression ratio in the model snapshot compression scenario than the state-of-the-art approaches. Additionally, QD-Compressor achieves an 11×-15× compression ratio to the residual model of the Federated Learning compression scenario.

Base-Delta Dynamic Block Length and Optimization on File Compression

Delta compression uses the previous block of bytes to be used as a reference in the compression process for the next blocks. This approach is increasingly ineffective due to the duplication of byte sequences in modern files. Another delta compression model uses the numerical difference approach of the sequence of bytes contained in a file. Storing the difference value will require fewer representation bits than the original value. Base + Delta is a compression model that uses delta which is obtained from the numerical differences in blocks of a fixed size. Developed with the aim of compressing memory blocks, this model uses fixed-sized blocks and does not have a special mechanism when applied to file compression in general. This study proposes a compression model by developing the concept of Base+Delta encoding which aims to be applicable to all file types. Modification and development carried out by adopting a dynamic block size using a sliding window and block header optimization on compressed and uncompressed blocks giving promising test results where almost all file formats tested can be compressed with a ratio that is not too large but consistent for all file formats where the ratio compression for all file formats obtained between 0.04 to 12.3. The developed compression model also produces compression failures in files with high uncompressed blocks where the overhead of additional uncompressed blocks of information causes files to become larger with a negative ratio obtained of -0.39 to -0.48 which is still relatively small and acceptable.

BΔ-NIS: Performance analysis of an efficient data compression technique for on-chip communication network

Nowadays, system-on-chip (SoC) integrates more processing cores and IPs with network-on-chip (NoC) for satisfying today's demands, including low power consumption and high performance, without affecting the elasticity and scalability. When the on-chip processing elements (PEs) are increased, burst traffic will be generated from memory and cache accesses. As a result, the packet transmission latency is increased, reducing the system's speed performance. This paper proposes a new hybrid Base delta Neighbourhood Indexing Sequence (BΔ-NIS) based compression method for compressing the network traffic. The previous solutions for data compression often contain incompressible packets for certain data patterns. The proposed method uses Neighbourhood Indexing Sequence (NIS) algorithm to determine the optimal codeword for uncompressible chunks of delta compression approach using zeros and ones based traversals. Because of the dynamic nature of BΔ-NIS, it is highly effective and compresses all chunks in the packets efficiently. Moreover, the concept of dynamic header flit extension is introduced in the proposed compression method, which provide unlimited scalability. The proposed BΔ-NIS algorithm is robust to packet losses and leads to significant power saving. The simulation results show that the proposed BΔ-NIS reduces the hardware overhead and overtakes the existing data compression techniques by attaining a mean compression ratio of 0.73. In addition, BΔ-NIS accomplishes 51% latency reduction compared to the baseline router (without any compression). It also minimizes the power consumption of the de/compression modules by 70.14% and 20.8% as compared to the NoΔ and FlitZip respectively.

Delta Encoding Correction for Mobile Edge Caching in Internet of Vehicles

As one of the key components of intelligent transportation systems, the Internet of Vehicles enables vehicles to acquire and exchange essential information with each other and other facilities. As an infrastructure that directly connects the vehicles to the Internet, the roadside unit (RSU) is responsible for collecting information generated by the vehicle and transmitting it to the control center for analysis. However, ever-increasing number of vehicles and sensors will lead to a large amount of data. Furthermore, there exists redundancy in the data collected within a data collection period by vehicles. Both of these can severely reduce the efficiency of data storage and upload. In this article, we consider that each RSU installs a mobile edge computing server and present delta encoding correction for saving data cache space and transmission time. First, we propose a new cost model and an evaluation method for delta encoding of COPY/ADD class. Then, the type of delta encoding and the corresponding merging rules are defined. Based on these rules, we proposed a minimum delta encoding cost (MDC) algorithm, which adopts the idea of divide and conquer to obtain a superior correcting sequence of delta encoding. Theoretical analysis proves that the proposed MDC algorithm can generate the encoding sequence with the smallest total cost without affecting information reconstruction. In addition, we conduct experiments based on the randomly produced synthetic dataset. Experiment results show that, compared with the Greedy, One-Pass, and 1.5-Pass algorithms, the delta encoding output by the MDC algorithm has better performance in saving cache costs.

클라우드 스토리지 서비스를 위한 Rsync 프로토콜 기반의 파일 동기화 프레임워크

In this paper, we propose a hybrid file synchronization framework based on rsync protocol and file transfer for clould storage services. When a client modifies a file, rsync uses delta encoding that transfers only the differences between two files of the client and server for file synchronization. On the other hand, the delta encoding requires additional processing cost in the process of determining the difference between the two files. The proposed framework selectively applies both the delta encoding and the file transfer to the file synchronization task. For a new file, the file transfer shows shorter synchronization delay than the delta encoding. For an appended file, the delta encoding shows shorter delay than the file transfer. For a modified file, the proposed hybrid synchronization method minimizes the delay time by selectively using the delta encoding and file transfer depending on the file size and modification ratio.

A new real-time lossless data compression algorithm for ECG and PPG signals

ObjectiveData compression is a useful process in tele-monitoring applications, in which lesser number of bits are needed to represent the same data. In this work, a run-time lossless compression of single-channel Electrocardiogram (ECG) and Photoplethysmogram (PPG) signals is proposed, maintaining all dominant features. MethodsThe single-channel data are first quantized using optimal quantization level, so that fewer number of bits are needed to represent it, maintaining low quantization error. Then, second order delta encoding and run-length encoding (RLE) based data compression are proposed in this work. A new approach of using ‘buffer array’ along with RLE is also introduced, so that minimum bits are needed to store. ResultsThis algorithm was tested on various single-lead ECG and PPG signals available in Physionet. An average compression ratio (CR) was achieved of 6.52, 3.82, and 2.49 for 547 PTBDB ECG records, 48 MITDB ECG records, and 53 MIMIC-II PPG records, respectively. This algorithm was also performed on single-channel ECG, collected from 10 healthy volunteers using AD8232 ECG module, with 125 Hz sampling frequency and 10-bit data resolution, which resulted in average CR of 2.34. ConclusionThis algorithm was also performed on a smartphone device that provided user-friendly operation. The low computational complications and standalone operation of data collection, compression, and transmission encouraged its implementation for run-time operation. SignificanceA comparative study of the proposed work with previously published works proved this fact that this algorithm provided better performance in the area of run-time patient health monitoring applications.

Low-Power Lossless Data Compression for Wireless Brain Electrophysiology.

Wireless electrophysiology opens important possibilities for neuroscience, especially for recording brain activity in more natural contexts, where exploration and interaction are not restricted by the usual tethered devices. The limiting factor is transmission power and, by extension, battery life required for acquiring large amounts of neural electrophysiological data. We present a digital compression algorithm capable of reducing electrophysiological data to less than 65.5% of its original size without distorting the signals, which we tested in vivo in experimental animals. The algorithm is based on a combination of delta compression and Huffman codes with optimizations for neural signals, which allow it to run in small, low-power Field-Programmable Gate Arrays (FPGAs), requiring few hardware resources. With this algorithm, a hardware prototype was created for wireless data transmission using commercially available devices. The power required by the algorithm itself was less than 3 mW, negligible compared to the power saved by reducing the transmission bandwidth requirements. The compression algorithm and its implementation were designed to be device-agnostic. These developments can be used to create a variety of wired and wireless neural electrophysiology acquisition systems with low power and space requirements without the need for complex or expensive specialized hardware.