Delta Compression Research Articles

A Bi-Level Data Lowering Method to Minimize Transferring Big Data in the Sensors of IoT Applications

In the IoT era, the number of devices connected to it continues to grow significantly. This can lead to an increase in the amount of reported data by these IoT devices. The reported data by the Sensor Nodes (SNs) to the Gateway (GW) drives these IoT sensors to consume their energy and storage. These problems can be solved by reducing the amount of data in the source nodes in order to reduce both the amount of energy consumed and the amount of storage required. Energy consumption represents one aspect of the Quality of Service (QoS) in the sensor nodes of the IoT. A Bi-Level Data Lowering (BLDL) approach is suggested in this article that operates at both the sensor node and gateway levels. To function in constrained IoT devices at the first level, lightweight data compression approaches were used. Delta encoding is accompanied by RLE. Two more optimization methods have been proposed for the sake of minimizing the amount of sent data as much as possible at the first level. In the second level, clustering hierarchically based on Minimum Description Length (MDL) theory was used to cluster the first level data sets. After that, the evaluation of BLDL efficiency is based on real data and the use of the OMNeT++ simulator. The findings indicate that the suggested approach reduces the overhead for the network resources as follows: At a maximum of 20.53% in BLDL and 6.14% in LBLDL for the ratio of data remaining, and a maximum of 62% in BLDL and 21% in LBLDL for the ratio of sent data sets to the GW, the required energy to send data sets to the GW is reduced from 6% to 43% in BLDL and from 79% to 85% in LBLDL, and accuracy is higher than 90% for the methods without loss and 80% for the methods with a loss when compared to existing methods.

Ensuring high reliability and performance with low space overhead for deduplicated and delta‐compressed storage systems

AbstractData deduplication is a widely used technique to remove duplicate data to reduce the storage overhead. However, deduplication typically cannot eliminate the redundancy among nonidentical but similar data chunks. To reduce the storage overhead further, delta compression is often applied to compress the post‐deduplication data. While the two techniques are effective in saving storage space, they introduce complex references among data chunks, which inevitably undermines the system reliability and introduces fragmentation that may degrade the restore performance. In this paper, we observe that the delta compressed chunks (DCCs) are much smaller than regular chunks (non‐DCCs). Also, most fragmentation caused by the base chunk of DCCs remain fragmented in consecutive backups. Based on these observations, we introduce a framework called , which combines replication and erasure coding and uses History‐aware Delta Selection to ensure high reliability and restore performance. Specifically, uses a delta‐utilization‐aware filter and a cooperative cache scheme (CCS) to maintain cache locality and avoid unnecessary container reads, respectively. Moreover, the system selectively performs delta compression by historical information to avoid cyclic fragmentation in consecutive backups. Experimental results based on four real‐world datasets demonstrate that significantly improves the restore performance by 58.3%–76.7% with a low storage overhead.

A Novel Lossless ECG Compression Algorithm for Active Implants.

A low complexity lossless ECG compression algorithm for active implants is proposed in this paper. The algorithm is based on adaptive length encoding by combining linear prediction with delta encoding. The algorithm is tested on forty-eight segments of 30-min ECG signals obtained from MIT-BIH Arrhythmia Database. The results show that with the data segment length of 33 and the predictor order of 2, the average compression rate of the algorithm reaches 2.43 and there is no difference between the reconstructed signal and the original one. It implies that it can realize the lossless compression with a high compression ratio. Meanwhile, the low complexity makes this novel algorithm suitable for ECG monitoring applications of active implants.

A cost‐efficient resemblance detection scheme for post‐deduplication delta compression in backup systems

SummaryDelta compression, which is efficient in removing repeated string among similar chunks, can be used as a complement to data deduplication in backup storage for extra space savings. The process of detecting similar candidates to use as the base for delta compression is called resemblance detection. Several indexes are required for resemblance detection. Maintaining them in RAM would limit the system scalability and increase system cost. Storing them on the disk suffers from low throughput due to poor random I/O performance of the disk. In this article, we present the history‐aware resemblance detection (HARD), a cost‐efficient resemblance detection approach that captures most of the similar chunks with a limited memory footprint. HARD is based on the observation that, for chunks in a backup, most of their similar chunks can be found in the most recent backups. HARD thus only indexes super‐features in the most recent backups for resemblance detection to reduce the memory footprint of resemblance indexes while captures most of the potential similar chunks for delta compression. Experimental results based on three real‐world datasets show that HARD achieves higher compression than the state‐of‐the‐art approach.

Modified Chorological Coded Data Compression Methodology for Wireless Sensor Networks

Wireless Sensor Network (WSN) is a fast evolving current technology which is being employed in various applications. Despite its wide usage, WSNs have a few challenges to overcome to be called as an ideal technology. Some of the challenges are battery life time, memory storage and deployment issues. Batteries are the primary source of power supply to WSN and one of the major challenges is the energy constraint. This paper aims to propose a few techniques to better the energy efficiency of the sensor networks by saving sensor energy using data compression methodology. In this paper, a modified chorological coded data compression methodology is proposed (MCDC). This algorithm deals with assigning the sequence value to the given input information. If the assigned sequence value is a double digit number, it is converted in to single digit number. Double digit numbers and single digit numbers are combined. Separate location tables are generated for all double digit and single digit numbers. This procedure continues until all the sequence values are changed in to single digit number. Then the final single digit is assign with an equivalent Sequence Code (SC). MCDC algorithm is compared with DELTA compression and RUNLENGTH compression and a better compression ratio was achieved when compared with DELTA compression and RUNLENGTH compression algorithms.

Выбор факторов для прогнозирования характеристик живой миграции виртуальных машин

Virtual machine migration is widely used in cloud data centers to scale and maintain the stability of cloud services. However, the performance metrics of virtual machine (VM) applications during migration that are set in the Service Level Agreements may deteriorate. Before starting a migration, it is necessary to evaluate the migration characteristics that affect the quality of service. These characteristics are the total migration time and virtual machine downtime, which are random variables that depend on a variety of factors. The prediction is based on the VM monitoring data. In this paper, we select the most suitable factors for forecasting five types of migrations: precopy migration, postcopy migration, and modification of precopy migration such as CPU throttling, data compression, and delta compression of modified memory pages. To do this, we analyzed a dataset that includes data on five types of migrations, approximately 8000 records of each type. Using correlation analysis, the factors that mostly affect the total migration time and the VM downtime are chosen. These characteristics are predicted using machine learning methods such as linear regression and the support vector machine. It is shown that the number of factors can be reduced almost twice with the same quality of the forecast. In general, linear regression provides relatively high accuracy in predicting the total migration time and the duration of virtual machine downtime. At the same time, the observed nonlinearity in the correlations shows that it is advisable to use the support vector machine to improve the quality of the forecast.

A new technique for data compression

Data compression has become more important than ever, due to the increasing demand for internet use and the exchange of a huge amount of images, videos, audio and documents as well as the growing demand for electronic archiving by government departments that produce thousands of documents per day. In this paper, the proposed technique for document compression will be presented.The proposed technique is a lossless and completed technique it is consists of two parts the compression part and decompression part. The compression part contains of some basic stages such as: pre-processing, blocks processing, run length encoding(RLE), replace maximum values by unused values, minimize levels, delta encoding, compression of ones values, encryption. After the encryption process is complete, the outputs are stored in two separate binary files with the extension of bmp;one of them is the header file and is considered a key for the second file which contains the compressed data. This technique applied on twenty documents and compared with other methods compression such as RLE, jpeg, tiff and png. The experimental results showed that the proposed technique gives a higher compression ratio than the rest of the methods.

Improving Restore Performance for In-Line Backup System Combining Deduplication and Delta Compression

Data deduplication, though being efficient in removing duplicate chunks, introduces chunk fragmentation which decreases restore performance. Rewriting algorithms are proposed to reduce the chunk fragmentation. Delta compression is often used as a complement for data deduplication to further improve storage efficiency. We observe that delta compression introduces a new type of chunk fragmentation stemming from improper delta compression for chunks of which the base chunks are fragmented. The new type of chunk fragmentation severely decreases restore performance and cannot be addressed by existing rewriting algorithms. To address this problem, we propose SDC, a scheme performing post-deduplication delta compression only for the chunks of which the bases can be directly found in the restore cache to eliminate additional disk reads for base chunks, thus avoiding the new type of chunk fragmentation. In addition, self-referenced chunks can be fragmented, which decrease restore performance, and these fragmented chunks can serve as bases to decrease the restore performance repeatedly. We propose a hybrid rewriting scheme for SDC to rewrite such fragmented chunks. Experimental results show that SDC improves the restore performance of the approach that directly performs delta compression after data deduplication by 2.9-16.9x, and achieves more than 95 percent of its compression gains.

A Lossless Compression Approach Based on Delta Encoding and T-RLE in WSNs

The sending/receiving of data (data communication) is the most power consuming in wireless sensor networks (WSN) since the sensor nodes are depending on batteries not generally rechargeable characterized by limited capacity. Data compression is among the techniques that can help to reduce the amount of the exchanged data between wireless sensor nodes resulting in power saving. Nevertheless, there is a lack of effective methods to improve the efficiency of data compression algorithms and to increase nodes’ energy efficiency. In this paper, we proposed a novel lossless compression approach based on delta encoding and two occurrences character solving (T-RLE) algorithms. T-RLE is an optimization of the RLE algorithm, which aims to improve the compression ratio. This method will lead to less storage cost and less bandwidth to transmit the data, which positively affects the sensor nodes’ lifetime and the network lifetime in general. We used real deployment data (temperature and humidity) from the sensor scope project to evaluate the performance of our approach. The results showed a significant improvement compared with some traditional algorithms.

Delta compression optimisation for UAV‐enabled mobile edge caching

Unmanned aerial vehicles (UAVs)-based sensor network is an effective mechanism for recognising and tracking of manoeuvring targets as well as expanding the monitoring coverage in the battlefield. However, there exists redundancy among the spectrum data collected by a UAV monitor within a data collection period, which may waste storage space and reduce the speed of data uploaded to the control centre. The authors assume that each UAV is equipped with an edge computing server and propose a delta compression method, which can save cache space and transfer time. First, they present a cost model and evaluation model for delta compression of the COPY/ADD class. Then an optimisation problem is formulated aiming to obtain the optimal delta encoding. Additionally, a maximal total length of copied fragments (MTLC) algorithm is proposed to find more mutually separated L-grams common fragments between the data collected at two adjacent moments. Theoretical analysis proves that the MTLC algorithm can generate a good delta encoding with the maximum total length and then the minimum total number of the COPYs. Moreover, numerical results show that MTLC has better performance in constructing a good delta encoding than the simple greedy and hash suffix array delta algorithms.

Lossless Compression of Intracortical Extracellular Neural Recordings using Non-Adaptive Huffman Encoding.

This paper investigates the effectiveness of four Huffman-based compression schemes for different intracortical neural signals and sample resolutions. The motivation is to find effective lossless, low-complexity data compression schemes for Wireless Intracortical Brain-Machine Interfaces (WI-BMI). The considered schemes include pre-trained Lone 1st and 2nd order encoding [1], pre-trained Delta encoding, and pre-trained Linear Neural Network Time (LNNT) encoding [2]. Maximum codeword-length limited versions are also considered to protect against overfit to training data. The considered signals are the Extracellular Action Potential signal, the Entire Spiking Activity signal, and the Local Field Potential signal. Sample resolutions of 5 to 13 bits are considered. The result show that overfit-protection dramatically improves compression, especially at higher sample resolutions. Across signals, 2nd order encoding generally performed best at lower sample resolutions, and 1st order, Delta and LNNT encoding performed best at higher sample resolutions. The proposed methods should generalise to other remote sensing applications where the distribution of the sensed data can be estimated a priori.

Compression protocol for optimized random phase color holographic videos by delta encoding

In this paper, we develop a tailor-made compression protocol with high performance, both for lossless and lossy compression of holographic videos generated by means of optimized random phases. We demonstrate that this holographic generation technique introduces an interframe correlation that makes delta encoding an efficient way to store holographic video data. By combining this encoding with common lossless compression algorithms, we achieve an increase of the compression ratio of 0.35 compared to a holographic video generated by direct application of the Gerchberg–Saxton algorithm. Additionally, we show that a high level of lossy compression can be achieved with this encoding, with better reconstruction quality than direct application of common video compression techniques, like MPEG-4 and motion-JPEG. Furthermore, we verify experimentally the performance of the compression scheme by using a space division multiplexing color holographic projection setup. Compression ratios of more than six are achieved while maintaining a high level of quality after experimental optical reconstruction.

CRUSH: A New Lossless Compression Algorithm

Multimedia is highly competitive world, one of the properties that is reflected is speed of download and upload of multimedia elements: text, sound, pictures, animation. This paper presents CRUSH algorithm which is a lossless compression algorithm. CRUSH algorithm can be used to compress files. CRUSH method is fast and simple with time complexity O(n) where n is the number of elements being compressed.Furthermore, compressed file is independent from algorithm and unnecessary data structures. As the paper will show comparison with other compression algorithms like Shannon–Fano code, Huffman coding, Run Length Encoding, Arithmetic Coding, Lempel-Ziv-Welch (LZW), Run Length Encoding (RLE), Burrows-Wheeler Transform.Move-to-Front (MTF) Transform, Haar, wavelet tree, Delta Encoding, Rice &Golomb Coding, Tunstall coding, DEFLATE algorithm, Run-Length Golomb-Rice (RLGR).

CRUSH: A New Lossless Compression Algorithm

Multimedia is highly competitive world, one of the properties that is reflected is speed of download and upload of multimedia elements: text, sound, pictures, animation. This paper presents CRUSH algorithm which is a lossless compression algorithm. CRUSH algorithm can be used to compress files. CRUSH method is fast and simple with time complexity O(n) where n is the number of elements being compressed.Furthermore, compressed file is independent from algorithm and unnecessary data structures. As the paper will show comparison with other compression algorithms like Shannon–Fano code, Huffman coding, Run Length Encoding, Arithmetic Coding, Lempel-Ziv-Welch (LZW), Run Length Encoding (RLE), Burrows-Wheeler Transform.Move-to-Front (MTF) Transform, Haar, wavelet tree, Delta Encoding, Rice &Golomb Coding, Tunstall coding, DEFLATE algorithm, Run-Length Golomb-Rice (RLGR).

Direct merging of delta encoded files

Delta encoding represents a target file making use of a source file by replacing common substrings by pointer references. Two similar, yet different, models are introduced and investigated in this paper: the Compressed Transitive Delta Encoding (CTDE) and the Compressed Source Delta Encoding (CSDE) paradigms. In these models we are given two delta files and the goal is to construct a third delta file working directly on the given compressed forms.Formally, given a source file S and two differencing files Δ(S,T) and Δ(T,R), where Δ(X,Y) is used to denote the delta file of the target file Y with respect to the source file X, the objective of the CTDE problem is to be able to attain R. Unlike the traditional way which uses S to decompress Δ(S,T), in order to attain T, and then applies Δ(T,R) on T to obtain R, CTDE constructs a delta file Δ(S,R) working directly on the two given delta files Δ(S,T) and Δ(T,R), without any decompression or the use of the base file S. Thus, avoiding the storage of the redundant intermediate file T. An algorithm for solving CTDE is proposed and its compression performance is compared to the traditional “double delta decompression”. Not only does it use constant space, as opposed to linear memory storage used by the traditional method, experiments show that the compression efficiency of the constructed delta file Δ(S,R) is usually better than both Δ(S,T) and Δ(T,R).The CSDE problem deals with a source file S and two differencing files Δ(S,T) and Δ(S,R), and the goal is still to be able to attain R. Although it is not always possible to construct the target file R by processing only the two input delta files, empirical experiments show that on typical real life data, usually about 99% of the file can be constructed using the proposed algorithm for the CSDE problem.

Study of Compressed Sensing and Predictor Techniques for the Compression of Neural Signals under the Influence of Noise.

In this paper an analysis of compression schemes based on compressed sensing (CS) and predictor techniques for neural signals is presented. The focus is on how much a compression algorithm can reduce data while not affecting the subsequent signal processing. Since neural signals are processed by means of spike sorting algorithms the evaluation is not trivial and not well defined, since there exists in fact many different ways to detect and cluster the spikes. Evaluating how much a compression scheme affects the result of spike sorting programs is a crucial step before implementing such compression technique. In the analysis two use cases are evaluated: in the first, spikes are detected and extracted and only thereafter compressed. In the second case, no information on the spikes is available and the whole raw signal is compressed. When dealing only with spike frames CS offers great compression at almost no loss, in the case of the whole recording its performances are greatly impaired and delta compression outperforms it in terms of data reduction and spike sorting results. In this case the reduction rates are modest but significant, ≈3 - 4 times data reduction and the whole signal is preserved avoiding big permanent losses of information.

Development of a real‐time adaptive delta compression algorithm for photoplethysmography system

Photoplethysmography (PPG) is a simple and low‐cost optical technique that measures the blood volume change of the microvascular bed. The multichannel PPG recording system enables researchers to study the characteristics of the PPG signal from multiple body sites. Long‐time, continuous PPG signal monitoring helps doctors in medical diagnosis and treatment. However, the development of the PPG signal acquisition system is limited by storage capacity and computational and hardware capability. Limitation in storage capacity and communication efficiency can be addressed by introducing data compression algorithms. In this paper, a single‐channel, high‐resolution (24‐bit) wireless PPG data acquisition system using Bluetooth connectivity with the adaptive delta compression algorithm is presented. A graphical user interface was developed using MATLAB to interface with the system using a serial port profile. The PPG signals were decompressed and stored in the local hard drive for future analysis. The performance of adaptive data compression algorithm was evaluated with 12 healthy subjects. Results show that the compression ratio achieved is 41.5. The mean percentile root‐mean‐square (RMS) difference at the sampling frequency of 100 Hz is 0.13%. The mean normalized percentile RMS difference value is 9.78%. The system was successfully tested in a continuous acquisition mode for up to 60 min. © 2018 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

DCDedupe: Selective Deduplication and Delta Compression with Effective Routing for Distributed Storage

Data deduplication has been an essential part of storage systems for big data. Traditional compare-by-hash (CBH) deduplication does not fully address the challenges for similar files with small changes. Delta compression can be complementary to further optimize the storage efficiency. In this paper, we designed and implemented a distributed storage system called DCDedupe that efficiently and intelligently use delta compression or deduplication to improve storage efficiency based on characteristics of data. Unlike prior studies, this system works well when the data locality is weak or even barely exists. In DCDedupe, we propose a pre-processing step to identify content similarity and data chunks are classified into different categories. Then, the appropriate routing algorithm ensures the data chunks are sent to the right target storage nodes in the distributed system to boost the storage efficiency. Our evaluation shows that generally storage space saving by DCDedupe outweighs the performance penalties. In some use cases, DCDeupe may become meaningful to trade off some throughput with optimized storage costs. The overheads to Input/Output (IO) operation and memory usage have also been studied with design recommendations.

Improved pre-copy algorithm using statistical prediction and compression model for efficient live memory migration

Xen hypervisor is used to execute and migrate the guests on different architectures using a pre-copy algorithm. There are three major categories to improve pre-copy using live migration algorithms: 1) reducing dirty pages; 2) predicating dirty pages; 3) compressing memory pages. The methods based on reducing dirty pages can lead to performance degradation so the new approach called combined approach (including prediction and compression) is proposed in this paper. The prediction of dirty pages during a migration is performed using auto-regressive integrated moving average (ARIMA) model. A least recently used (LRU) stack distance-based delta compression algorithm is proposed for compression model to achieve efficient virtual machine migration. The results show that ARIMA-based model is able to predict 93% in the case of high dirty pages environment. The combined approach is able to reduce 19.16% downtime and 10.76% total migration time on an average compared to Xen's pre-copy algorithm.

RSD: Rate-Based Sync Deferment for Personal Cloud Storage Services

Cloud storage services, such as Dropbox, Google Drive, and OneDrive, to cite a few, are becoming an increasingly “vital” tool in our everyday life. Unluckily, these services can incur large network overhead in different usage scenarios. To reduce it, these systems utilize several techniques, such as source-based deduplication, chunking, delta compression, and so on. One of these techniques is sync deferment, which relies on the packing of updates to intentionally defer the synchronization process for some time, and increase the volume of useful data per overhead byte. The scientific literature has shown this technique to be very helpful, though there are still some limitations on current solutions. To resolve them, we present here a new adaptive sync deferment method, which is comparable to the current state of the art in terms of network overhead, but is also able to minimize the file synchronization time up to 12 times.