Data Compression Techniques Research Articles

Global Artificial Bee Colony Search Algorithm for Data Clustering

Data clustering is a widespread data compression, vector quantization, data analysis, and data mining technique. In this work, a modified form of ABC, i.e. global artificial bee colony search algorithm (GABCS) is applied to data clustering. In GABCS the modification is due to the fact that experienced bees can use past information of quantity of food and position to adjust their movements in a search space. Due to this fact, solution search equations of the canonical ABC are modified in GABCS and applied to three famous real datasets in this work i.e. iris, thyroid, wine, accessed from the UCI database for the purpose of data clustering and results were compared with few other stated algorithms such as K-NM-PSO, TS, ACO, GA, SA and ABC. The results show that while calculating intra-clustering distances and computation time on all three real datasets, the proposed GABCS algorithm gives far better performance than other algorithms whereas calculating computation numbers it performs adequately as compared to typical ABC.

Practices of backuping homomorphically encrypted databases

Ideal homomorphic encryption is theoretically achievable but impractical in reality due to tremendous computing overhead. Homomorphically encrypted databases, such as CryptDB, leverage replication with partially homomorphic encryption schemes to support different SQL queries over encrypted data directly. These databases reach a balance between security and efficiency, but incur considerable storage overhead, especially when making backups. Unfortunately, general data compression techniques relying on data similarity exhibit inefficiency on encrypted data. We present CryptZip, a backup and recovery system that could highly reduce the backup storage cost of encrypted databases. The key idea is to leverage the metadata information of encryption schemes and selectively backup one or several columns among semantically redundant columns. The experimental results show that CryptZip could reduce up to 90.5% backup storage cost on TPC-C benchmark.

A Novel Point Cloud Compression Algorithm Based on Clustering

Due to the enormous volume of point cloud data, transmitting and storing the data requires large bandwidth and storage space. It could be a critical bottleneck, especially in tasks such as autonomous driving. In this letter, we propose a novel point cloud compression algorithm based on clustering. The proposed scheme starts with a range image-based segmentation step, which segments the three-dimensional (3-D) range data into ground and main objects. Then, it introduces a novel prediction method according to the segmented regions’ shape. This prediction method is inspired by the depth modeling modes used in 3-D high-efficiency video coding for depth map coding. Finally, the few prediction residual is efficiently compressed with several lossless or lossy data compression techniques. Experimental results show that the proposed algorithm can largely eliminate the spatial redundant information of the point cloud data. The lossless compression scheme reaches a compression ratio of nearly 5%, which means that the point cloud is compressed to 5% of its original size without any distance distortion. Compared with other methods, the proposed compression algorithm also shows better performance.

An FPGA implementation of the matching pursuit algorithm for a compressed sensing enabled e-Health monitoring platform

Wireless monitoring of physiological signals is an evolving direction in personalized medicine and home-based e-Health systems. There are several constraints in designing such systems, with two of the most important being energy consumption and data compression. Compressed Sensing (CS) is an emerging data compression technique that can be used to overcome those constraints. This work presents a low-complexity CS hardware implementation on a Field-Programmable Gate Array (FPGA) for the reconstruction of compressively sensed signals using the matching pursuit (MP) algorithm, targeting health-care applications. The proposed hardware design is based on pipeline optimization of the Programmable Logic (PL) implementation performed on the Zynq FPGA, which provides a significant performance enhancement, namely an increased processing speed and a reduced computational time since it is 115x faster than the Matlab implementation and 75x faster than the Processing System (PS) implementation carried out on the same Zynq FPGA device, while achieving alternative a high-quality signal recovery with a Peak Signal to Noise Ratio (PSNR) of 23.8 dB. Comparisons against other state-of-the-art methods showed that the low complexity of the MP algorithm can be exploited for providing almost similar results to more complex algorithms using 87–583 less Digital Signal Processor (DSP) cores, 28–540 less Block RAMs and 10,300 to 84,700 less Look-Up Table (LUT) slices.

Optimizing the restoration performance of deduplication systems through an energy-saving data layout

While data deduplication is an important data compression technique that removes copies of repeated data to enhance storage utilization, security and privacy risks arise since sensitive or delicate user data are at risk to both insider and outsider attacks. A distinct negative factor to performance of the technique is data fragmentation, which not only slows down the restoration process but also leads to massive power consumption. In this paper, we address this problem from the perspective of data layout. The kernel point of our method is a novel RAID-5-based cross grouping data layout (CGDL). We introduce a selective deduplication algorithm (SDD) to perform data replication and restoration. A new CGDL-based disk scheduling algorithm (LDP) is also proposed that predicts location dependence to save energy by eliminating the redundant disk read/write operations. We evaluate our new method on the Linux MD (multiple device) driver modules. The experiments show that, under a 10 disks 3 groups storage configuration, our method drastically (by 20%) improves restoration efficiency with only 7.6% reduction on the deduplication ratio, while reducing 23% power consumption.

A heuristics based global navigation satellite system data reduction algorithm integrated with map-matching

The transmission and storage of global navigation satellite system (GNSS) data places very high demands on mobile networks and centralised data processing systems. GNSS applications including community based navigation and fleet management require GNSS data to be transmitted from a vehicle to a centralised system and then processed by a map-matching algorithm to determine the location of a vehicle within a road segment. Various data compression techniques have been developed to reduce the volume of data transmitted. There is also an independent literature relating to map-matching algorithms. However, no previous research has integrated data compression with a map-matching algorithm that accepts compressed data as an input without the need for decompression. This paper develops a novel GNSS data reduction algorithm with deterministic error bounds, which was seamless integrated with a specifically designed map-matching algorithm. The approach significantly reduces the volume of GNSS data communicated and improves the performance of the map-matching algorithm. The data compression extracts critical points in the trajectory and velocity–time curve of a vehicle. During the process of selecting critical points, the error of restoring vehicle trajectories and velocity–time curves are used as parameters to control the number of critical points selected. By setting different error bound values prior to the execution of the algorithm, the accuracy and volume of reduced data is controlled precisely. The compressed GNSS data, particularly the critical points selected from the vehicle’s trajectory is directly input to the map-matching algorithm without the need for decompression. An experiment indicated that the data reduction algorithm is very effective in reducing data volume. This research will be useful in many fields including community driven navigation and fleet management.

A lossless image compression algorithm using wavelets and fractional Fourier transform

The necessity of data transfer at a high speed, in fast-growing information technology, depends on compression algorithms. Maintaining quality of data reconstructed at high compression rate is a very difficult part of the data compression technique. In this paper, a new lossless image compression algorithm is proposed, which uses both wavelet and fractional transforms for image compression. Even though wavelets are the best choice for feature extraction from the source image at different frequency resolutions, the low-frequency sub-bands of wavelet decomposition are the untouched part in compression method in most of the existing methods. On the other hand, fractional Fourier transform is a convenient form of generalized Fourier transform that helps in the compact lossless coding of the source image with optimal fractional orders. Hence, we have used discrete fractional Fourier transform to compress those sensitive sub-bands of the wavelet transform, carefully. In this method, an image is split into low- and high-frequency sub-bands by using Daubechies wavelet filter and level 1 quantization is applied for both low-frequency and high-frequency sub-bands. The low-frequency sub-bands are compressed by using fractional Fourier transform with optimal fractional orders, and at the same time, high-frequency sub-bands are compressed by eliminating zeroes and storing only nonzero blocks and its position. The compressed wavelet coefficients are further compressed by the application of level 2 quantization and stored as a reduced array. This reduced array is encoded by using arithmetic encoder followed by run-length coding. The experimental results of the proposed algorithm with a different set of test images are compared with some of the existing image compression algorithms. The results show that the proposed method has significant improvement in image reconstruction quality.

Narrow big data in a stream: Computational limitations and regression

We consider the on-line model for a data stream: data points are waiting in a queue and are accessible one-by-one by a special instruction. When a data point is processed, it is dropped forever. The data stream is assumed to be so long that it cannot be stored in memory in full: the size of memory is assumed to be polynomial in the dimension of data, but not the number of observations. This is a natural model for Narrow Big Data. First we prove a negative theorem illustrating that this model leads to serious limitations: we show that some elementary statistics, such as sample quantiles, cannot be computed in this model (the proof is based on a Kolmogorov complexity argument). This raises a crucial question which data-analytic procedures can be implemented in the stream data model and which cannot be performed at all, or only approximately (with some loss of information). After the negative results, we turn our attention to several positive results from multivariate linear regression with Narrow Big Data. We prove that least-squares based estimators and regression diagnostic statistics, such as statistics based on the residual sum of squares, can be computed in this model efficiently. The class of statistics efficiently computable in the stream data model also includes two-stage procedures involving auxiliary regressions, such as White’s heteroscedasticity test of Breusch–Godfrey autocorrelation test (which may be surprising because the procedures, as defined, seem to require a data point to be processed several times). The computation is done exactly: we do not use preprocessing steps involving data compression techniques with information loss (such as sampling or grouping) for a reduction of the size of the data set.

Wireless High-Resolution Acceleration Measurements for Structural Health Monitoring of Wind Turbine Towers

Structural health monitoring (SHM) will be pivotal for safe and economic operation of wind turbines. Timely discovery of changes within the structure and means of prediction of required maintenance will reduce production costs of electricity and catastrophic failures. Long-term structural acceleration recording can support damage detection on turbine towers and document progression of fatigue. Conventional acceleration recordings are based on wired sensor nodes at fixed positions with privileged accessibility and electric power supply. However, such positions might be near vibration nodes and not necessarily experience the maximum vibration amplitude. Shifts in eigenfrequencies can be an indicator of changes in structural stiffness, hence damage, but also be caused by environmental effects, e.g., temperature. Damages generate local effects while the structure’s vibration spectrum is a global evaluation. If a sensor is close to the location of damage, the probability of detection is increased. Wireless sensors powered by batteries are advantageous for this task as they are independent of cabling for power supply and data transmission. Such monitoring of turbine tower structures is not common in practice and requires new data-enabled techniques to discover deviations from the optimal way of wind turbine operation. This paper proposes a new approach using wireless high-resolution acceleration measurement sensor nodes, exploiting the vibration response of wind turbine towers. Influences of acceleration resolution and sensor node locations onto the accuracy of eigenfrequency determination are demonstrated. A comparison between acceleration recordings by wireless sensor nodes and their wired counterparts is presented to prove the equivalence of the wireless sensing method. Finally, new data compression techniques used with the sensor nodes are discussed to reduce wireless transmission to a minimum.

An Aging-Aware GPU Register File Design Based on Data Redundancy

Nowadays, GPUs sit at the forefront of high-performance computing thanks to their massive computational capabilities. Internally, thousands of functional units, architected to be fed by large register files, fuel such a performance. At deep nanometer technologies, the SRAM memory cells that implement GPU register files are very sensitive to the Negative Bias Temperature Instability (NBTI) effect. NBTI ages cell transistors by degrading their threshold voltage $V_{th}$ over the lifetime of the GPU. This degradation, which manifests when a cell keeps the same logic value for a relatively long period of time, compromises the cell read stability and increases the transistor switching delay, which can lead to wrong read values and eventually exceed the processor cycle time, respectively, so resulting in faulty operation. This work proposes architectural mechanisms leveraging the redundancy of the data stored in GPU register files to attack NBTI aging. The proposed mechanisms are based on data compression, power gating, and register address rotation techniques. All these mechanisms working together balance the distribution of logic values stored in the cells along the execution time, reducing both the overall $V_{th}$ degradation and the increase in the transistor switching delays. Experimental results show that a conventional GPU register file suffers the worst case for NBTI, since a significant fraction of the cells maintain the same logic value during the entire application execution (i.e., a 100 percent ‘0’ and ‘1’ duty cycle distributions). On average, the proposal reduces these distributions by 58 and 68 percent, respectively, which translates into $V_{th}$ degradation savings by 54 and 62 percent, respectively.

Analysis of lossless compression techniques time-frequency-based in ECG signal compression.

Technological advances introduce different methods for telecardiology. Telecardiology includes many applications and is one of the long-standing medical fields that have grown very well. In telecardiology, a very high amount of ECG data is recorded. Therefore, to compress electrocardiogram data (ECG), there is a need for an efficient technique and lossless compression. The compression of ECG data reduces the storage needs for a more efficient cardiological system and for analyzing and diagnosing the condition of the heart. In this paper, ECG signal data compression techniques are analyzed using the MIT-BIH database and then compared with the Apnea-ECG and Challenge 2017 Training bases. During the study, some of the various techniques of frequency analysis, range and time are widely used, such as run-time coding, AZTEC, Spin-Coding, FFT, DCT, DST, SAPA/FAN and DCT-II, where DCT and SAPA/FAN have the best compression performance compared to other methods.

SPADE: Activity Prediction in Smart Homes Using Prefix Tree Based Context Generation

An automated smart home system is a key contributor to user assistance technology in modern civilization. Crucial merit of such a system is its ability to train itself through recorded data and recognize patterns in resident behaviors. Lack of sufficient prediction accuracy, exponential memory consumption, and extensive runtime prevent many of the current activity prediction approaches from being seamlessly integrated into consumer residences. This research introduces a sequence prediction algorithm which uses a prefix tree-based data model in order to learn and predict user actions. The algorithm applies episode discovery to detect correlated sensor events and learns the activities using a lossless data compression technique. This process assigns a probability of occurrence to sensor events and uses these probabilities to detect patterns in resident behavior. A complexity analysis of the algorithm is done to prove its efficiency in terms of memory usage and runtime. Using the presented technique, predictions are performed on popular datasets and contrasted with existing algorithms. The proposed algorithm achieves an 8.22% improvement in prediction accuracy over its predecessors, along with 66.69% better memory efficiency and 37% faster runtime.

Adaptive event-triggered anomaly detection in compressed vibration data

Abstract Anomaly detection is a crucial task in Prognostics and Condition Monitoring (PCM) of machinery. In modern remote PCM systems, data compression techniques are regularly used to reduce the need for bandwidth and storage. In these systems the challenge arises of how the compressed (distorted) vibration data affects the condition monitoring algorithms. This paper introduces a novel algorithm that can adaptively establish normal bounds of operation from continuous noisy vibration profiles working with compressed vibration data. The proposed technique is based on four modules, including feature extraction, feature fusion, extreme value vibration modeling and adaptive thresholding for anomaly detection. The proposed method has been validated with experiments using three time-series datasets. The experimental results indicate that the proposed algorithm is able to perform detection of malfunctions in rotating machines effectively without faulty reference data. Moreover, the proposed method is able to produce accurate early warning and alarm indications from both the raw and compressed (distorted) datasets with equal veracity.

Efficient Lossless Compression of Multitemporal Hyperspectral Image Data

Hyperspectral imaging (HSI) technology has been used for various remote sensing applications due to its excellent capability of monitoring regions-of-interest over a period of time. However, the large data volume of four-dimensional multitemporal hyperspectral imagery demands massive data compression techniques. While conventional 3D hyperspectral data compression methods exploit only spatial and spectral correlations, we propose a simple yet effective predictive lossless compression algorithm that can achieve significant gains on compression efficiency, by also taking into account temporal correlations inherent in the multitemporal data. We present an information theoretic analysis to estimate potential compression performance gain with varying configurations of context vectors. Extensive simulation results demonstrate the effectiveness of the proposed algorithm. We also provide in-depth discussions on how to construct the context vectors in the prediction model for both multitemporal HSI and conventional 3D HSI data.

Lifetime-aware FTL to improve the lifetime and performance of solid-state drives

Data compression techniques have been deployed in SSDs to prolong their lifetime due to the good tradeoff between data reduction and performance degradation. However, existing schemes either consume plenty of DRAM resources, or cause the read amplification problem, which may further result in significant performance degradation. In this paper, we propose a lifetime-aware FTL scheme (LAFTL) to improve both the lifetime and performance of SSDs by compressing several pages into one physical page. We then design a compression-aware page mapping table (PMT) to reduce the memory consumption, and propose a latency-aware approach to improve the performance. We conduct extensive trace-driven evaluations based on real-world workloads, and results show that our LAFTL reduces the average response time of reads by 3%–78% and ensures a minimal resource consumption in DRAM while obtaining an acceptable data reduction rate compared to other schemes.

Statistical Data Compression and Differential Coding for Digital Radio-Over-Fiber-Based Mobile Fronthaul

Digital radio over fiber (D-RoF), one of the candidates for 5G mobile fronthaul networks, is known for its high reliability and strong robustness against nonlinear channel degradations, which makes it suitable for short-reach fronthaul links supporting ultra-reliable low-latency communication in 5G. However, traditional D-RoF technology is limited by its lower bandwidth efficiency. In this paper, based on our previous work, advanced data-compression techniques with adaptive non-uniform quantizers and differential coding are discussed for a significant improvement of bandwidth efficiency in fronthaul networks. High-order differential coding based on a least- mean-square algorithm has been proposed to further improve the compression ratio with low complexity and high adaptability. By jointly applying a non-uniform quantizer and a differentiator, the signal-to-quantization-noise ratio and bandwidth efficiency can be improved by around 10 dB and 40%-60%, respectively, depending on the modulation formats in our proposed solution. We have experimentally demonstrated the transmission of 200 Gbps fronthaul links over a fiber distance of 80 km. The system is capable of encapsulating 110 × 120 MHz 5G new radio carriers with error-vector magnitude lower than 0.8%.

Adaptive Band Limit Estimation based PPG data compression for portable home monitors

Continuous long term monitoring of Photoplethysmogram (PPG) in home monitoring applications necessitates the need for data compression to reduce the storage requirement. Limited PPG data compression techniques are available which have fully exploited the signal redundancies. This paper presents a PPG data compression technique based on adaptive estimation of the useful band-limit from the individual signal characteristic. Each data block is then represented using the corresponding discrete Fourier transform coefficients, which are encoded by adaptive quantization and optimal bit allocation to further enhance the compression performance. The algorithm achieves an overall compression ratio of 35.95 with signal distortion maintained below 4%, as tested with 16 bit quantized PPG data sampled at 125 Hz. The algorithm is also implemented and validated on a microcontroller based Arduino board. High compression performance, implementation simplicity and noise robustness justifies the potentiality of the algorithm for its use with portable home monitoring devices.

Test Data Compression with Alternating Equal-Run-Length Coding

This paper presents a new X-filling algorithm for test power reduction and a novel encoding technique for test data compression in scan-based VLSI testing. The proposed encoding technique focuses on replacing redundant runs of the equal-run-length vector with a shorter codeword. The effectiveness of this compression method depends on a number of repeated runs occur in the fully specified test set. In order to maximize the repeated runs with equal run length, the unspecified bits in the test cubes are filled with the proposed technique called alternating equal-run-length (AERL) filling. The resultant test data are compressed using the proposed alternating equal-run-length coding to reduce the test data volume. Efficient decompression architecture is also presented to decode the original data with lesser area overhead and power. Experimental results obtained from larger ISCAS'89 benchmark circuits show the efficiency of the proposed work. The AERL achieves up to 82.05 % of compression ratio as well as up to 39.81% and 93.20 % of peak and average-power transitions in scan-in mode during IC testing.

DWT Based Image Compression for Health Systems

There are calls for enhancing present healthcare sectors when it comes to handling huge data size of patients’ records. The huge files contain lots of duplicate copies. Therefore, the ideal of compression comes into play. Image data compression removes redundant copies (multiple unnecessary copies) that increase the storage space and transmission bandwidth. Image data compression is pivotal as it helps reduce image file size and speeds up file transmission rate over the internet through multiple wavelet analytics methods without loss in the transmitted medical image data. Therefore this report presents data compression implementation for healthcare systems using a proposed scheme of discrete wavelet transform (DWT), Fourier transform (FT) and Fast Fourier transform with capacity of compressing and recovering medical image data without data loss. Healthcare images such as those of human heart and brain need fast transmission for reliable and efficient result. Using DWT which has optimal reconstruction quality greatly improves compression. A representation of enabling innovations in communication technologies with big data for health monitoring is achievable through effective data compression techniques. Our experimental implementation shows that using Haar wavelet with parametric determination of MSE and PSNR solve our aims. Many imaging techniques were also deployed to further ascertain DWT method’s efficiency such as image compression and image de-noising. The proposed compression of medical image was excellent. It is essential to reduce the size of data sets by employing compression procedures to shrink storage space, reduce transmission rate, and limit massive energy usage in health monitoring systems. The motivation for this work was to implement compression method to modify traditional healthcare platform to lower file size, and reduce cost of operation. Image compression aims at reconstructing images from extensively lesser estimations than were already thought necessary in relations with non-zero coefficients. Rationally, fewer well-chosen interpretations is adequate to reproduce the new sample exactly as the source image. We look at DWT to implement our compression method.

A multi-mode real-time system verification model using efficient event-driven dataset

Real-time systems experience many safety and performance issues at run time due to different uncertainties in the environment. Systems are now becoming highly interactive and must be able to execute in a changing environment without experiencing any failure. A real-time system can have multiple modes of operation such as safety and performance. The system can satisfy its safety and performance requirements by switching between the modes at run time. It is essential for the designers to ensure that a multi-mode real-time system operates in the expected mode at run time. In this paper, we present a verification model that identifies the expected mode at run time and checks whether the multi-mode real-time system is operating in the correct mode or not. To determine the expected mode, we present a monitoring module that checks the environment of the system, identifies different real-world occurrences as events, determines their properties and creates an event-driven dataset for failure analysis. The dataset consumes less memory in comparison to the raw input data obtained from the monitored environment. The event-driven dataset also facilitates onboard decision-making because the dataset allows the system to perform a safety analysis by determining the probability of failure in each environmental situations. We use the probability of failure of the system to determine the safety mode in different environmental situations. To demonstrate the applicability of our proposed scheme, we design and implement a real-time traffic monitoring system that has two modes: safety, and performance. The experimental analysis of our work shows that the verification model can identify the expected operating mode at run time based on the safety (probability of failure) and performance (usage) requirements of the system as well as allows the system to operate in performance mode (in 3295 out of 3421 time intervals) and safety mode (in 126 out of 3421 time intervals). The experimental result demonstrates that the volume of the dataset generated using our approach is significantly less in comparison to existing data compression techniques.