Data Compression Scheme Research Articles

A Low-Resources Hardware-based Audio Data Compression Scheme for Wireless Sensors Networks

A Low-Resources Hardware-based Audio Data Compression Scheme for Wireless Sensors Networks

New fog computing enabled lossless EEG data compression scheme in IoT networks

The rapid evolution of the Internet of Things (IoT) leads to emerging several applications such as smart home, ehealth, smart agriculture and farming, etc. The edge of the IoT networks will receive a huge amount of data generated by these smart applications which regularly require to be delivered to the servers of the remote data centers for additional real-time treatment. However, transmitting the whole of these IoT data across the network toward the data center of the cloud will impose a high overhead on the IoT network. The data exchange and processing long delay have a significant impact on the response speed of the real-time IoT applications. It can reduce the time responsiveness of these applications. The fog computing has provided for the IoT applications as an intermediate layer between the smart end devices and the cloud to reduce IoT data and improve the response time for IoT applications. In this paper, new fog computing enabled lossless EEG data compression scheme is proposed to minimize the amount of IoT EEG data uploaded to the cloud. The EEG data compression scheme consists of two phases: clustering and compression. First, the received data is clustered into clusters using agglomerative hierarchical clustering. The Huffman encoding is applied to each resulted cluster in the second phase. Finally, the compressed files of smaller clusters are combined and uploaded from fog to cloud. Several experiments have been implemented and the comparison results show that the suggested combined Hierarchical Clustering and Huffman Encoding (HCHE) method significantly decreases the volume of EEG data uploaded to the cloud platform. The proposed HCHE method achieves 4.33 of average compression power that is more than twice as much the compression power of some existing methods.

Distributed stochastic principal component analysis using stabilized Barzilai-Borwein step-size for data compression with WSN

The popularity of diverse IoT-based applications and services continuously generating tremendous amount of data has revealed the significance of data compression (DC). Principal component analysis (PCA) is one of the most commonly employed algorithms for DC. However, when dealing with large-scale matrices, the standard PCA takes a very long time and requires a lot of memory. Therefore, this paper presents a novel distributed stochastic PCA algorithm (DSPCA) for hierarchical sensor network based on gradient-based adaptive PCA (GA-PCA), where the standard PCA is reformulated as a single-pass stochastic setting to find the direction of approximate maximal variance. The step-size in each iteration is obtained by incorporating the stabilized Barzilai-Borwein method with the gradient optimization. This enables DSPCA to be processed with low computational complexity while maintaining a high convergence speed. Computer simulation with two types of datasets displays that the proposed scheme consistently outperforms the representative DC schemes in terms of reconstruction accuracy of original data and explained variance.

Energy Efficient Data Compression in Cloud Based IoT

In this study, an adaptive data compression scheme (ADCS) is proposed for efficiently controlling the IoT device compression rate and energy consumption in the cloud based IoT network. The ADCS consists of two data compression schemes, the sensor Lempel-Ziv-Welch (S-LZW) scheme and the sequential lossless entropy compression (S-LEC) scheme. In Auto state, the ADCS can select the appropriate energy efficient data compression scheme for each IoT device, while taking into consideration the IoT device's processing capability, the available energy in each IoT device battery, and the amount of compression power. Our proposed scheme has been developed using mixed integer linear programming. The result verifies that the proposed ADCS scheme saves power by an average of 40% compared to the non-compression scheme (NCS) due to reducing the traffic load and the number of hops in the network, which leads to an ability to handle higher traffic demands and increasing the lifetime of IoT devices by 50% compared to NCS systems.

On Choosing Structure for a Machine Learning-based Reaction Force Predictor for Walking Robots

This paper focuses on the topic of contact reaction prediction for walking robots, namely on the analysis of performances on different structures of the machine-learning-based predictors. Predicting reaction forces is important due to the fact that it can allow us to retrieve a simplified model of the contact scenario, as was proposed in the literature preciously. This allows to turn the problem of contact model identification into a data collection and processing problem. In order to do it effectively, both a data compression (feature extraction) and regression strategies might be used. This research provides an analysis of both with respect to the discussed problem.

Real-time energy data compression strategy for reducing data traffic based on smart grid AMI networks

In the future, the Internet of things (IoT) may provide huge volumes of data. Smart grids are a class of IoT electricity distribution systems that can control bidirectional energy flows between consumers and service providers (Barman et al. “IOT Based Smart Energy Meter for Efficient Energy Utilization in Smart Grid”, 978-1-5386-4769-1 1831.00, 2018 IEEE). A typical smart grid features an advanced metering infrastructure (AMI), which automatically collects meter data from widely distributed sensors. A utility company that intends to use AMI must deploy smart meters, data concentrator units, and a meter data management system (MDMS). Concentrators collect ubiquitous messages from smart meters and transmit aggregated data to their MDMS. Although ubiquitous messages may enhance the efficiency of some electricity grids, any excessive volume of messages causes data congestion. This research considers the Reference Energy Disaggregation Data Set. Our proposed algorithm can compress meter data efficiently. Our contributions are as follows: First, thus far, numerous researchers have attempted to address smart grid problems with AMI systems, and here, we provide a relatively complete review of these attempts. Second, this paper presents a strategy to analyze this problem and propose a compression algorithm to compress meter data. This proposed algorithm combines several existing compression algorithms and operates from 2 to 10% more efficiently than previously published algorithms.

New variable-length data compression scheme for solution representation of meta-heuristics

Solution representation is an important aspect of the development of a meta-heuristic. However, most meta-heuristics focus on the algorithm aspect rather than the solution representation. As the scale of the optimization problem under study rises, the solution representation becomes ever more crucial because the computational performance tends to deteriorate for three reasons: (1) increased landscape complexity, (2) exponential growth of search space, and (3) costly function evaluation. In this study, data compression of solution representation is explored to improve the performance of meta-heuristics in terms of compression efficiency and solution quality. A proposed new data compression scheme – variable-length solution representation compression for meta-heuristics – is developed and tested against benchmarked problem instances from the quadratic assignment problem library (QAPLIB) and additional large problem instances. The analyzed results indicate that the compressed solution representation scheme under a meta-heuristic framework performs well against the QAPLIB and large problem instances with less space complexity and computation time.

Redundant Gaussian dictionary in compressed sensing for ambulatory photoplethysmography monitoring

With the rapid development of wireless communication technology, wireless photoplethysmography sensors have emerged and been applied in clinics or daily healthcare. However, energy efficiency is still a major obstacle for such devices in long-term use. Data compression can reduce airtime over energy-hungry wireless links and improve the energy efficiency of the sensors. In this study, we explored a redundant dictionary-based compressed sensing scheme for ambulatory photoplethysmography data compression. We first proposed a redundant Gaussian dictionary, which was the column combination of circle shift atoms of Gaussian function. Besides, we came up with a simple method for dictionary generation. To demonstrate its efficiency improvement for compactly representing photoplethysmography signal, the proposed dictionary was compared with the dictionaries of discrete cosine transform, wavelet transform, Fourier transform, and Gabor transform in experiments using the database of IEEE Signal Processing Cup 2015. Results indicated the recovered photoplethysmography signals were essentially undistorted by the combination of redundant Gaussian dictionary, sparse binary measurement matrix, and smoothed l0 pseudo norm recovery algorithm. The average percentage of root-mean-square difference was 8.00%, and its ratio over 9% was 0.32 when the compression ratio was equal to 30%. Furthermore, after a 20 Hz anti-aliasing low-pass filtering, perfect full-band signals were recovered by using our method, where every frame satisfied the requirement of non-distortion diagnosis at 70% data reduction. The outstanding performance and low-complexity indicate the proposed redundant Gaussian dictionary is an excellent choice for using in compressed sensing enabled ambulatory photoplethysmography.

A data reduction and compression description for high throughput time-resolved electron microscopy

Fast, direct electron detectors have significantly improved the spatio-temporal resolution of electron microscopy movies. Preserving both spatial and temporal resolution in extended observations, however, requires storing prohibitively large amounts of data. Here, we describe an efficient and flexible data reduction and compression scheme (ReCoDe) that retains both spatial and temporal resolution by preserving individual electron events. Running ReCoDe on a workstation we demonstrate on-the-fly reduction and compression of raw data streaming off a detector at 3 GB/s, for hours of uninterrupted data collection. The output was 100-fold smaller than the raw data and saved directly onto network-attached storage drives over a 10 GbE connection. We discuss calibration techniques that support electron detection and counting (e.g., estimate electron backscattering rates, false positive rates, and data compressibility), and novel data analysis methods enabled by ReCoDe (e.g., recalibration of data post acquisition, and accurate estimation of coincidence loss).

Fast digital lossy compression for X-ray ptychographic data.

Increases in X-ray brightness from synchrotron light sources lead to a requirement for higher frame rates from hybrid pixel array detectors (HPADs), while also favoring charge integration over photon counting. However, transfer of the full uncompressed data will begin to constrain detector design, as well as limit the achievable continuous frame rate. Here a data compression scheme that is easy to implement in a HPAD's application-specific integrated circuit (ASIC) is described, and how different degrees of compression affect image quality in ptychography, a commonly employed coherent imaging method, is examined. Using adaptive encoding quantization, it is shown in simulations that one can digitize signals up to 16383 photons per pixel (corresponding to 14 bits of information) using only 8 or 9 bits for data transfer, with negligible effect on the reconstructed image.

An Improved Gravitational Clustering Based on Local Density

Gravitational clustering algorithm (Gravc) is a novel and excellent dynamic clustering algorithm that can accurately cluster complex dataset with arbitrary shape and distribution. However, high time complexity is a key challenge to the gravitational clustering algorithm. To solve this problem, an improved gravitational clustering algorithm based on the local density is proposed in this paper, called FastGravc. The main contributions of this paper are as follows. First of all, a local density-based data compression strategy is designed to reduce the number of data objects and the number of neighbors of each object participating in the gravitational clustering algorithm. Secondly, the traditional gravity model is optimized to adapt to the quality differences of different objects caused by data compression strategy. And then, the improved gravitational clustering algorithm FastGravc is proposed by integrating the above optimization strategies. Finally, extensive experimental results on synthetic and real-world datasets verify the effectiveness and efficiency of FastGravc algorithm.

Strategies for on-chip digital data compression for X-ray pixel detectors

The continued desire for X-ray pixel detectors with higher frame rates will stress the ability of application-specific integrated circuit (ASIC) designers to provide sufficient off-chip bandwidth to reach continuous frame rates in the 1 MHz regime. To move from the current 10 kHz to the 1 MHz frame rate regime, ASIC designers will continue to pack as many power-hungry high-speed transceivers at the periphery of the ASIC as possible. In this paper, however, we present new strategies to make the most efficient use of the off-chip bandwidth by utilizing data compression schemes for X-ray photon-counting and charge-integrating pixel detectors. In particular, we describe a novel in-pixel compression scheme that converts from analog to digital converter units to encoded photon counts near the photon Poisson noise level and achieves a compression ratio of > 1.5× independent of the dataset. In addition, we describe a simple yet efficient zero-suppression compression scheme called “zeromask” (ZM) located at the ASIC's edge before streaming data off the ASIC chip. ZM achieves average compression ratios of > 4×, > 7×, and > 8× for high-energy X-ray diffraction, ptychography, and X-ray photon correlation spectroscopy datasets, respectively. We present the conceptual designs, register-transfer level block diagrams, and the physical ASIC implementation of these compression schemes in 65 nm CMOS. When combined, these two digital compression schemes could increase the effective off-chip bandwidth by a factor of 6–12×.

A Reliable Data Compression Scheme in Sensor-Cloud Systems Based on Edge Computing

The rapid development of the IoT and cloud computing has spawned a new network structure - sensor-cloud system (SCS) where sensors, sensor networks, and cloud computing are integrated to perform data sensing, collection, transmission, and decision making. The large-scale deployment of sensors creates a massive amount of data, posing new challenges in data transmission and storage. As an intermediate platform between IoT and cloud platforms, edge computing provides IoT with data collection, processing, and scheduling services. This paper proposes a hybrid data compression scheme that incorporates lossy and lossless compression in SCS based on edge computing to address the increasing challenges. Moreover, we propose a new reliable lossy compression algorithm DFan, based on the simplified Fan algorithm with a high compression ratio (CR). By introducing the data tolerable deviation, DFan transforms single-factor decision-making into multi-factor decision-making, reducing the error of lossy compression. Through experiments on IntelLab and MIT-BIH datasets, the proposed hybrid data compression scheme achieves an overall CR of 4.21× and 3.88×, respectively. The lossy CR of DFan is 6.42× and 5.1×, respectively, and the Percentage RMS Difference (PRD) caused by lossy compression is 0.27% and 0.56%, respectively. The hybrid compression scheme, high compression ratio, and reliable data restoration make this scheme attractive to the data processing of sensors in SCS.

A new efficient encryption and decryption method using a Lossless Data Compression Scheme

A new efficient encryption and decryption method using a Lossless Data Compression Scheme

Edge Learning with Timeliness Constraints: Challenges and Solutions

Future machine learning (ML) powered applications, such as autonomous driving and augmented reality, involve training and inference tasks with timeliness requirements and are communication- and computation-intensive, which demands the edge learning framework. The real-time requirements drive us to go beyond accuracy for ML. In this article, we introduce the concept of timely edge learning, aiming to achieve accurate training and inference while minimizing the communication and computation delay. We discuss key challenges and propose corresponding solutions from data, model, and resource management perspectives to meet the timeliness requirements. In particular, for edge training, we argue that the total training delay rather than rounds should be considered, and propose data or model compression, and joint device scheduling and resource management schemes for both centralized training and federated learning systems. For edge inference, we explore the dependency between accuracy and delay for communication and computation, and propose dynamic data compression and flexible pruning schemes. Two case studies show that the timeliness performance, including the training accuracy under a given delay budget and the completion ratio of inference tasks within deadline, are highly improved with the proposed solutions.

High-Throughput Lossy-to-Lossless 3D Image Compression.

The rapid increase in medical and biomedical image acquisition rates has opened up new avenues for image analysis, but has also introduced formidable challenges. This is evident, for example, in selective plane illumination microscopy where acquisition rates of about 1-4 GB/s sustained over several days have redefined the scale of I/O bandwidth required by image analysis tools. Although the effective bandwidth could, principally, be increased by lossy-to-lossless data compression, this is of limited value in practice due to the high computational demand of current schemes such as JPEG2000 that reach compression throughput of one order of magnitude below that of image acquisition. Here we present a novel lossy-to-lossless data compression scheme with a compression throughput well above 4 GB/s and compression rates and rate-distortion curves competitive with those achieved by JPEG2000 and JP3D.

David Annetts’ best of Exploration Geophysics

When airborne electromagnetic (AEM) data is acquired as a streamed or time-series data set, the great redundancy in the data favours compression as a first step in processing. Traditional data compression schemes are time windowing and spatial averaging. An alternative, more efficient data compression scheme is to transform time or frequency domain data to time-constant tau space, which has the effect of removing the waveform dependence of the AEM response.When there are many local anomalies and a variable background, the next stage of rapid processing is to transform the response to a conductivity-depth image (CDI) to facilitate geological interpretation of the background response. Use of the full time range of recorded data, particularly the inclusion of on-time data, improves the stability of the CDI process.The final AEM data processing step for mineral exploration is to assess the likelihood that any local anomaly corresponds to a desired economic target. This step involves the extraction of target geometry and conductivity information from the AEM data. The only economically feasible route at the present time is to parameterise both the data (using inductive and resistive limits) and the model to allow inversion of the local anomalies. A fit to one or two platelike conductors can be achieved in seconds; fits to a blocklike body take minutes on a fast PC. A significant research challenge remains to speed up and stabilise this process.

Transfer Function-Guided Saliency-Aware Compression for Transmitting Volumetric Data

We introduce a transfer-function-guided three-dimensional (3-D) block-based saliency-aware compression scheme for volumetric data that is both content and spatially scalable. Salient 3-D volumetric blocks are identified and weighted with the help of a transfer function which is used to render the data. We describe our method in the form of a framework for processing, progressive transmission, and visualization of volumetric data on a target device, such as a mobile device with limited computational resources. In particular, we address the transmission bottleneck incurred when transferring 3-D volumetric data. Identified salient regions are progressively transmitted to the target device. The received data are rendered progressively in the respective order with a predefined or user-defined transfer function. Our method is developed with medical applications in mind, where preservation of all information is essential for clinical diagnosis. Because our method is integrated into a resolution scalable coding scheme with an integer wavelet transform of the image, it allows the rendering of each significant region at a different resolution up to fully lossless reconstruction. We perform a thorough qualitative and quantitative evaluation of the saliency detection method and the resulting saliency-aware compression schemes. Our results show reduced error in representation of the volumetric data with our method.

Gradient-based adaptive modeling for IoT data transmission reduction

Spatial and temporal correlation between sensor observations in an Internet of Things environment can be exploited to eliminate unnecessary transmissions. Transmitting less data certainly contributes to meet the growing need for energy-saving and robust transmissions, thus prolong the lifespan of the entire WSN. Spatiotemporal correlation-based dual prediction (DP) and data compression (DC) schemes aim to reduce the amount of data transmission while ensuring data accuracy. In practice, however, the existing methods restrict the stability of the system when the model hyper-parameters are uncertain. Thus adaptive model has lately attracted extensive attention for the development of resource-constrained WSN. In this paper, we propose a gradient-based adaptive model that implements both schemes in a two-tier data reduction framework. To the best of our knowledge, the proposed scheme is the first attempt to introduce adaptiveness into both the DP and DC schemes by using a simple gradient optimization method. Gradient-based Optimal Step-size LMS (GO-LMS) is introduced to make the DP aspects adaptive, while a Gradient-based Adaptive PCA (GA-PCA) approach is used for the DC aspects. The Barzilai–Borwein method is incorporated into the gradient optimization to enable adaptive computation of the step-size for each iteration. Through extensive simulations, the developed framework was found to outperform other state-of-the-art schemes in terms of both the transmission reduction ratio and data recovery accuracy.

Privacy-Preserving Multilayer In-Band Network Telemetry and Data Analytics: For Safety, Please do Not Report Plaintext Data

With the evolution of Internet infrastructure and network services, multilayer in-band network telemetry (ML-INT) and data analytics (DA) have been considered as key enabling techniques to realize real-time and fine-grained network monitoring, especially for backbone IP-over-Optical networks. However, the existing ML-INT&DA systems have privacy and security issues, because plaintext ML-INT data is reported from the data plane and gets analyzed in the control plane. In this work, we address these issues by designing a privacy-preserving ML-INT&DA system for IP-over-Optical networks. We first leverage vector homomorphic encryption (VHE) to design a lightweight encryption scheme, which overcomes the security breaches due to eavesdropping and preserves the delicate correlations buried in multi-dimensional ML-INT data. Then, we develop an effective data compression scheme to further encode the encrypted ML-INT data and make the results suitable for hash-based signature. The signature is for data certification and enables the DA in the control plane to verify the integrity of received ML-INT data. Hence, the threats from data tampering are removed. Next, we architect a deep learning (DL) model that can directly operate on encrypted ML-INT data for anomaly detection. Finally, we implement the proposed ML-INT&DA system, and experimentally demonstrate its effectiveness in a real IP over elastic optical network (IP-over-EON) testbed, whose key elements, i.e., optical line system (OLS), bandwidth-variable wavelength-selective switches (BV-WSS’) and programmable data plane (PDP) switches, are all commercial products.