Lossy Compression Algorithm Research Articles

A New Algorithm for the On-Board Compression of Hyperspectral Images

Hyperspectral sensors are able to provide information that is useful for many different applications. However, the huge amounts of data collected by these sensors are not exempt of drawbacks, especially in remote sensing environments where the hyperspectral images are collected on-board satellites and need to be transferred to the earth’s surface. In this situation, an efficient compression of the hyperspectral images is mandatory in order to save bandwidth and storage space. Lossless compression algorithms have been traditionally preferred, in order to preserve all the information present in the hyperspectral cube for scientific purposes, despite their limited compression ratio. Nevertheless, the increment in the data-rate of the new-generation sensors is making more critical the necessity of obtaining higher compression ratios, making it necessary to use lossy compression techniques. A new transform-based lossy compression algorithm, namely Lossy Compression Algorithm for Hyperspectral Image Systems (HyperLCA), is proposed in this manuscript. This compressor has been developed for achieving high compression ratios with a good compression performance at a reasonable computational burden. An extensive amount of experiments have been performed in order to evaluate the goodness of the proposed HyperLCA compressor using different calibrated and uncalibrated hyperspectral images from the AVIRIS and Hyperion sensors. The results provided by the proposed HyperLCA compressor have been evaluated and compared against those produced by the most relevant state-of-the-art compression solutions. The theoretical and experimental evidence indicates that the proposed algorithm represents an excellent option for lossy compressing hyperspectral images, especially for applications where the available computational resources are limited, such as on-board scenarios.

Reducing storage of global wind ensembles with stochastic generators

Wind has the potential to make a significant contribution to future energy resources. Locating the sources of this renewable energy on a global scale is however extremely challenging, given the difficulty to store very large data sets generated by modern computer models. We propose a statistical model that aims at reproducing the data-generating mechanism of an ensemble of runs via a Stochastic Generator (SG) of global annual wind data. We introduce an evolutionary spectrum approach with spatially varying parameters based on large-scale geographical descriptors such as altitude to better account for different regimes across the Earth's orography. We consider a multi-step conditional likelihood approach to estimate the parameters that explicitly accounts for nonstationary features while also balancing memory storage and distributed computation. We apply the proposed model to more than 18 million points of yearly global wind speed. The proposed SG requires orders of magnitude less storage for generating surrogate ensemble members from wind than does creating additional wind fields from the climate model, even if an effective lossy data compression algorithm is applied to the simulation output.

Lossy Compression for Embedded Computer Vision Systems

Computer vision applications are rapidly gaining popularity in embedded systems, which typically involve a difficult tradeoff between vision performance and energy consumption under a constraint of real-time processing throughput. Recently, hardware (FPGA and ASIC-based) implementations have emerged, which significantly improves the energy efficiency of vision computation. These implementations, however, often involve intensive memory traffic that retains a significant portion of energy consumption at the system level. To address this issue, we are the first researchers to present a lossy compression framework to exploit the tradeoff between vision performance and memory traffic for input images. To meet various requirements for memory access patterns in the vision system, a line-to-block format conversion is designed for the framework. Differential pulse-code modulation-based gradient-oriented quantization is developed as the lossy compression algorithm. We also present its hardware design that supports up to 12-scale 1080p@60fps real-time processing. For histogram of oriented gradient-based deformable part models on VOC2007, the proposed framework achieves a 49.6%–60.5% memory traffic reduction at a detection rate degradation of 0.05%–0.34%. For AlexNet on ImageNet, memory traffic reduction achieves up to 60.8% with less than 0.61% classification rate degradation. Compared with the power consumption reduction from memory traffic, the overhead involved for the proposed input image compression is less than 5%.

Mutual Information Correlation with Human Vision in Medical Image Compression.

Background: The lossy compression algorithm produces different results in various con-trasts areas. Low contrast area image quality declines greater than that of high contrast regions using equal compression ratio. These results were obtained in a subjective study. The objective image quali-ty metrics are more effective if the calculation method is more closely related to the human vision re-sults.Methods: This study first measured the PSNR and MI for discrimination between different contrast areas responding to lossy image compression in a SMPTE electronic pattern. The MI was consistent with human vision results in SMPTE electronic phantom but PSNR was not. The measurement was also applied to compressed medical images in different contrast cropping regions.Results: The MI was found to be close to human vision in CT and MR but not CRX. Both weighted PSNR and weighted MI were created to respond to the gray value and the contrast areas affected the quality estimation.Conclusion: The W-PSNR and W-MI showed that they can discriminate between different contrast areas using image compression ratios and the series of lines are equal to the contrast values and better than the tranditional approach. The W-MI measures were found to perform better than W-PSNR and can be used as an image quality index.

Towards Decoupling the Selection of Compression Algorithms from Quality Constraints – An Investigation of Lossy Compression Efficiency

Data intense scientific domains use data compression to reduce the storage space needed. Lossless data compression preserves information accurately but lossy data compression can achieve much higher compression rates depending on the tolerable error margins. There are many ways of defining precision and to exploit this knowledge, therefore, the field of lossy compression is subject to active research. From the perspective of a scientist, the qualitative definition about the implied loss of data precision should only matter. With the Scientific Compression Library (SCIL), we are developing a meta-compressor that allows users to define various quantities for acceptable error and expected performance behavior. The library then picks a suitable chain of algorithms yielding the user’s requirements, the ongoing work is a preliminary stage for the design of an adaptive selector. This approach is a crucial step towards a scientifically safe use of much-needed lossy data compression, because it disentangles the tasks of determining scientific characteristics of tolerable noise, from the task of determining an optimal compression strategy. Future algorithms can be used without changing application code. In this paper, we evaluate various lossy compression algorithms for compressing different scientific datasets (Isabel, ECHAM6), and focus on the analysis of synthetically created data that serves as blueprint for many observed datasets. We also briefly describe the available quantitiesof SCIL to define data precision and introduce two efficient compression algorithms for individualdata points. This shows that the best algorithm depends on user settings and data properties.

An Improved Compression Algorithm for Hyperspectral Images based on DVAT-SVD

Hyperspectral image (HSI) compression has recently become a popular research area in remote sensing applications. However, existing clustering and compression techniques are not based on the intensity value and work by forming static rules. To overcome these issues, a novel lossy (nearly lossless) compression algorithm based on the distributed visual assessment cluster tendency (DVAT) – singular value decomposition (SVD) technique is proposed. At first, the given HSI image is preprocessed by cellular automata filtering to remove the white Gaussian and speckle noise. After that, the preprocessed image is split into bands, which are formed to a distance matrix. Here, the distances between the image bands are predicted to identify pixels with related intensity values. Hence, the DVAT technique is implemented to cluster the nearest distance images. Then, the label of the cluster is optimally selected for segregating the band into a band index. Consequently, the cluster index is formed and the image matrix is clustered based on the label features. After that, the SVD technique is applied to encode the HSI band image. Therefore, the discrete wavelet transform technique is applied for transformation and the transformed cell redundancy check technique is employed to encode the image band. Then, the image is reconstructed by reversing the above processes. The major advantages of the proposed method comprise a cleared clustered output, a correct index value, and the reduction of data losses during compression. During experimental results, the performance of the proposed method was evaluated in terms of the compression ratio, the mean-square error, and the peak signal-to-noise ratio.

Lossy image compression based on prediction error and vector quantisation

Lossy image compression has been gaining importance in recent years due to the enormous increase in the volume of image data employed for Internet and other applications. In a lossy compression, it is essential to ensure that the compression process does not affect the quality of the image adversely. The performance of a lossy compression algorithm is evaluated based on two conflicting parameters, namely, compression ratio and image quality which is usually measured by PSNR values. In this paper, a new lossy compression method denoted as PE-VQ method is proposed which employs prediction error and vector quantization (VQ) concepts. An optimum codebook is generated by using a combination of two algorithms, namely, artificial bee colony and genetic algorithms. The performance of the proposed PE-VQ method is evaluated in terms of compression ratio (CR) and PSNR values using three different types of databases, namely, CLEF med 2009, Corel 1 k and standard images (Lena, Barbara etc.). Experiments are conducted for different codebook sizes and for different CR values. The results show that for a given CR, the proposed PE-VQ technique yields higher PSNR value compared to the existing algorithms. It is also shown that higher PSNR values can be obtained by applying VQ on prediction errors rather than on the original image pixels.

Analyzing the Effect and Performance of Lossy Compression on Aeroacoustic Simulation of Gas Injector

Computational fluid dynamic simulations involve large state data, leading to performance degradation due to data transfer times, while requiring large disk space. To alleviate the situation, an adaptive lossy compression algorithm has been developed, which is based on regions of interest. This algorithm uses prediction-based compression and exploits the temporal coherence between subsequent simulation frames. The difference between the actual value and the predicted value is adaptively quantized and encoded. The adaptation is in line with user requirements, that consist of the acceptable inaccuracy, the regions of interest and the required compression throughput. The data compression algorithm was evaluated with simulation data obtained by the discontinuous Galerkin spectral element method. We analyzed the performance, compression ratio and inaccuracy introduced by the lossy compression algorithm. The post processing analysis shows high compression ratios, with reasonable quantization errors.

Adaptive Vector Quantization for Lossy Compression of Image Sequences

In this work, we present a scheme for the lossy compression of image sequences, based on the Adaptive Vector Quantization (AVQ) algorithm. The AVQ algorithm is a lossy compression algorithm for grayscale images, which processes the input data in a single-pass, by using the properties of the vector quantization to approximate data. First, we review the key aspects of the AVQ algorithm and, subsequently, we outline the basic concepts and the design choices behind the proposed scheme. Finally, we report the experimental results, which highlight an improvement in compression performances when our scheme is compared with the AVQ algorithm.

Cluster-Based Quality-Aware Adaptive Data Compression for Streaming Data

Wireless sensor networks (WSNs) are widely applied in data collection applications. Energy efficiency is one of the most important design goals of WSNs. In this article, we examine the tradeoffs between the energy efficiency and the data quality. First, four attributes used to evaluate data quality are formally defined. Then, we propose a novel data compression algorithm, Quality-Aware Adaptive data Compression (QAAC), to reduce the amount of data communication to save energy. QAAC utilizes an adaptive clustering algorithm to build clusters from dataset; then a code for each cluster is generated and stored in a Huffman encoding tree. The encoding algorithm encodes the original dataset based on the Haffman encoding tree. An improvement algorithm is also designed to reduce the information loss when data are compressed. After the encoded data, the Huffman encoding tree and parameters used in the improvement algorithm have been received at the sink, a decompression algorithm is used to retrieve the approximation of the original dataset. The performance evaluation shows that QAAC is efficient and achieves a much higher compression ratio than lossy and lossless compression algorithms, while it has much smaller information loss than lossy compression algorithms.

Comparative study of lossy and lossless data compression in distributed optical fiber sensing systems

Typical fully distributed optical fiber sensors (DOFS) with dozens of kilometers are equivalent to tens of thousands of point sensors along the whole monitoring line, which means tens of thousands of data will be generated for one pulse launching period. Therefore, in an all-day nonstop monitoring, large volumes of data are created thereby triggering the demand for large storage space and high speed for data transmission. In addition, when the monitoring length and channel numbers increase, the data also increase extensively. The task of mitigating large volumes of data accumulation, large storage capacity, and high-speed data transmission is, therefore, the aim of this paper. To demonstrate our idea, we carried out a comparative study of two lossless methods, Huffman and Lempel Ziv Welch (LZW), with a lossy data compression algorithm, fast wavelet transform (FWT) based on three distinctive DOFS sensing data, such as Φ-OTDR, P-OTDR, and B-OTDA. Our results demonstrated that FWT yielded the best compression ratio with good consumption time, irrespective of errors in signal construction of the three DOFS data. Our outcomes indicate the promising potentials of FWT which makes it more suitable, reliable, and convenient for real-time compression of the DOFS data. Finally, it was observed that differences in the DOFS data structure have some influence on both the compression ratio and computational cost.

Spline approximation-based data compression for sensor arrays in the wireless hydrologic monitoring system

A sensor array produces lots of bits of data every sample period, which may cause a heavy burden on the long-distance wireless data transmission, especially in the scenarios of wireless sensor networks. A lossy but error-bounded sensor array data compression algorithm is proposed, in which the major part of the sensor array data are approximated by the Catmull-Rom spline curve and the residual errors are quantized and encoded with Huffman coding. The performance of this algorithm has been evaluated with a real data set from the wireless hydrologic monitoring system deployed in Qinhuangdao Port of China. The results show that the algorithm performs well for the hydrologic sensor array data.

SURF: Subject-Adaptive Unsupervised ECG Signal Compression for Wearable Fitness Monitors

Recent advances in wearable devices allow non-invasive and inexpensive collection of biomedical signals including electrocardiogram (ECG), blood pressure, respiration, among others. Collection and processing of various biomarkers are expected to facilitate preventive healthcare through personalized medical applications. Since wearables are based on size- and resource-constrained hardware, and are battery operated, they need to run lightweight algorithms to efficiently manage energy and memory. To accomplish this goal, this paper proposes SURF, a subject-adaptive unsupervised signal compressor for wearable fitness monitors. The core idea is to perform a specialized lossy compression algorithm on the ECG signal at the source (wearable device), to decrease the energy consumption required for wireless transmission and thus prolong the battery lifetime. SURF leverages unsupervised learning techniques to build and maintain, at runtime, a subject-adaptive dictionary without requiring any prior information on the signal. Dictionaries are constructed within a suitable feature space, allowing the addition and removal of code words according to the signal’s dynamics (for given target fidelity and energy consumption objectives). Extensive performance evaluation results, obtained with reference ECG traces and with our own measurements from a commercial wearable wireless monitor, show the superiority of SURF against state-of-the-art techniques, including: 1) compression ratios up to 90-times; 2) reconstruction errors between 2% and 7% of the signal’s range (depending on the amount of compression sought); and 3) reduction in energy consumption of up to two orders of magnitude with respect to sending the signal uncompressed, while preserving its morphology. SURF, with artifact prone ECG signals, allows for typical compression efficiencies (CE) in the range $\text {CE} \in [{40,50}]$ , which means that the data rate of 3 kbit/s that would be required to send the uncompressed ECG trace is lowered to 60 and 75 bit/s for CE = 40 and CE = 50, respectively.

Block-based KLT compression for multispectral images

An efficient lossy compression algorithm for multispectral images based on block Karhunen–Loève transform (KLT) is proposed. First, a two-dimensional discrete wavelet transform is performed on each band of multispectral images to remove the spatial correlation. Subsequently, each band is partitioned into non-overlapping blocks of the same size, and the transform coefficients of each block in the wavelet domain are treated as a single object. Blocks that are co-located in the spectral orientation are affected by an adaptive Karhunen–Loève transform to remove their spectral correlation. Finally, embedded block coding with optimized truncation is performed on all principal components to generate the final bit-stream. Experimental results show that the proposed algorithm, based on block KLT, outperforms the algorithm based on global KLT, without significant increase of complexity.

A block Karhunen–Loève Transform based efficient lossy compression algorithm for optical remote sensing imagery

In present study, a block Karhunen–Loève Transform (KLT) based efficient lossy compression algorithm for optical remote sensing imagery is proposed. A Discrete Wavelet Transform (DWT) is performed on each band of the imagery to remove the spatial correlation. Each band of the imagery is decomposed into non-overlapping blocks of similar size and the transform coefficients of each block in the wavelet domain are treated as a single object. A rate-distortion optimization is introduced to perform rate allocation of multiple bands. Each band is partitioned into code-blocks. The embedded block coding with optimized truncation algorithm is executed on the code-blocks to produce final bit-stream. The complexity of the proposed algorithm is compared with global KLT-DWT. The result reports the complexity of JPEG 2000 (Part 1) is lowest with encoding time 114 ms as compared to global KLT-DWT (N = 1024), global DWT-KLT (N = 1024), block-based DWT-KLT (N = 512), block-based DWT-KLT (N = 256), block-based DWT-KLT (N = 128), block-based DWT-KLT (N = 64) and block-based DWT-KLT (N = 32).

Hybrid encoding algorithm for real time compressed electrocardiogram acquisition

Abstract Tele-healthcare systems have shown a great promise toward flexibility in wireless medical signal collection and increased mobility of the patient to achieve early diagnosis. We describe a simple, lossy compression algorithm for single lead Electrocardiograms (ECG) using a hybrid encoding strategy, which can be utilized for patient monitoring application. It detects the ECG beats in real time, divides it into equal size ‘plain’ or ‘complex’ blocks based on complexity measure and allocate bits using two different principles to optimize the entropy (bits per sample efficiency) in the encoded bit-stream. With MIT BIH arrhythmia data at 1 kHz sampling and 10-bit quantization, the algorithm achieved an average CR and PRDN of 10.99 and 3.27 respectively, with average bit per sample value of 1.49. The average distortion in clinical signatures using PTB diagnostic ECG data was found to be less than 3%, which is clinically acceptable.

Reducing disk storage of full-3D seismic waveform tomography (F3DT) through lossy online compression

Full-3D seismic waveform tomography (F3DT) is the latest seismic tomography technique that can assimilate broadband, multi-component seismic waveform observations into high-resolution 3D subsurface seismic structure models. The main drawback in the current F3DT implementation, in particular the scattering-integral implementation (F3DT-SI), is the high disk storage cost and the associated I/O overhead of archiving the 4D space-time wavefields of the receiver- or source-side strain tensors. The strain tensor fields are needed for computing the data sensitivity kernels, which are used for constructing the Jacobian matrix in the Gauss–Newton optimization algorithm. In this study, we have successfully integrated a lossy compression algorithm into our F3DT-SI workflow to significantly reduce the disk space for storing the strain tensor fields. The compressor supports a user-specified tolerance for bounding the error, and can be integrated into our finite-difference wave-propagation simulation code used for computing the strain fields. The decompressor can be integrated into the kernel calculation code that reads the strain fields from the disk and compute the data sensitivity kernels. During the wave-propagation simulations, we compress the strain fields before writing them to the disk. To compute the data sensitivity kernels, we read the compressed strain fields from the disk and decompress them before using them in kernel calculations. Experiments using a realistic dataset in our California statewide F3DT project have shown that we can reduce the strain-field disk storage by at least an order of magnitude with acceptable loss, and also improve the overall I/O performance of the entire F3DT-SI workflow significantly. The integration of the lossy online compressor may potentially open up the possibilities of the wide adoption of F3DT-SI in routine seismic tomography practices in the near future.

Adaptive Dictionary Reconstruction for Compressed Sensing of ECG Signals.

This paper proposes a novel adaptive dictionary (AD) reconstruction scheme to improve the performance of compressed sensing (CS) with electrocardiogram signals (ECG). The method is based on the use of multiple dictionaries, created using dictionary learning (DL) techniques for CS signal reconstruction. The modified reconstruction framework is a two-stage process that leverages information about the signal from an initial signal reconstruction stage. By identifying whether a QRS complex is present and if so, determining a location estimate of the QRS, the most appropriate dictionary is selected and a second stage more refined signal reconstruction can be obtained. The performance of the proposed algorithm is compared with state-of-the-art CS implementations in the literature, as well as the set partitioning in hierarchical trees (SPIHT) wavelet-based lossy compression algorithm. The results indicate that the proposed reconstruction scheme outperforms all existing CS implementations in terms of signal fidelity at each compression ratio tested. The performance of the proposed approach also compares favorably with SPIHT in terms of signal reconstruction quality. Furthermore, an analysis of the overall power consumption of the proposed ECG compression framework as would be used in a body area network (BAN) demonstrates positive results for the proposed CS approach when compared with existing CS techniques and SPIHT.

An approximate dynamic programming approach for improving accuracy of lossy data compression by Bloom filters

Bloom filters are a data structure for storing data in a compressed form. They offer excellent space and time efficiency at the cost of some loss of accuracy (so-called lossy compression). This work presents a yes–no Bloom filter, which as a data structure consisting of two parts: the yes-filter which is a standard Bloom filter and the no-filter which is another Bloom filter whose purpose is to represent those objects that were recognized incorrectly by the yes-filter (that is, to recognize the false positives of the yes-filter). By querying the no-filter after an object has been recognized by the yes-filter, we get a chance of rejecting it, which improves the accuracy of data recognition in comparison with the standard Bloom filter of the same total length. A further increase in accuracy is possible if one chooses objects to include in the no-filter so that the no-filter recognizes as many as possible false positives but no true positives, thus producing the most accurate yes–no Bloom filter among all yes–no Bloom filters. This paper studies how optimization techniques can be used to maximize the number of false positives recognized by the no-filter, with the constraint being that it should recognize no true positives. To achieve this aim, an Integer Linear Program (ILP) is proposed for the optimal selection of false positives. In practice the problem size is normally large leading to intractable optimal solution. Considering the similarity of the ILP with the Multidimensional Knapsack Problem, an Approximate Dynamic Programming (ADP) model is developed making use of a reduced ILP for the value function approximation. Numerical results show the ADP model works best comparing with a number of heuristics as well as the CPLEX built-in solver (B&B), and this is what can be recommended for use in yes–no Bloom filters. In a wider context of the study of lossy compression algorithms, our research is an example showing how the arsenal of optimization methods can be applied to improving the accuracy of compressed data.

Lossy Compression for Wireless Seismic Data Acquisition

In this paper, we rigorously compare compressive sampling (CS) to four state of the art, on-mote, lossy compression algorithms [ $K$ -run-length encoding (KRLE), lightweight temporal compression (LTC), wavelet quantization thresholding and run-length encoding (WQTR), and a low-pass filtered fast Fourier transform (FFT)]. Specifically, we first simulate lossy compression on two real-world seismic data sets, and we then evaluate algorithm performance using implementations on real hardware. In terms of compression ratios, recovered signal error, power consumption, on-mote execution runtime, and classification accuracy of a seismic event detection task (on decompressed signals), results show that CS performs comparable to (and in many cases better than) the other algorithms evaluated. A main benefit to users is that CS, a lightweight and nonadaptive compression technique, can guarantee a desired level of compression performance (and thus, radio usage and power consumption) without subjugating recovered signal quality. Our contribution is a novel and rigorous comparison of five state of the art, on-mote, lossy compression algorithms in simulation on real-world data sets and in implementations on hardware.