Data Compression Techniques Research Articles

Compression of complex-valued hologram data using line index coding and image compression standards

Human Visual System (HVS) records 3D information of a scene while photography of a scene can record only 2D information. A hologram can record the 3D information of a scene as that of HVS and reproduce the same. Hologram recording in digital form requires large amount of memory and opens a window to challenging area of Hologram data compression techniques. The hologram records the depth information of a scene in the phase of complex data. Complex data represented in both rectangular and polar form are exploited to produce Line Index Coding (LIC) process. Before Line Index Coding, normalization of hologram is done to produce normalized hologram. After normalization Line Index Coding corresponding to complex representation is done for selected step size. After Line Index Coding the JPEG or JPEG2000 compression scheme is used to provide more compression. Analysis of the results through subjective and visual quality methods show that the combination of line index coding and JPEG coding produce significant compression when compared to the result obtained through LIC alone. This paper gives a scope for Digital Hologram Watermarking Systems, speeding up of transmission of 3D scenes, storage requirements in surveillance, multimedia and biomedical applications.

Strategies in Jpeg Compression Using Convolutional Neural Network(Cnn)

Interests in digital image processing are growing enormously in recent decades. As a result, different data compression techniques have been proposed which are concerned mostly with the minimization of information used for the representation of images. With the advances of deep neural networks, image compression can be achieved to a higher degree. This paper describes an overview of JPEG Compression, Discrete Fourier Transform (DFT), Convolutional Neural Network (CNN), quality metrics to measure the performance of image compression and discuss the advancement of deep learning for image compression mostly focused on JPEG, and suggests that adaptation of model improve the compression.

Vehicle License Plate Detector in Compressed Domain

Data compression techniques allow data size to be reduced prior to data transmission and involve decompression upon transfer. This study shows for the first time that license plate (LP) detection can be accomplished without full decompression of the encoded data. Therefore, by determining in advance which images are required for LP recognition, computational costs of the system can be reduced. The proposed approach is realized on High Efficiency Video Coding (HEVC) based compressed video sequences. Two methods are provided that generate images from HEVC attributes. Fully decoded pixel domain images are also generated for comparative purposes from the same encoded data. The YOLO V3 Tiny Object Detector is used in order to detect LPs in the generated images. EnglishLP, a public dataset, is used to interpret the findings in terms of speed and precision and for comparison with previous studies. An additional contribution of the paper is that a new compressed domain LP database has been created and made publicly available, comprising images captured by a commercial license plate recognition system. Using at least two-orders-of-magnitude less amount of data, the proposed compressed domain LP detector achieved similar precision and recall values to those of the state-of-the-art LP detection schemes tested on both datasets. Moreover, the proposed method results in more than 30% saving in inference time. The results suggest that the proposed method can be utilized for rapid video archive searching applications.

Data Compression Algorithms for Wireless Sensor Networks: A Review and Comparison

Energy consumption has risen to be a bottleneck in wireless sensor networks. This is caused by the challenges faced by these networks due to their tiny sensor nodes that have limited memory storage, small battery capacity, limited processing capability, and bandwidth. Data compression has been used to reduce energy consumption and improve network lifetime, as it reduces data size before it can be forwarded from the sensing node to the sink node in the network. In this paper, a survey and comparison of currently available data compression techniques in wireless sensor networks are conducted. Suitable sets of criteria are defined to classify existing data compression algorithms. An adaptive lossless data compression algorithm (ALDC) is analyzed through MATLAB coding and simulation from the reviewed data compression techniques. The analysis aims to discover strategies that can be used to reduce the amount of data further before it is transmitted. From this analysis, it was discovered that encoding residue samples, rather than raw data samples, reduced the bitstream from 112 bits to a range of 30 to 36 bits depending on the sample block sizes. The average length of data samples to be passed to the encoder was minimized from the original 14 bits per symbol to 1.125 bits per symbol. This demonstrated a 0.875 code efficiency or redundancy. It resulted in an energy saving of 67.8% to 73.2%. This work further proposes a data compression algorithm that encodes the residue samples with fewer bits than the ALDC algorithm. The algorithm reduced the bitstream to 26 bits. The average length of the code is equal to the entropy of the data samples, demonstrating zero redundancy and an improved energy saving of 76.8% compared to ALDC . The proposed algorithm, therefore, shows improved energy efficiency through data compression.

Design of Effective Lossless Data Compression Technique for Multiple Genomic DNA Sequences

In recent years, a massive amount of genomic DNA sequences are being created which leads to the development of new storing and archiving methods. There is a major challenge to process, store or transmit the huge volume of DNA sequences data. To lessen the number of bits needed to store and transmit data, data compression (DC) techniques are proposed. Recently, DC becomes more popular, and large number of techniques is proposed with applications in several domains. In this paper, a lossless compression technique named Arithmetic coding is employed to compress DNA sequences. In order to validate the performance of the proposed model, the artificial genome dataset is used and the results are investigated interms of different evaluation parameters. Experiments were performed on artificial datasets and the compression performance of Arithmetic coding is compared to Huffman coding, LZW coding, and LZMA techniques. From simulation results, it is clear that the Arithmetic coding achieves significantly better compression with a compression ratio of 0.261 at the bit rate of 2.16 bpc.

Error-Control Truncated SVD Technique for In-Network Data Compression in Wireless Sensor Networks

In-network data compression plays an important role in the elimination of redundant time-series data in a wireless sensor network (WSN). Inconsistency of data and high computational process in cluster formation remain to be challenging issues of in-network data compression particularly for energy-constraint WSNs. This paper develops a new data clustering technique for in-network data preprocessing and compression called Error-Control Truncated Singular Value Decomposition (ETSVD) to achieve online outlier detection and adaptive data compression. The ETSVD is divided into two modules which are Adaptive Recursive Outlier Detection and Smoothing (ARODS) and Adaptive Data Compression (DC). Firstly, the ARODS pre-processes the collected data for outlier detection and smoothing in order to improve the data quality. Secondly, the DC decomposes the pre-processed data into vector space to compress the spatio-temporal correlated data based on the predefined error threshold at the sending end. After the compression of correlated data, the distinct decomposed data are reconstructed at the receiver end which is performed offline. The simulation results show that the proposed technique is able to compress 91.49% of spatio-temporal environmental temperature data with reconstruction error having a minimum tolerance of $\pm 1.0^\circ $ C. The performance improvement of ETSVD in terms of error and accuracy compared to the performance of conventional SVD are 85.26% and 33.49%, respectively. Moreover, the ETSVD provides efficient error-control data preprocessing and compression solutions within the networks with minimum space and time complexities.

LOW POWER CODER FOR WIRELESS SENSOR NETWORKS

An Energy efficiency technique is a important subject for Wireless Sensor Networks (WSNs), that have received extensive attention to reduce energy consumption and improve network lifetime. Most of the power in sensor networks is consumed by data transmission and reception. This power can be reduced by using suitable data compression technique which reduces transmitted data over wireless channels. In this article, the design of an energy efficient coder is proposed. When this algorithm is applied to recursive systematic convolutional (RSC) code, it reduces both computation complexity and decode error probability. Applying this RSC code to wireless sensor network ensures correct data reception and reduced energy consumption. To validate and evaluate our work, simulation is done in Xilinx software and Power is calculated using Xilinx xpower tool. The simulation result shows that the proposed algorithm performs better than existing one

Improving Reconstructed Image Quality Via Hybrid Compression Techniques

Data compression is one of the core fields of study for applications of image and video processing. The raw data to be transmitted consumes large bandwidth and requires huge storage space as a result, it is desirable to represent the information in the data with considerably fewer bits by the mean of data compression techniques, the data must be reconstituted very similarly to the initial form. In this paper, a hybrid compression based on Discrete Cosine Transform (DCT), Discrete Wavelet Transform (DWT) is used to enhance the quality of the reconstructed image. These techniques are followed by entropy encoding such as Huffman coding to give additional compression. Huffman coding is optimal prefix code because of its implementation is more simple, faster, and easier than other codes. It needs less execution time and it is the shortest average length and the measurements for analysis are based upon Compression Ratio, Mean Square Error (MSE), and Peak Signal to Noise Ratio (PSNR). We applied a hybrid algorithm on (DWT–DCT 2 × 2, 4 × 4, 8 × 8, 16 × 16, 32 × 32) blocks. Finally, we show that by using a hybrid (DWT–DCT) compression technique, the PSNR is reconstructed for the image by using the proposed hybrid algorithm (DWT–DCT 8 × 8 block) is quite high than DCT.

An Approach to Efficient Dictionary Utilization and Improved Data Compression Technique for LZW Algorithm

This paper proposes an improved data compression technique compared to existing Lempel-Ziv-Welch (LZW) algorithm. LZW is a dictionary-updation based compression technique which stores elements from the data in the form of codes and uses them when those strings recur again. When the dictionary gets full, every element in the dictionary are removed in order to update dictionary with new entry. Therefore, the conventional method doesn’t consider frequently used strings and removes all the entry. This method is not an effective compression when the data to be compressed are large and when there are more frequently occurring string. This paper presents two new methods which are an improvement for the existing LZW compression algorithm. In this method, when the dictionary gets full, the elements that haven’t been used earlier are removed rather than removing every element of the dictionary which happens in the existing LZW algorithm. This is achieved by adding a flag to every element of the dictionary. Whenever an element is used the flag is set high. Thus, when the dictionary gets full, the dictionary entries where the flag was set high are kept and others are discarded. In the first method, the entries are discarded abruptly, whereas in the second method the unused elements are removed once at a time. Therefore, the second method gives enough time for the nascent elements of the dictionary. These techniques all fetch similar results when data set is small. This happens due to the fact that difference in the way they handle the dictionary when it’s full. Thus these improvements fetch better results only when a relatively large data is used. When all the three techniques' models were used to compare a data set with yields best case scenario, the compression ratios of conventional LZW is small compared to improved LZW method-1 and which in turn is small compared to improved LZW method-2.

A highly efficient chain code for compression using an agent-based modeling simulation of territories in biological beavers

The accelerated developments in technology led to a tremendous increase in the volumes of data to be transferred and exchanged between various network channels. These advancements create a huge demand for researchers to investigate new data compression techniques. Recent evidence from the literature shows that agent-based modeling is a promising direction to reduce the size of the data and change its original representation. In this article, the objective is to build an agent-based modeling simulation for chain coding and take advantage of it in data compression. Our agent-based model is inspired by the concept of defended territories of biological beavers. To this end, we use the pixel distribution in a bi-level image to construct a virtual environment of agents, add the beavers, and build territories around them. The main idea of defended beaver territories is to allow each beaver to maintain its area and protects it from intruders. To put it another way, defended territories allow beavers to work on different parts of an image while the algorithm tracks and records their movements, as well as manages disputes between them. Our research findings represent a further step towards employing the generated codes of movements in image processing operations other than coding and compression. Additionally, the experimental results showed that the current model was prosperous, and it could outperform many existing image compression techniques, including JBIG family methods. What’s more, paired-samples t-tests reveal that the mean differences between the outcomes of the current approach and each of the other standardized benchmarks we employed in comparison are statistically significant.

Deep learning for in situ data compression of large turbulent flow simulations

Large turbulent flow simulations can lead to information bottlenecks as data is generated faster than it can be processed and saved. Innovative in-situ data compression techniques are needed. Here we examine a deep learning approach to in-situ compression using a novel autoencoder architecture customized for three-dimensional turbulent flows. We compare it to a randomized single-pass singular value decomposition method and demonstrate improved compression and reconstruction, particularly with respect to important statistical quantities such as turbulent kinetic energy, enstrophy, and Reynolds stresses, at lower computational cost.

A new technique for data compression

Data compression has become more important than ever, due to the increasing demand for internet use and the exchange of a huge amount of images, videos, audio and documents as well as the growing demand for electronic archiving by government departments that produce thousands of documents per day. In this paper, the proposed technique for document compression will be presented.The proposed technique is a lossless and completed technique it is consists of two parts the compression part and decompression part. The compression part contains of some basic stages such as: pre-processing, blocks processing, run length encoding(RLE), replace maximum values by unused values, minimize levels, delta encoding, compression of ones values, encryption. After the encryption process is complete, the outputs are stored in two separate binary files with the extension of bmp;one of them is the header file and is considered a key for the second file which contains the compressed data. This technique applied on twenty documents and compared with other methods compression such as RLE, jpeg, tiff and png. The experimental results showed that the proposed technique gives a higher compression ratio than the rest of the methods.

An efficient secure data compression technique based on chaos and adaptive Huffman coding

Data stored in physical storage or transferred over a communication channel includes substantial redundancy. Compression techniques cut down the data redundancy to reduce space and communication time. Nevertheless, compression techniques lack proper security measures, e.g., secret key control, leaving the data susceptible to attack. Data encryption is therefore needed to achieve data security in keeping the data unreadable and unaltered through a secret key. This work concentrates on the problems of data compression and encryption collectively without negatively affecting each other. Towards this end, an efficient, secure data compression technique is introduced, which provides cryptographic capabilities for use in combination with an adaptive Huffman coding, pseudorandom keystream generator, and S-Box to achieve confusion and diffusion properties of cryptography into the compression process and overcome the performance issues. Thus, compression is carried out according to a secret key such that the output will be both encrypted and compressed in a single step. The proposed work demonstrated a congruent fit for real-time implementation, providing robust encryption quality and acceptable compression capability. Experiment results are provided to show that the proposed technique is efficient and produces similar space-saving (%) to standard techniques. Security analysis discloses that the proposed technique is susceptible to the secret key and plaintext. Moreover, the ciphertexts produced by the proposed technique successfully passed all NIST tests, which confirm that the 99% confidence level on the randomness of the ciphertext.

Joint Optimization of Transmission Bandwidth Allocation and Data Compression for Mobile-Edge Computing Systems

This letter investigates a multiuser mobile-edge computing (MEC) system with data compression technique to reduce the redundancy of sensed data, save the energy consumption and reduce the latency for wireless transmission. In this regard, the problem of minimizing the total energy consumption by jointly optimizing the transmission bandwidth allocation and data compression ratio is firstly considered under the constraints of latency and limited computation resource. Moreover, Lagragian and iterative-based algorithms are proposed to solve the non-convex optimization problem. Finally, simulation results are shown to verify the efficiency of the proposed algorithms and demonstrate that the application of data compression technique into MEC system can effectively reduce the energy consumption and latency.

Lora-WAN Powered by Renewable Energy, and Its Operation with Siri / Google Assistant

LoRa WAN is a newly emerged game changing communication technology for sending small data packets of size 50 bytes or less, wirelessly over an area of up to 10 Km without the need of an internet connection. LoRa WAN has its own frequency band and the band is different for every country. This technology is now starring to boost WSN technology better than ever before. This paper aims to, power up a LoRa Enabled Device or a LoRa Gateway by using a reliable dual mode non-conventional energy resource for storage and utilization, find peak performances altering the data rate that can be achieved in a LoRa WAN Communication (using Indoor RAK Gateway), make use data compression techniques, data packet encoding / decoding, Coding Apple Shortcuts, setting up Siri and Google Assistant for voice control and future scope.

Hybrid compression of biomedical ECG and EEG signals based on differential clustering and encoding techniques

AbstractSignal processing techniques incorporated with data compression processes enrich the signals and boost up storage efficiency and transmission reliability. Transmitting uncompressed original data consume wide bandwidth, which increases transmission time and leads to data hammering. These limitations enforce to look for strategic data compression techniques. Lossless compression techniques are requisite where it is important that the original and the decompressed data should be identical or where deviations from the original data would lead to catastrophic consequences, especially in biomedical signal analysis and diagnostics. For which, the input signal preprocessed with differential pulse code modulation (DPCM) reduces the interchannel dependencies to get the desired output. A whole array of unique compression techniques are being utilized in the compression process. The combination of (K‐means clustering, arithmetic encoding [AE], Huffman encoding [HE]) clustering and coding compression techniques are analyzed using electro cardiogram (ECG) and electroencephalogram (EEG) signals. The proposed method employs k‐means clustering combined with Huffman encoding (DiKHE) and k‐means clustering combined with arithmetic encoding (DiKAE) individually. Compression ratio (CR) is analyzed with these combinations of compression techniques for various cluster size K (K = 2,3,4,5,6). A maximum CR of 6.03144 and 4.54126 is obtained for ECG and EEG signals respectively. The compressions based on these techniques are efficient since the compressed signal is reconstructed perfectly as it matches exactly with the original signal.

Biometric recognition system performance measures for lossy compression on EEG signals

Abstract Electroencephalogram (EEG) plays an essential role in analysing and recognizing brain-related diseases. EEG has been increasingly used as a new type of biometrics in person identification and verification systems. These EEG-based systems are important components in applications for both police and civilian works, and both areas process a huge amount of EEG data. Storing and transmitting these huge amounts of data are significant challenges for data compression techniques. Lossy compression is used for EEG data as it provides a higher compression ratio (CR) than lossless compression techniques. However, lossy compression can negatively influence the performance of EEG-based person identification and verification systems via the loss of information in the reconstructed data. To address this, we propose introducing performance measures as additional features in evaluating lossy compression techniques for EEG data. Our research explores if a common value of CR exists for different systems using datasets with lossy compression that could provide almost the same system performance with those using datasets without lossy compression. We performed experiments on EEG-based person identification and verification systems using two large EEG datasets, CHB MIT Scalp and Alcoholism, to investigate the relationship between standard lossy compression measures and our proposed system performance measures with the two lossy compression techniques, discrete wavelet transform—adaptive arithmetic coding and discrete wavelet transform—set partitioning in hierarchical trees. Our experimental results showed a common value of CR exists for different systems, specifically, 70 for person identification systems and 50 for person verification systems.

Continuous Subsurface Tomography Over Cellular Internet of Things (IoT)

Continuous monitoring of dynamic systems in the energy industry can lead to significant cost reductions by optimizing operations and reducing downtime through automatic integrity monitoring. In this paper, we focus on a challenging scenario, continuous subsurface monitoring through passive seismic tomography. Our proposed solution to address this challenge is enabled by marrying low-cost low-power sensing design, large-scale wireless (cellular) Internet of Things (IoT) networking, and advanced edge/cloud computing technologies. We demonstrate that the proposed data compression technique, application protocol design, and signal processing approach at edge sensors and cloud solution, enable us to handle data-demanding subsurface tomography problem using commodity cellular IoT technologies paving the way towards changing the traditional wisdom in our business. Compared to the state-of-the-art, the proposed solution is scalable, easily replicable, and considerably more power-efficient. The simulation results as well as a proof-of-concept (PoC) field test in collaboration with our partners (Vodafone and Nikhef) corroborate the feasibility of the proposed ideas.

Editorial for the Special Issue “Advanced Machine Learning for Time Series Remote Sensing Data Analysis”

This Special Issue intended to probe the impact of the adoption of advanced machine learning methods in remote sensing applications including those considering recent big data analysis, compression, multichannel, sensor and prediction techniques. In principal, this edition of the Special Issue is focused on time series data processing for remote sensing applications with special emphasis on advanced machine learning platforms. This issue is intended to provide a highly recognized international forum to present recent advances in time series remote sensing. After review, a total of eight papers have been accepted for publication in this issue.

A Sequential Approach for Sentinel-1 TOPS Time-Series Co-Registration Over Low Coherence Scenarios

In the coming era of synthetic aperture radar (SAR) big data, the in-orbit Sentinel-1 mission will provide unprecedented data with an increasing volume. As a fundamental step of time-series analysis for such large and growing amount data, terrain observation by progressive scans (TOPS) co-registration still presents a relative challenge: 1) low coherence scenarios may degrade the estimate accuracy and 2) unprecedented and growing data volume increases the computational burden. To overcome both limitations, this article presents a sequential approach for TOPS time-series co-registration, with an emphasis on the enhanced spectral diversity (ESD) estimate accuracy over low coherence scenes. We first employ double sample over the burst overlap region to improve the statistical proprieties of sample covariance matrix, followed by ESD phase estimation using a phase linking algorithm. Then, we carry out the sequential co-registration on each mini-stack without the necessity for reprocessing the entire stack by introducing a data compression technique. Using synthetic data and real Sentinel-1 TOPS data over densely vegetated areas in the Yunan-Kweichow plateau, we fully evaluate the performance of presented approach and compare the results with those obtained from the state-of-the-art techniques. We found that the sequential approach can provide better time-series co-registration accuracy over low coherence scenes with the moderate computational efficiency.