Data Compression Method Research Articles

Huffman Algorithm Valuation Using Residue Number System

The Huffman algorithm is a widely used method for lossless data compression, which assigns variable-length codes to characters based on their frequency of occurrence in the input data. However, the traditional implementation of Huffman coding using binary arithmetic can be computationally intensive, particularly for large data sets. In recent years, the Residue Number System (RNS) has emerged as a promising alternative to binary arithmetic for certain types of computations, due to its potential for parallel processing and reduced hardware complexity. This paper evaluates the use of RNS as a basis for implementing the Huffman algorithm, comparing its performance with the traditional binary approach. The results demonstrate that RNS-based Huffman coding can achieve comparable or superior compression ratios, while reducing the computational requirements and potentially enabling faster compression and decompression. The study also highlights the importance of choosing appropriate RNS moduli and operands to optimize performance. Overall, the evaluation suggests that RNS can be a viable and efficient alternative to binary arithmetic for implementing the Huffman algorithm, particularly in applications with high computational demands or limited hardware resources. However, further research is needed to explore the potential benefits and limitations of RNS in other areas of data compression and signal processing.

Modelling and Analysis of Hybrid Transformation for Lossless Big Medical Image Compression

Due to rapidly developing technology and new research innovations, privacy and data preservation are paramount, especially in the healthcare industry. At the same time, the storage of large volumes of data in medical records should be minimized. Recently, several types of research on lossless medically significant data compression and various steganography methods have been conducted. This research develops a hybrid approach with advanced steganography, wavelet transform (WT), and lossless compression to ensure privacy and storage. This research focuses on preserving patient data through enhanced security and optimized storage of large data images that allow a pharmacologist to store twice as much information in the same storage space in an extensive data repository. Safe storage, fast image service, and minimum computing power are the main objectives of this research. This work uses a fast and smooth knight tour (KT) algorithm to embed patient data into medical images and a discrete WT (DWT) to protect shield images. In addition, lossless packet compression is used to minimize memory footprints and maximize memory efficiency. JPEG formats' compression ratio percentages are slightly higher than those of PNG formats. When image size increases, that is, for high-resolution images, the compression ratio lies between 7% and 7.5%, and the compression percentage lies between 30% and 37%. The proposed model increases the expected compression ratio and percentage compared to other models. The average compression ratio lies between 7.8% and 8.6%, and the expected compression ratio lies between 35% and 60%. Compared to state-of-the-art methods, this research results in greater data security without compromising image quality. Reducing images makes them easier to process and allows many images to be saved in archives.

Feature Representation and Compression Methods for Event-Based Data

Event camera emerges as a new type of neuromorphic sensor that records scenes in an asynchronous paradigm which provides high temporal resolution and high dynamic range. The huge volume of event-based data brings a computational burden to data transmission and real-time interaction in IntelliSense applications. Compared with the general encoding method and current event-based data compression method Spike Coding, we successively propose two event-based data compression methods by analyzing the statistical features of the event characteristic parameters ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${x}$ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${y}$ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${t}$ </tex-math></inline-formula> , and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${f}$ </tex-math></inline-formula> ). Based on the data features of event parameters, we design the “characteristic parameter jointed coding (CPJC)” method with the idea of fixed-length coding, which reduces the redundancy of delimiters between modules. As for the limitation of the CPJC method in pixel scale, we further propose the “ASCII coding based on bit operation (ACBO)” method based on the data storage principle in computer memory, reducing the redundancy caused by data storage. We introduce compression coefficient <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${K}$ </tex-math></inline-formula> as the metric to evaluate the compression performance on a real-world dataset. And the metrics <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${K}$ </tex-math></inline-formula> of the proposed two methods are improved by 17.93% and 14.92%, respectively, compared with the current event-based compression method. The proposed ACBO method also eliminates the limitation in data pixel scale and has certain adoption for event noise and event dispersion in the scene.

A divide-and-conquer method for compression and reconstruction of smart meter data

As smart grid sensors, smart meters generate abundant valuable data, laying the foundation for data-driven applications. However, the data collection brings huge communication pressure to electric utilities. In this context, considering that different types of devices have different power consumption patterns, and different types of data compression methods have their own applicable scenarios, we propose a divide-and-conquer method for compression and reconstruction of smart meter data. First, based on algorithm of voice activity detection (VAD), a load power fluctuation segment location method is proposed, which is combined with load event detection method to divide the load data into the event segments, fluctuation segments, and steady-state segments. Then, for the fluctuation segments, a cloud-device collaboration adaptive strategy based on the compressive sensing (CS) theory is designed, in which the sparse basis and measurement matrix are updated accordingly to ensure the high reconstruction accuracy in different scenarios. For the steady-state segments, a data compression method based on the improved symbolic aggregation approximation (SAX) is established, in which the dividing rectangle (DIRECT) algorithm and the irregular time partitioning method are combined to reduce the data volume for transmission without losing important information. For the event segments, the original data values are retained since the event power curves are relatively more complex and short duration. Finally, the received compressed data are reconstructed into the original power time series data in the master station on cloud to support advanced data analytics. Comparative experiments are conducted on the private and public datasets of 12 households in North America and China. The results show that our method has higher data reconstruction accuracy and compression efficiency compared to the existing methods.

A data compression and encryption method for green edge computing

A data compression and encryption method for green edge computing

An Area-Efficient Large Integer NTT-Multiplier Using Discrete Twiddle Factor Approach

Number theoretic transform (NTT) method shows great advantages in speed and efficiency for applications such as homomorphic encryption. However, the twiddle factor data consumes a lot of memories. In this brief, we propose a novel data compression method, together with the corresponding data storage scheme and addressing algorithm. Furthermore, we design a 768k-bit multiplier with a full pipeline structure. Our proposed compression method has achieved a compression rate of 98.8% for the twiddle factor data. Compared with the state-of-the-art FPGA implementations, our design shows up to 44.2% improvement in terms of area-efficiency.

Spatial-derivative-based compression approach for distributed temperature data.

The new generation of distributed optical sensors with improved interrogation, multiplexing, and acquisition techniques with the possibility of performing measurements with high spatial resolution over tens of kilometers of optical fiber has led to the accumulation of a vast volume of data that can present a big challenge to process and store all this data. Looking for simple solutions to this problem, we present in this paper a data compression method for distributed temperature sensors. This compression approach performs the spatial derivative of the temperature signal, constituting a simple and effective method to remove redundant information. Also, this compression methodology is suitable for temperature data, as it follows thermal variations over time and can be applied to any temperature profile with multiple thermal events along the sensing fiber, whether in heating or cooling circumstances. Tests performed with a large amount of experimental data showed that an average compression ratio of 1.5× can be obtained by removing redundant spatial temperature variations without losing spatial resolution.

C2SP-Net: Joint Compression and Classification Network for Epilepsy Seizure Prediction.

Recent developments in brain-machine inter-face technology have rendered seizure prediction possible. However, the transmission of a large volume of electro-physiological signals between sensors and processing apparatuses and the related computation become two major bottlenecks for seizure prediction systems due to the constrained bandwidth and limited computational resources, especially for power-critical wearable and implantable medical devices. Although many data compression methods can be adopted to compress the signals to reduce communication bandwidth requirement, they require complex compression and reconstruction procedures before the signal can be used for seizure prediction. In this paper, we propose C2SP-Net, a framework to jointly solve compression, prediction, and reconstruction without extra computation overhead. The framework consists of a plug-and-play in-sensor compression matrix to reduce transmission bandwidth requirements. The compressed signal can be utilized for seizure prediction without additional reconstruction steps. Reconstruction of the original signal can also be carried out in high fidelity. Compression and classification overhead from the energy consumption perspective, prediction accuracy, sensitivity, false prediction rate, and reconstruction quality of the proposed framework are evaluated using various compression ratios. The experimental results illustrate that our proposed framework is energy efficient and outperforms the competitive state-of-the-art baselines by a large margin in prediction accuracy. In particular, our proposed method produces an average loss of 0.6% in prediction accuracy with a compression ratio ranging from 1/2 to 1/16.

Edge-Assisted Real-Time Instance Segmentation for Resource-Limited IoT Devices

Instance segmentation exploits the potential of Internet of Things (IoT) devices to perceive environmental semantic information and achieve complex interactions with the physical world. However, IoT devices usually have limited computing and storage resources and cannot afford the intensive computational costs of instance segmentation networks. This article proposes an edge-assisted instance segmentation method for resource-limited IoT devices; it selectively offloads some computation-intensive tasks from IoT devices to edge servers to accelerate the inference processes of instance segmentation networks. To reduce the communication cost caused by computation offloading, a data compression method is proposed to adaptively adjust the downsampling interval based on an attention mechanism. Considering the susceptibility of the computation offloading scheme to network conditions, an adaptive computation offloading strategy that can jointly optimize the offloading point and the data compression ratio is proposed so that the edge-assisted instance segmentation can meet the preset latency requirements while achieving maximal accuracy under volatile network conditions. Extensive experiments are conducted to verify the feasibility and efficiency of our method. The experimental results show that our method induces less latency than existing instance segmentation methods with a slight drop in accuracy.

A Three-Dimensional Vibration Data Compression Method for Rolling Bearing Condition Monitoring

In condition monitoring for rolling bearings, it has achieved good diagnostic performance and clear mechanistic interpretation based on vibration data. The high sampling frequency of data collection preserves fault characteristics but brings the problem of big data. An effective way to reduce this problem is to apply data compression. However, in order not to affect the diagnostic performance of data, it is difficult to improve the compression ratio further. Inspired by the binarization method, the compression dimension of the bit cost of a single sample point is first introduced into the fault-mechanism-based method in this article. On this basis, a three-dimensional data compression method is proposed, and it is subsequently validated with two real-bearing datasets. Two performance metrics, including a newly defined one, are utilized to compare the proposed method with the five existing methods. The comparison results show that the proposed method significantly improves the compression ratio of data but maintains good diagnostic performance.

A Simple Lossless Algorithm for On-Board Satellite Hyperspectral Data Compression

As resolution of on-board imaging spectrometer keeps improving, data acquisition rate increases and resource limited satellite environment necessitates for computationally simple data compression methods to meet timing, bandwidth and resource requirements with error resilience. This letter proposes a new lossless, prediction based algorithm for on-board satellite hyperspectral data compression that utilizes spectral as well as spatial correlation and at the same time, is computationally less complex. Concept of non-binary tree traversal is used with nearest neighbor method and implemented using neighbor driven decision making in pre-processing stage. Previously processed pixels are used to minimize the prediction residual, which makes more than 80% calculations causal in nature and thereby reducing the computational complexity of the algorithm. The prediction residual is then encoded using sample adaptive Golomb coding in band-sequential order. CCSDS corpus of data for hyperspectral images is used for evaluating the performance of the algorithm. The proposed method shows reduced computational complexity and lesser data dependencies compared to the CCSDS 123.0-B-1 standard when similar spectral vicinity is considered, and comparable compression performance compared to other state-of-the-art on-board lossless compression methods.

Dynamic Predictive Quantization for Staggered SAR: Experiments With Real Data

Present and future spaceborne synthetic aperture radar (SAR) missions are designed to acquire an increasingly large amount of onboard data. This is a consequence of the use of large bandwidths, multiple polarizations, and the acquisition of large swath widths at fine spatial resolutions, which result in challenging requirements in terms of onboard memory and downlink capacity. In this scenario, SAR raw data quantization represents an essential aspect, as it affects the volume of data to be stored and transmitted to the ground as well as the quality of the resulting SAR products. Dynamic predictive block-adaptive quantization (DP-BAQ) is a novel technique, recently proposed by the authors, consisting of a low-complexity data compression method, and its application is particularly suitable for staggered SAR systems. DP-BAQ exploits the existing correlation among the azimuth raw data samples by applying linear predictive coding (LPC). This results in a data rate reduction of up to 25% with respect to state-of-the-art SAR quantization methods. In this letter, we test and validate the potential of DP-BAQ on airborne SAR data which emulates the system scenario of Tandem-L, a German Aerospace Center (DLR) mission proposal for a bistatic L-band system. For this purpose, an experimental SAR image has been acquired at the L-band by the airborne DLR flugzeug-SAR (F-SAR) sensor over the Kaufbeuren area, in Southern Germany. In order to simulate the staggered SAR acquisition mode, we implemented a dedicated resampling and filtering of the data. Our analyses confirm the effectiveness of DP-BAQ for efficient data volume reduction, exhibiting a consistent and promising performance when tested on areas characterized by different land cover types and backscatter statistics.

SAR Imaging in Frequency Scan Mode: System Optimization and Potentials for Data Volume Reduction

Frequency scanning (FScan) is an innovative acquisition mode for synthetic aperture radar (SAR) systems. The method is based on the frequency-dependent beam pointing capabilities of phased array antennas, artificially increased via the combined use of true time delays and phase shifters within the array antenna. By this, typical limitations of conventional SAR systems in terms of achievable swath width and azimuth resolution can be mitigated, and so a wide swath can be imaged maintaining a fine azimuthal resolution. In the first part of the article, we introduce the theoretical concept, which is necessary to evaluate the reduced echo window length (EWL) with respect to equivalent stripmap data and the implications for the transmit pulse characterization. An FScan sensor flying in a TerraSAR-X-like orbit is shown to be capable of imaging an 80-km wide swath with 1-m azimuth resolution. The resulting time–frequency properties of the recorded raw data make the traditional SAR data compression algorithms such as block-adaptive quantization (BAQ) highly inefficient in this case. Therefore, the second part of the article investigates dedicated quantization methods for efficient data volume reduction in FScan systems. Different solutions are investigated and evaluated through simulations. Various transformations of the raw data have been exploited to optimize the encoding process, including deramping, fast Fourier transform (FFT), and blockwise approaches. Compared with standard BAQ in the time domain, the suggested data compression methods significantly improve the resulting signal-to-quantization noise ratio, allowing for the reduction in the overall data volume by about 60% for the considered system scenario, while maintaining robustness in the presence of inhomogeneous scene characteristics at the cost of a modest complexity increase for its on-board implementation.

A Dynamic and Parallel Two-Stage Lossless Data Compression Method for Smart Grid

A Dynamic and Parallel Two-Stage Lossless Data Compression Method for Smart Grid

DEVELOPMENT AND SOFTWARE IMPLEMENTATION OF AN ALGORITHMIC MODEL FOR BROADCASTING DOCUMENTS ON VARIOUS DEVICES

The work designed, developed and implemented a web application intended for the transmission of electronic documents on various devices among the audience located at a distance from each other. The work is devoted to the development of approaches, algorithms and services for broadcasting documents of various formats on various devices without the use of a projector. Classical methods of data compression and encryption and parallelism were used to build the algorithms. The software is implemented in the JavaScript language using the Node.js and Vue.js frameworks. The MongoDB database is also used to store user data. To display the results of the work, a service was developed for broadcasting documents of various formats through a mobile device to other devices. A convenient software architecture has been developed, which allows you to easily maintain and improve the service in the future. A convenient and clear graphical interface for interaction with the user has been implemented. As you know, the direct use of classical methods and algorithms of data compression and encryption enables reliable use and storage of user data. Among many algorithms, the RSA method was used. The RSA method is a public-key cryptographic algorithm based on the computational complexity of the problem of multiplying large integers. Also, for greater efficiency in the service, parallelism methods and microservice architecture were developed. Their purpose is to distribute the load of the service on different subservices for greater efficiency of the program.

Methods of transient em data compression

A transient electromagnetic signal is recorded in an earth formation while drilling at multiple times. Data for subsequent interpretation is transmitted via a hydrochannel with a narrow bandwidth, for that reason the amount of transmitted information is limited. Therefore, data compression is an indispensable element of processing. It is important to determine what parameters and how many of them are necessary to describe the measured EMF curve. The idea is to transfer to the surface only significant parameters allowing to restore the recorded curve with a given accuracy. The paper proposes simple and fast data compression methods based on the spline approximation of the signal. Also, an orthogonal basis in the measurement space is used to describe each signal by a linear combination of vectors from this basis. We analyze transient curves and consider various magnetic field components. Numerical experiments show how many parameters are required to describe and reconstruct the signal of a deep-reading electromagnetic tool.

Research on Deep Compression Method of Expressway Video Based on Content Value

Aiming at the problem that the storage space and network bandwidth of expressway surveillance video are occupied largely due to data redundancy and sparse information, this paper proposes a deep compression method of expressway video depth based on content value. Firstly, the YOLOv4 algorithm is used to analyze the content value of the original video, extract video frames with vehicle information, and eliminate unintentional frames. An improved CNN is then designed by adding Feature Pyramids and the Inception module to accelerate the extraction and fusion of features at all levels and improve the performance of image classification and prediction. Finally, the whole model is integrated into HEVC encoder for compressing the preprocessed video. The experimental results show that at the expense of only a 5.96% increase of BD-BR, and only a 0.19 dB loss of BD-PSNR, the proposed method achieves a 64% compression ratio and can save 62.82% coding time compared with other classic data compression methods based on deep learning.

Reconstruction probability-based anomaly detection using variational auto-encoders

Anomaly detection is a method of categorizing unexpected data points or events in a dataset. Variational Auto-Encoders (VAEs) have proved to handle complex problems in a variety of disciplines. We propose a technique for detecting anomalies based on the reconstruction probability of VAEs. The proposed method trains VAEs on three different datasets. The reconstruction probability is a much more principled and realistic anomaly score than the reconstruction error utilized by auto-encoders and other data compression methods because of the theoretical background and by including the concept of variability. The paper describes recent deep learning models for anomaly detection, as well as a comparison to other methodologies. Variational auto-encoders are trained on three different datasets, in an unsupervised setup to classify the anomalies, based on reconstruction probability. Further, the in-depth study of anomaly detection techniques is presented in this paper. The data are reconstructed using the VAEs generative characteristics to investigate the root cause of the anomalies.

GPU implementations of deflate encoding and decoding

SummaryDeflate coding is a very popular lossless data compression method used in zlib, gzip (GNU zip), and zip, which performs the LZSS compression algorithm with Huffman coding. Deflate encoding and decoding involve sequential operations and their parallel acceleration using a GPU is quite hard. The main purpose of this paper is to present GPU implementations for encoding and decoding of Deflate coding. For efficient GPU implementations of Deflate coding, we have used multiple small hash tables for finding matching subsequences in the dictionary by multiple threads in parallel and applied the Single Kernel Soft Synchronization (SKSS) technique to fully utilize GPU computing resources. We have also adopted Huffman coding with gap arrays to accelerate parallel Huffman decoding. We have evaluated the performance of our GPU implementations using an NVIDIA A100 GPU and compared them with parallel/sequential Deflate encoding and decoding on the Intel X86 multicore CPUs using multiple threads/a single thread. Our GPU implementation of Deflate decoding is 1.66x–8.33x faster than the multiple thread implementation and 4.13x–36.56x faster than the single thread implementation.

Implementasi Algoritma Redundant Pattern Encoding Untuk Menyisipkan Pesan Teks Ke Dalam Gambar dan Algoritma Fibonacci Codes Untuk Kompresi

Information is a collection of data in the form of unified text that is public or confidential. Confidential data is data that contains something or that is not for publication. One way to secure the data is to use Steganography techniques. This provides tremendous benefits for maintaining data confidentiality. In steganography there are several methods, one of which is Redundant Pattern Encoding. However, from this process, the object that will be used as a container to accommodate the data will experience a change in the size of the data. This can raise the suspicion of many parties who want to access the data. This study describes how the procedure is carried out to compress objects that are used as containers to insert messages. The process uses the Fibonacci Codes method which is a simple and very simple data compression method. The results of this research will produce a stronger level of data security.