Data Compression Techniques Research Articles

Optimizing Top-K query processing on GPU using data compression and pre-filtering

In large-scale data analysis, efficient Top-K query processing is critical for numerous applications in science, industry, and society. Traditional approaches often involve substantial data transfer and computational overhead, making it difficult to meet the scalability and efficiency demands of modern datasets. This paper proposes a GPU-accelerated Top-K query processing method that integrates data compression and pre-filtering techniques to address these challenges. By partitioning and compressing data on the host side, it alleviates common PCIe bottlenecks in heterogeneous computing environments. A metadata-driven pre-filtering technique further reduces the data volume processed on the GPU, significantly improving query performance, particularly when handling anti-correlated datasets. Experimental results demonstrate that this method markedly reduces data transfer and processing time, confirming its effectiveness in enhancing the efficiency and scalability of Top-K query processing compared to existing methods.

Efficient Video Compression Using Afterimage Representation.

Recent advancements in large-scale video data have highlighted the growing need for efficient data compression techniques to enhance video processing performance. In this paper, we propose an afterimage-based video compression method that significantly reduces video data volume while maintaining analytical performance. The proposed approach utilizes optical flow to adaptively select the number of keyframes based on scene complexity, optimizing compression efficiency. Additionally, object movement masks extracted from keyframes are accumulated over time using alpha blending to generate the final afterimage. Experiments on the UCF-Crime dataset demonstrated that the proposed method achieved a 95.97% compression ratio. In binary classification experiments on normal/abnormal behaviors, the compressed videos maintained performance comparable to the original videos, while in multi-class classification, they outperformed the originals. Notably, classification experiments focused exclusively on abnormal behaviors exhibited a significant 4.25% improvement in performance. Moreover, further experiments showed that large language models (LLMs) can interpret the temporal context of original videos from single afterimages. These findings confirm that the proposed afterimage-based compression technique effectively preserves spatiotemporal information while significantly reducing data size.

Comparative Analysis of Huffman Coding Implementations for Efficient Data Communication Using Greedy and Divide-and-Conquer Techniques

Efficient data compression techniques are required to minimize storage and processing overhead due to modern systems' growing amount of data. Huffman Coding is a lossless compression technique that maintains data integrity by assigning shorter bit codes to characters appearing frequently, reducing size. Our analysis focuses on two implementation methodologies: greedy technique and divide and conquer. To find efficient solutions, divide-and-conquer algorithms partition problems into smaller components. In contrast, greedy algorithms strive to attain the utmost attainable result at each level. Our extensive investigation centers on the timing and space intricacies of diverse methodologies, enabling a comparative analysis that underscores their respective merits and drawbacks.

Using the Discrete Wavelet Transform for Lossy On‐the‐Fly Compression of GPU Fluid Simulations

ABSTRACTHigh‐performance computing in fluid dynamics frequently confronts substantial memory demands, especially in large‐scale applications. Data compression techniques can alleviate these memory constraints, but introduce new challenges. This paper introduces an innovative on‐the‐fly low‐overhead lossy compression technique tailored for GPU‐based fluid simulations, utilizing the discrete wavelet transform (DWT). The technique is applicable to any numerical scheme where the data is stored on a regular grid and the time step is computed using a stencil. Our approach significantly diminishes memory requirements, achieving up to a 10‐fold long‐term reduction on a D3Q27 simulation, while minimally impacting the simulation accuracy. The methodology is built around careful design choices to achieve a satisfactory compression ratio/speed trade‐off. It effectively maintains mass conservation and accurately preserves essential discontinuities in simulations. Extensive testing with a D3Q27 Lattice‐Boltzmann method (LBM) simulation on a single GPU has shown that large‐scale grids can be processed with minimal impact on the simulation accuracy and acceptable compression times. This compression technique demonstrates a robust capability to handle memory limitations in fluid simulations, opening the door to more complex and larger‐scale simulations.

Hardware optimization for effective switching power reduction during data compression in GOLOMB rice coding.

Loss-less data compression becomes the need of the hour for effective data compression and computation in VLSI test vector generation and testing in addition to hardware AI/ML computations. Golomb code is one of the effective technique for lossless data compression and it becomes valid only when the divisor can be expressed as power of two. This work aims to increase compression ratio by further encoding the unary part of the Golomb Rice (GR) code so as to decrease the amount of bits used, it mainly focuses on optimizing the hardware for encoding side. The algorithm was developed and coded in Verilog and simulated using Modelsim. This code was then synthesised in Cadence Encounter RTL Synthesiser. The modifications carried out show around 6% to 19% reduction in bits used for a linearly distributed data set. Worst-case delays have been reduced by 3% to 8%. Area reduction varies from 22% to 36% for different methods. Simulation for Power consumption shows nearly 7% reduction in switching power. This ideally suggest the usage of Golomb Rice coding technique for test vector compression and data computation for multiple data types, which should ideally have a geometrical distribution.

Sustainable energy: Advancing wind power forecasting with grey wolf optimization and GRU models

Wind power forecasting is critical for optimizing energy use and ensuring the reliability of wind power systems in renewable energy. This paper introduces a novel method that combines the Grey Wolf Optimization (GWO) algorithm with data compression techniques to enhance feature selection and reduce redundancy in wind speed prediction. By employing GWO, essential features were identified by grouping the dataset into intervals and analyzing their frequencies. Performance evaluation was conducted using various compression measures, including Rate DC-Miss, Rate DC-MEF, and Rate DC-BDG, compared with other models such as extreme gradient boosting, space-time graph neural networks, and deep learning models. The study's results show significant improvements in accuracy and efficiency for predicting wind speed compared to existing techniques. The proposed approach addresses both larger datasets and the impact of noise samples on prediction errors. Additionally, an MLDDR model was introduced to predict DC power generated from wind datasets, encompassing five stages: Data Preparation, Feature Selection, Data Compression, GRU-Based Predictions, and Rate of Reduction. Data reduction results are notable. The original wind dataset (104857613) was reduced to 1093913 after processing missing values, achieving a reduction rate of 0.136. Applying the MEF-GWO algorithm further reduced the dataset to 109395, with a reduction rate of 0.385. The BDG dataset was compressed to 1805, with a reduction rate of 0.607. In terms of prediction performance, the GRU model was evaluated on three datasets: the original, MEF, and BDG-GWO datasets. The GRU model demonstrated the highest accuracy (99.20 %) with the BDG-GWO dataset, with precision (0.9965), recall (0.9978), and F1 scores (0.9897) indicating superior performance. Training and testing times varied significantly, highlighting the computational challenges associated with deep learning techniques. This research addresses both programming and application challenges. Programming challenges include high computational demands and the trial-and-error nature of parameter determination in deep learning, mitigated by using GWO. Application challenges involve reducing large datasets, grouping them into intervals, and evaluating performance using different compression measures. The main research questions addressed include the suitability of the GWO algorithm for dataset reduction in terms of dimensions (features and records) and the effectiveness of combining GWO with deep learning, specifically GRU, for enhanced prediction results. The study concludes that the GWO-PCA and GRU combination significantly improves prediction accuracy and reduces implementation time.

Optimized Tiny Machine Learning and Explainable AI for Trustable and Energy-Efficient Fog-Enabled Healthcare Decision Support System

The Internet of things (IoT)-based healthcare decision support system plays a crucial role in modern medicine, especially with the rise in chronic illnesses and an aging population necessitating continuous remote health monitoring. Current healthcare decision support systems struggle to deliver timely and accurate decisions with minimal latency due to limited real-time healthcare data and inefficient computational resources. There is a critical need for an optimized, energy-efficient machine learning model that reliably supports remote health monitoring within IoT and fog computing environments. Our study proposes an Optimized Tiny Machine Learning (TinyML) and Explainable AI (XAI) binary classification model for a trustable and energy-efficient healthcare decision support system, leveraging fog computing to optimize performance. The fog-based approach improves response times and enhances bandwidth usage, addressing critical needs such as reduced latency, higher bandwidth utilization, and decreased packet loss. To further improve efficiency, we incorporate the innovative mLZW data compression technique, significantly enhancing data communication efficiency and reducing response time to critical health alerts. However, limited real-time healthcare data records challenge machine learning classification performance. By implementing a TinyML algorithm, our system demonstrates superior performance to other machine learning models. The proposed optimized TinyML model achieves an impressive F1 score of 0.93 for health abnormalities detection, emphasizing its robustness and effectiveness. This paper highlights the potential of TinyML and XAI in delivering robust, trustworthy, and energy-aware healthcare solutions, making significant contributions toward effective remote health monitoring and decision support in fog-enabled IoT networks.Graphical abstract

An efficient data compression and storage technique with key management authentication in cloud space

Cloud computing is one of the promising technologies that offers cost-effective choices for processing and storing the huge volumes of data. In today’s world, data is the most important asset that one can have but it needs to be handled and protected properly. Portability of data can be increased by reducing the size of the data to be stored because of the limited storage space. As a result, data compression has arisen significantly. Data compression is a useful technique for reducing data size and increasing the effectiveness of data transit and storage. Data compression reduces the size of a data file while using lossy or lossless compression. One of the newest techniques for data compression is data duplication, which can reduce the amount of data saved while removing unnecessary data and maintaining an exact copy of the data. This analysis presents an Efficient data compression and storage technique with key management authentication in cloud space. This approach uses Regressive probabilistic key encryption (RPKE) to encrypt the cloud data and Lempel-Ziv-77-Huffman coding (LZ77-HM) is used to compress the huge amounts of cloud data. The Performance of presented approach is evaluated in terms of compression ratio and compression rate.

On resolution coresets for constrained clustering

Specific data compression techniques, formalized by the concept of coresets, proved to be powerful for many optimization problems. In fact, while tightly controlling the approximation error, coresets may lead to significant speed up of the computations and hence allow to extend algorithms to much larger problem sizes. The present paper deals with a weight-balanced clustering problem, and is specifically motivated by an application in materials science where a voxel-based image is to be processed into a diagram representation. Here, the class of desired coresets is naturally confined to those which can be viewed as lowering the resolution of the input data. While one might expect that such resolution coresets are inferior to unrestricted coreset we prove bounds for resolution coresets which improve known bounds in the relevant dimensions and also lead to significantly faster algorithms in practice.

Accurate and efficient daily carbon emission forecasting based on improved ARIMA

Major nations across the globe are increasingly concerned about the rising trends in carbon dioxide (CO2) emissions, particularly in societies of varying scales. Against this backdrop, precise prediction of carbon emissions becomes critically important, especially for the formulation and adjustment of near-term carbon reduction policies. However, the non-linear, non-stationary, and complex nature of daily carbon emission data poses a great challenge for daily-level forecasting especially in the big data context. To address this issue, we propose a novel composite forecasting approach named DCEF dedicated to the estimation of daily carbon emissions. In concrete, our approach employs the Empirical Mode Decomposition (EMD) for data stabilization and the Auto-regressive Integrated Moving Average (ARIMA) model for forecasting, while integrating the Truncated singular value decomposition(TSVD) technique for data compression and mitigating noise. Finally, DCEF is empirically validated with real daily carbon emission datasets collected from 6 sectors in 13 countries of varying sizes. Experimental results demonstrate the advantages of our approach compared to other baseline models in terms of prediction accuracy and efficiency.

Extremely-Compressed SSDs with I/O Behavior Prediction

As the data volume continues to grow exponentially, there is an increasing demand for large storage system capacity. Data compression techniques effectively reduce the volume of written data, enhancing space efficiency. As a result, many modern SSDs have already incorporated data compression capabilities. However, data compression introduces additional processing overhead in critical I/O paths, potentially affecting system performance. Currently, most compression solutions in flash-based storage systems employ fixed compression algorithms for all incoming data without leveraging differences among various data access patterns. This leads to sub-optimal compression efficiency. This article proposes a data-type-aware Flash Translation Layer (DAFTL) scheme to maximize space efficiency without compromising system performance. First, we propose an I/O behavior prediction method to forecast future access on specific data. Then, DAFTL matches data types with distinct I/O behaviors to compression algorithms of varying intensities, achieving an optimal balance between performance and space efficiency. Specifically, it employs higher-intensity compression algorithms for less frequently accessed data to maximize space efficiency. For frequently accessed data, it utilizes lower-intensity but faster compression algorithms to maintain system performance. Finally, an improved compact compression method is proposed to effectively eliminate page fragmentation and further enhance space efficiency. Extensive evaluations using a variety of real-world workloads, as well as the workloads with real data we collected on our platforms, demonstrate that DAFTL achieves more data reductions than other approaches. When compared to the state-of-the-art compression schemes, DAFTL reduces the total number of pages written to the SSD by an average of 8%, 21.3%, and 25.6% for data with high, medium, and low compressibility, respectively. In the case of workloads with real data, DAFTL achieves an average reduction of 10.4% in the total number of pages written to SSD. Furthermore, DAFTL exhibits comparable or even improved read and write performance compared to other solutions.

Computed tomography of chemiluminescence using a data-driven sparse sensing framework

The aim of this work is to demonstrate the use of data-driven sparse sensing for the reconstruction of the three-dimensional chemiluminescence field of a flame from the two-dimensional line-of-sight integrated chemiluminescence signal. By leveraging the intrinsic low-dimensionality of physical phenomena, sparse sensing can be employed to reconstruct a signal from few samples. This makes it a good candidate to be employed in computed tomography of chemiluminescence, an imaging technique used to reconstruct the chemiluminescence field from chemiluminescence images. In the proposed methodology, the transforming basis used to project the system’s state in the low-dimensional manifold is computed using the proper orthogonal decomposition, which is a popular machine-learning technique for data compression. The methodology is demonstrated on a virtual experiment based on the data coming from the large eddy simulation of a jet flame in a vitiated coflow, where OH is employed as a surrogate of OH∗. This methodology is then compared to conventional techniques that are generally employed to solve tomographic problems. The results show that the proposed technique is able to reconstruct the chemiluminescence field using only one view, compared to the multiple views required by conventional methods. Moreover, our techniques is also capable of predicting the distribution of other features such as temperature and species concentrations.

A Comprehensive Survey for Hiding Medical Data Based DNA Lossless Technique

The health information data includes reports on the patient’s condition, including addresses, names, tests, treatments, diagnoses, and medical history. It is sensitive information for patients, and all means of protection must be provided to prevent third parties from manipulation or fraudulent use. The goal of this paper presents studies of safeguard the transfer of sensitive data over an unsecured network using DNA techniques to enhance security. A comparison analysis is used to show the security issues of the various DNA-based data (ciphering, concealing, and compression) techniques that are studied and investigated. Also, in our study is to provide future researchers with the necessary knowledge to conduct data security research to develop or improve DNA-based data- ciphering, compression-hiding strategies that are more reliable, secure, and efficient. Thus, future researchers need to propose a trustworthy DNA-based data (ciphering, hiding, and compression) techniques. The result of our study appears the perceived vulnerability of medical data and health insurance to concerns of security as well as common security issues, also the nature of these common security concerns, for reducing medical data security risks.

High performance holographic video compression using spatio-temporal phase unwrapping

In this work, we present a high-performance holographic data compression technique. This approach is based in the temporal correlation found in a holographic video generated using an optimized random phase. This temporal correlation makes possible high levels of compression by using a spatio-temporal (ST) representation, partial phase unwrapping, and a lossless compression algorithm like DEFLATE. In this approach, two holographic videos are created using the optimized random phase (ORAP) technique and the Gerchberg-Saxton (GS) algorithm. The phase holograms corresponding to each frame of the videos are arranged into a ST representation, enabling spatial and temporal phase unwrapping. We test phase unwrapping along the spatial and temporal dimensions separately, followed by volume reduction after applying the DEFLATE lossless compression algorithm to the unwrapped phase in each case. The holographic video is then reconstructed from the compressed temporal and spatial unwrapped phase. The results demonstrate the high efficiency of the proposed method, with ORAP holograms achieving a compression percentage of 92.670 % after applying DEFLATE to a ST representation with partial phase unwrapping along the temporal dimension.

Enhanced Data Security and Storage Efficiency in Cloud Computing: A Survey of Data Compression and Encryption techniques

Enhanced Data Security and Storage Efficiency in Cloud Computing: A Survey of Data Compression and Encryption techniques

P‐57: A Novel Compression Algorithm using Machine Learning for Mura Compensation of OLED Panel

In the modern display panel manufacturing industry, the Mura problem that occurs during initial shipment has a significant impact on improving image quality. The purpose of this paper is to effectively solve these Mura defects while reducing additional production costs. We propose an innovative method of storing and compressing the De‐Mura compensation data of the display panel in external memory. Two‐dimensional compensation data compression based on machine learning using surface fitting and vector quantization is presented as an effective response to the Mura problem. This approach, which achieves up to 8:1 compression ratio while minimizing image quality degradation through vector quantization, is expected to provide economic benefits to manufacturers. The paper also introduces a new surface prediction method that replaces the existing DC‐shift and uses residual vector quantization, which can reduce the implementation complexity of vector quantization. Through experimental results, this study demonstrates that the compensated data compression technique yields excellent results in post‐compression restoration performance.

A Survey on Data Compression Techniques for Automotive LiDAR Point Clouds.

In the evolving landscape of autonomous driving technology, Light Detection and Ranging (LiDAR) sensors have emerged as a pivotal instrument for enhancing environmental perception. They can offer precise, high-resolution, real-time 3D representations around a vehicle, and the ability for long-range measurements under low-light conditions. However, these advantages come at the cost of the large volume of data generated by the sensor, leading to several challenges in transmission, processing, and storage operations, which can be currently mitigated by employing data compression techniques to the point cloud. This article presents a survey of existing methods used to compress point cloud data for automotive LiDAR sensors. It presents a comprehensive taxonomy that categorizes these approaches into four main groups, comparing and discussing them across several important metrics.

Classical vs. neural network-based PCA approaches for lossy image compression: Similarities and differences

The paper describes three lossy data compression techniques based on the principal component analysis (PCA), which are compared using the image compression task. The presented approach uses both classical PCA method based on the eigen-decomposition of the image data covariance matrix and two different neural network structures. The first neural structure is a two-layer feed-forward network with supervised learning acting as the so-called autoencoder, and the second one is a single-layered network with an unsupervised learning method commonly known as the generalized Hebbian algorithm. For each compression method mentioned above, the influence of the number of image segments (frames) and the number of eigenvalues/eigenvectors on the compression ratio and the image quality are examined using three different gray-scale test images. The paper also addresses a vital implementation issue regarding selecting appropriate data types to represent the compressed data. The paper’s main conclusion is that the classical PCA method outperforms its neural counterparts, both in terms of the decompressed image quality and the time required to perform the compression procedure. The positive aspect of using neural networks as a tool for PCA-based lossy data compression is that they do not require calculating the correlation matrix explicitly and thus can be used in online data acquisition schemes.

Evaluation of Color and Pigment Changes in Tomato after 1-Methylcyclopropene (1-MCP) Treatment.

The Polar Qualification System (PQS) was applied on hue spectra fingerprinting to describe color changes in tomato during storage. The cultivar 'Pitenza' was harvested at six different maturity stages, and half of the samples were subjected to gaseous 1-methylcyclopropene (1-MCP) treatment. Reference color parameters were recorded with a vision system colorimeter instrument, and the fruit pigment concentration was assessed with the DA-index®. Additionally, acoustic firmness (Stiffness) was measured. All acquired reference parameters were used to grade fruit in the supply chain. The applied 1-MCP treatments were used to control the ripening of climacteric horticultural produce. Both the DA-index® and stiffness values, presented as chlorophyll concentration and acoustic firmness, showed significant differences among maturity stages and treated and control samples and in their kinetics during storage. The machine vision parameter PQS-X was significantly affected by 1-MCP treatment (F = 10.18, p < 0.01), while PQS-Y was primarily affected by storage time (F = 18.18, p < 0.01) and maturity stage (F = 11.15, p < 0.01). A significant correlation was achieved for acoustic firmness with normalized color (r > 0.78) and PQS-Y (r > 0.80), as well as for the DA-index® (r > 0.9). The observed color changes agreed with the reference measurements. The significant statistical effect on the PQS coordinates suggests that hue spectra fingerprinting with this data compression technique is suitable for quality assessment based on color.

Blind-Touch: Homomorphic Encryption-Based Distributed Neural Network Inference for Privacy-Preserving Fingerprint Authentication

Fingerprint authentication is a popular security mechanism for smartphones and laptops. However, its adoption in web and cloud environments has been limited due to privacy concerns over storing and processing biometric data on servers. This paper introduces Blind-Touch, a novel machine learning-based fingerprint authentication system leveraging homomorphic encryption to address these privacy concerns. Homomorphic encryption allows computations on encrypted data without decrypting. Thus, Blind-Touch can keep fingerprint data encrypted on the server while performing machine learning operations. Blind-Touch combines three strategies to efficiently utilize homomorphic encryption in machine learning: (1) It optimizes the feature vector for a distributed architecture, processing the first fully connected layer (FC-16) in plaintext on the client side and the subsequent layer (FC-1) post-encryption on the server, thereby minimizing encrypted computations; (2) It employs a homomorphic encryption-compatible data compression technique capable of handling 8,192 authentication results concurrently; and (3) It utilizes a clustered server architecture to simultaneously process authentication results, thereby enhancing scalability with increasing user numbers. Blind-Touch achieves high accuracy on two benchmark fingerprint datasets, with a 93.6% F1- score for the PolyU dataset and a 98.2% F1-score for the SOKOTO dataset. Moreover, Blind-Touch can match a fingerprint among 5,000 in about 0.65 seconds. With its privacy-focused design, high accuracy, and efficiency, Blind-Touch is a promising alternative to conventional fingerprint authentication for web and cloud applications.