Data Compression Research Articles

Arithmetic N-gram: an efficient data compression technique

Due to the increase in the growth of data in this era of the digital world and limited resources, there is a need for more efficient data compression techniques for storing and transmitting data. Data compression can significantly reduce the amount of storage space and transmission time to store and transmit given data. More specifically, text compression has got more attention for effectively managing and processing data due to the increased use of the internet, digital devices, data transfer, etc. Over the years, various algorithms have been used for text compression such as Huffman coding, Lempel-Ziv-Welch (LZW) coding, arithmetic coding, etc. However, these methods have a limited compression ratio specifically for data storage applications where a considerable amount of data must be compressed to use storage resources efficiently. They consider individual characters to compress data. It can be more advantageous to consider words or sequences of words rather than individual characters to get a better compression ratio. Compressing individual characters results in a sizeable compressed representation due to their less repetition and structure in the data. In this paper, we proposed the ArthNgram model, in which the N-gram language model coupled with arithmetic coding is used to compress data more efficiently for data storage applications. The performance of the proposed model is evaluated based on compression ratio and compression speed. Results show that the proposed model performs better than traditional techniques.

On the bitmap compression for joint coding and data hiding of AMBTC compressed images

On the bitmap compression for joint coding and data hiding of AMBTC compressed images

LeCo: Lightweight Compression via Learning Serial Correlations

Lightweight data compression is a key technique that allows column stores to exhibit superior performance for analytical queries. Despite a comprehensive study on dictionary-based encodings to approach Shannon's entropy, few prior works have systematically exploited the serial correlation in a column for compression. In this paper, we propose LeCo (i.e., Learned Compression), a framework that uses machine learning to remove the serial redundancy in a value sequence automatically to achieve an outstanding compression ratio and decompression performance. LeCo presents a general approach to this end, making existing algorithms such as Frame-of-Reference (FOR), Delta Encoding, and Run-Length Encoding (RLE) special cases under our framework. Our microbenchmark with three synthetic and eight real-world data sets shows that a prototype of LeCo achieves a Pareto improvement on both compression ratio and random access speed over the existing solutions. When integrating LeCo into widely-used applications, we observe up to 5.2× speed up in a data analytical query in the Arrow columnar execution engine, and a 16% increase in RocksDB's throughput.

Incremental sampling methods for multi-fidelity surrogate modeling: Application on a furnace operating in MILD combustion conditions

This study introduces a framework for developing a multi-fidelity reduced order model (MF-ROM) of a combustion furnace operating under Moderate and Intense Low-oxygen Dilution (MILD) conditions. It integrates Proper Orthogonal Decomposition for data compression, Procrustes manifold alignment for fidelity transfer, and CoKriging for interpolation. Design parameters such as air injector diameter, fuel composition, and equivalence ratio were used to generate two- and three-dimensional simulations and build the MF-ROM. Additionally, the key question concerning the optimal number of high-fidelity simulations to balance accuracy and training cost is addressed when building the MF-ROM. Through incremental sampling strategies, it is demonstrated that around half of the training cost can be conserved while maintaining comparable error values.

An autonomous wheelchair with health monitoring system based on Internet of Thing

Assistive powered wheelchairs will bring patients and elderly the ability of remain mobile without the direct intervention from caregivers. Vital signs from users can be collected and analyzed remotely to allow better disease prevention and proactive management of health and chronic conditions. This research proposes an autonomous wheelchair prototype system integrated with biophysical sensors based on Internet of Thing (IoT). A powered wheelchair system was developed with three biophysical sensors to collect, transmit and analysis users’ four vital signs to provide real-time feedback to users and clinicians. A user interface software embedded with the cloud artificial intelligence (AI) algorithms was developed for the data visualization and analysis. An improved data compression algorithm Minimalist, Adaptive and Streaming R-bit (O-MAS-R) was proposed to achieve a higher compression ratio with minimum 7.1%, maximum 45.25% compared with MAS algorithm during the data transmission. At the same time, the prototype wheelchair, accompanied with a smart-chair app, assimilates data from the onboard sensors and characteristics features within the surroundings in real-time to achieve the functions including obstruct laser scanning, autonomous localization, and point-to-point route planning and moving within a predefined area. In conclusion, the wheelchair prototype uses AI algorithms and navigation technology to help patients and elderly maintain their independent mobility and monitor their healthcare information in real-time.

The Data Compression Method and FPGA Implementation in the Mars Rover Subsurface-Penetrating Radar on the Tianwen-1 Mission

Since Mars is far away from Earth, the propagation delay between Mars and Earth is very large. To ensure the effective use of the link transmission bandwidth, China’s first Mars exploration mission has put forward a demand for data compression for all scientific payloads. The on-board mature algorithms for data compression are mainly focused on optical images and microwave imaging radar applications. No articles have been published on data compression methods that are applied to subsurface-penetrating radar. Based on the background of this application, this paper proposes a logarithmic lossy compression algorithm which can meet the mission requirements for high compression ratios of 4:1 and 2.5:1. Its compression error is less than that of the block adaptive quantization (BAQ) algorithm. The algorithm is not only easy to implement on field-programmable gate array (FPGA) platforms, but also offers simple ground decompression and fast imaging. The experimental results show that high compression ratios of 4:1 and 2.5:1 can be realized, even if the data in and between traces do not have a strong correlation. And its relative error is less than 2%, which is a new type of high-efficiency data compression method that can be implemented on-board to meet with the demand of subsurface penetrating radar.

Two‐step attribute reduction for AIoT networks

AbstractThe evolution of Artificial Intelligence of Things (AIoT) pushes connectivity from human‐to‐things and things‐to‐things, to AI‐to‐things, has resulted in more complex physical networks and logical associations. This has driven the demand for Internet of Things (IoT) devices with powerful edge data processing capabilities, leading to exponential growth in device quantity and data generation. However, conventional data preprocessing methods, such as data compression and encoding, often require edge devices to allocate computational resources for decoding. Additionally, some lossy compression methods, like JPEG, may result in the loss of important information, which has negative impact on the AI training. To address these challenges, this paper proposes a two‐step attribute reduction approach, targeting devices and dimensions, to reduce the massive amount of data in the AIoT network while avoiding unnecessary utilization of edge device resources for decoding. The device‐oriented and dimension‐oriented attribute reductions identify important devices and dimensions, respectively, to mitigate the multimodal interference caused by the large‐scale devices in the AIoT network and the curse of dimensionality associated with high‐dimensional AIoT data. Numerical results and analysis show that this approach effectively eliminates redundant devices and numerous dimensions in the AIoT network while maintaining the basic data correlation.

A method for assessing the characteristics of the virtual machine migration process, taking into account the type of migrations and applications

Dynamic resource allocation in cloud computing is a critical aspect. It is necessary for computing resources scaling to maintain application performance, reduce costs and ensure the reliability of IT systems. Dynamic resource allocation is possible using open virtual machines (VM), but live migration is a relatively expensive and resource-intensive operation. A number of VM migration algorithms have been proposed, each with different performance characteristics depending on the state of the host system, network, and the VM workload. Recently, a large number of works have emerged that have developed models for assessing the effectiveness of of VM migration, but many of them do not provide a standard of satisfactory prediction accuracy and are built for a single migration algorithm, which limits the use of data models. This work uses a statistical approach to estimate the total migration time and VM downtime based on the approximation of probability densities by Gram-Charlier and Laguerre Series. The proposed method, in comparison with the known ones, allows one to answer the question of what is the probability of total migration time and VM downtime for a given migration type and VM application. The analysis of total migration time and VM downtime is based on the Virtual Machine Live Migration Dataset includes more than 40 000 virtual machine migration records with five different live migration algorithms: post-copy migration, pre-copy migration and its modifications: CPU throttling delta compression, and data compression. The dataset includes approximately 8000 migration records of each migration algorithm with 9 types of workloads. Analysis of the results allows us to conclude that the type of application significantly influences both the shape of the empirical distribution (histograms) of total migration time and its characteristics, therefore it is advisable to obtain an analytical expression for the distribution law of the total migration time and VM downtime taking into account these circumstances. The results of the experiments shows that this method does not depend on the migration algorithm and the type of working application. The use of the Laguerre series can be recommended as giving more reliable results compared to Gram-Charlier series. A method for estimating total migration time and VM downtime can be integrated into a management system to select the best monitoring window for servers and evaluate service-level agreement terms.

Data Compression and Damage Evaluation of Underground Pipeline With Musicalized Sonar GMM

The underground cement pipeline network is critical in urban infrastructure, such as sewage drainage, fluid transportation, etc. Pipelines are deeply buried, so the damage is difficult to be detected. It is of great significant to monitor the crack of pipeline in real time. In this research, a data compression and damage evaluation method of underground pipeline using musicalized sonar GMM (Gaussian Mixed Model) was proposed. First, the sonar echo signal was sampled and discretized into four musicalized indicators by musicalized model. Then, the distribution of multi parameters were analyzed by GMM. The Frechet similarity was constructed to compare the difference between each working condition and baseline condition, so that the damage state could be identified. The engineering experiment shown that this method could greatly compress the monitoring data and evaluate four crack levels.

Near-infrared spectroscopy combined with fuzzy fast pseudoinverse linear discriminant analysis to discriminate mee tea grades

Mee tea, one of the major types of green tea in China, is often used for export because of its elegant appearance, high fragrance and strong taste. However, the quality of tea differs greatly due to the difference in raw material selection and production technology level. In order to accurately and quickly differentiate different grades of Mee tea, fuzzy fast pseudoinverse linear discriminant analysis (FFPLDA) was proposed based on fast pseudoinverse linear discriminant analysis (FPLDA) for extracting discriminant information from near-infrared (NIR) spectra. Firstly, NIR spectra of Mee tea samples were acquired, and then they were preprocessed by multiplicative scatter correlation (MSC). Secondly, the compression of data was achieved by principal component analysis (PCA). Thirdly, linear discriminant analysis (LDA), FPLDA, FFPLDA and fuzzy Foley-Sammon transformation (FFST) were respectively performed to retrieve discriminant information from NIR data. Finally, the K-nearest neighbor (KNN) was utilized to classify Mee tea grades. In this study, experimental results showed that the accuracy of FFPLDA was higher than that of LDA, FFST and FPLDA. Therefore, NIR spectroscopy coupled with FFPLDA and KNN has a good effect in discrimination of Mee tea grades and also a great application potential.

Using convolutional neural networks to select the optimal wavelet for audio compression

The vast amount of data generated and stored across domains is overwhelming. Noteworthy examples include the Event Horizon Telescope using 3.5 petabytes for the first black hole image, LIGO amassing 4.5 petabytes, and CERN with 487 petabytes. This research addresses the challenge of data storage and transmission using machine learning and wavelet analysis. The goal is to create an efficient data compression method that significantly reduces storage needs without compromising data integrity. Wavelets, mathematical tools which break down signals into coefficients, show promise for compression. Choosing the right wavelet is crucial; our research demonstrates that machine learning can effectively solve this problem. By training a convolutional neural network on spectrograms of audio files, we identify the optimal wavelet for data compression, achieving up to 50% compression with minimal loss.

A Compact Depth Separable Convolutional Image Filter for Clinical Color Perception Test

<p>Deep convolutional neural networks have achieved good performance in the application of computer vision, but there are also problems, such as a large amount of computation, time consuming, and high memory demand. In this paper, a depthwise separable convolution filter pruning method based on PCA is proposed. First, this paper uses depthwise separable convolution to replace the conventional convolution in ResNet to reduce the number of parameters and the amount of computation in the network. The specific operation process is to first use depthwise convolution to separate the spatial dimension to increase the network width and expand the range of feature extraction, and then use pointwise convolution to reduce the computational complexity of conventional convolution operation. Second, PCA is used to distinguish stacked similar filters and perform dimensionality reduction, which not only alleviates the dimensional disaster, but also achieves compression of data and minimizes information loss. Experimental results show that this method can significantly improve the calculation speed and accuracy of the deep convolutional neural network model, and further compress the model size. On the clinical Color Perception Test Chart, this method reduced the amount of model parameters and MACs on ResNet by about 91% while maintaining the test accuracy at about 95%. With almost no loss of accuracy, this method greatly shortened the running time of the model.</p> <p> </p>

Intelligent Event-Triggered Control Supervised by Mini-Batch Machine Learning and Data Compression Mechanism for T-S Fuzzy NCSs Under DoS Attacks

Intelligent Event-Triggered Control Supervised by Mini-Batch Machine Learning and Data Compression Mechanism for T-S Fuzzy NCSs Under DoS Attacks

A Proposed IOT-based Smart Healthcare Management Framework for Performing Lossless Data Compression using Concept of Ontology

A Proposed IOT-based Smart Healthcare Management Framework for Performing Lossless Data Compression using Concept of Ontology

First test results of the HGCAL concentrator ASICs: ECON-T and ECON-D

With over 6 million channels, the High Granularity Calorimeter for the CMS HL-LHC upgrade presents a unique data transmission challenge. The ECON ASICs provide a critical stage of on-detector data compression and selection for the trigger path (ECON-T) and data acquisition path (ECON-D) of the HGCAL. The ASICs, fabricated in 65 nm CMOS, are radiation tolerant up to 200 Mrad and require low power consumption: < 2.5 mW/sensor-channel per chip. We report on the first functionality and radiation tests for the ECON-D-P1 full-functionality prototype. We present a comparison of single event effect (SEE) cross sections measured for different methods of triple modular redundancy using test results from the ECON-T-P1 full-functionality prototype.

ADVANCES IN NETWORK SECURITY FOR A RESILIENT DIGITAL CYBERSPACE INFRASTRUCTURE

In network security, safeguarding digital cyberspace infrastructure against malicious intrusions remains a paramount concern. This study addresses this imperative by proposing a novel approach that integrates digital watermarking, Huffman coding, and sophisticated attack classification techniques within the Controller Area Network (CAN) protocol framework. The proliferation of interconnected systems and the increasing sophistication of cyber threats necessitate novel strategies to fortify network defenses and ensure resilience in the face of adversarial activities. The research problem revolves around the need to develop an effective method for detecting and categorizing reconnaissance and violent attacks within cyberspace networks, particularly those utilizing the CAN protocol. Despite advancements in security protocols and intrusion detection systems, existing methodologies often fall short in accurately distinguishing between benign and malicious network activities, leaving critical infrastructure vulnerable to exploitation. This research uses digital watermarking to embed metadata within network packets, Huffman coding for efficient data compression, and advanced attack classification algorithms for real-time threat identification. The experimental results demonstrate the efficacy of the proposed approach in accurately differentiating between reconnaissance probes and violent attacks, thereby enabling timely and targeted responses to mitigate security threats.

Compression AutoEncoder for High-Resolution Ocean Sound Speed Profile Data

High-resolution ocean sound speed profile (HROSSP) data is essential for ocean acoustic modeling and sonar performance evaluation. However, the large volume and storage requirements of this data severely restrict its practical application in ocean acoustics. In this paper, we propose a compression autoencoder specifically designed for managing HROSSP data (CAE-HROSSP) and investigate the optimal network structure. Experimental results demonstrate that by using the min-max normalization method for input data and the corresponding inverse normalization for output data, along with employing the LeakyReLU function as the final activation layer, the accuracy of decompressed data reconstruction can be significantly improved. To tackle the challenges of fitting the distribution of surface sound speed data caused by significant variations and noise, we propose two loss functions: slice mean square error and elemental mean square error. These loss functions are combined with mean squared error through weighted summation to enhance CAE-HROSSP’s ability to fit the distribution of surface sound speed values and minimize the reconstruction errors of compressed data. Performance evaluation experiments reveal that CAE-HROSSP outperforms two existing methods in compressing HROSSP data, achieving superior performance with smaller data reconstruction errors at higher compression ratios. Furthermore, transfer learning is utilized to enhance the training of CAE-HROSSP, employing HROSSP data from the area where the mesoscale eddy is situated, as well as at the convergence of cold and warm ocean currents. The compression performance of both the training set and the validation set is comparable in the sea, where the structure of the sound speed profile varies greatly. This indicates that CAE-HROSSP can compress highly variable sound speed profile data in more sea areas using transfer learning, and has the potential to be extended globally. The findings and insights obtained from this study provide guidance for future endeavors in utilizing autoencoders to compress HROSSP data.

Single-Frame-Based Data Compression for CAN Security

To authenticate a controller area network (CAN) data frame, a message authentication code (MAC) must be sent along with the CAN frame, but there is no space reserved for the MAC in the CAN frame. Recently, difference-based compression (DBC) algorithms have been used to create a space inside the frame. DBC has the advantage of being very efficient, but its drawback is that, if an error occurs in one frame, the effects of that error propagate to subsequent frames. In this paper, a CAN data compression algorithm is proposed that compresses the current frame without relying on previous frames. Therefore, an error generated in one frame cannot be propagated to subsequent frames. In addition, a CAN signal grouping technique is proposed based on entropy analysis. To efficiently authenticate CAN frames, the length of the compressed data must be 4 bytes or less (4BL). Simulation shows that the 4BL-compression ratio of a Kia Sorento vehicle is 99.36% in the DBC method, but 100% in the proposed method. In an LS Mtron tractor, the 4BL-compression ratio is 98.58% in the DBC method, but 100% in the proposed method. In addition, the execution time of the proposed compression algorithm is only 27.39% of that of the DBC algorithm. The results show that the proposed algorithm has better compression characteristics for CAN security than the DBC algorithms.

Auxiliary active noise control system based on signal reconstruction

Faced with the problem of existing active control systems failing to obtain a reference signal and reducing or deteriorating the effectiveness of interior noise control, an Auxiliary Active Noise Control system (AANC) based on signal reconstruction technology is proposed, which main contains offline update of multi-network weights and online AANC controller. Firstly, considering the non-linear and non-stationary of the interior noise, and aiming to balance the modeling efficiency and accuracy of algorithms, based on the data fusion and compression sensing technology, a signal reconstruction model for multi-networks based on decomposition optimization is design; Then, in the online AANC controller, the offline calculation of the weight of signal reconstruction model is used to obtain the reconstruction reference signal of the control position, and based on the reconstructed signal, the controller achieves adaptive suppression of passenger ear-sides noise through the VSS-LMS algorithm. Finally, the effectiveness of the proposed AANC is verified using real vehicle data. The results show that the proposed system achieves the balance the modeling efficiency and accuracy, and is superior to existing ANC system in terms of robustness, guaranteeing the stable operation of the interior noise control.

Reconfigurable perovskite X-ray detector for intelligent imaging

X-ray detection is widely used in various applications. However, to meet the demand for high image quality and high accuracy diagnosis, the raw data increases and imposes challenges for conventional X-ray detection hardware regarding data transmission and power consumption. To tackle these issues, we present a scheme of in-X-ray-detector computing based on CsPbBr3 single-crystal detector with convenient polarity reconfigurability, good linear dynamic range, and robust stability. The detector features a stable trap-free device structure and achieves a high linear dynamic range of 106 dB. As a result, the detector could achieve edge extraction imaging with a data compression ratio of ~50%, and could also be programmed and trained to perform pattern recognition tasks with a high accuracy of 100%. Our research shows that in-X-ray-detector computing can be used in flexible and complex scenarios, making it a promising platform for intelligent X-ray imaging.