Data Compression Method Research Articles

Learning-based short text compression using BERT models

Learning-based data compression methods have gained significant attention in recent years. Although these methods achieve higher compression ratios compared to traditional techniques, their slow processing times make them less suitable for compressing large datasets, and they are generally more effective for short texts rather than longer ones. In this study, MLMCompress, a word-based text compression method that can utilize any BERT masked language model is introduced. The performance of MLMCompress is evaluated using four BERT models: two large models and two smaller models referred to as “tiny”. The large models are used without training, while the smaller models are fine-tuned. The results indicate that MLMCompress, when using the best-performing model, achieved 3838% higher compression ratios for English text and 42% higher compression ratios for multilingual text compared to NNCP, another learning-based method. Although the method does not yield better results than GPTZip, which has been developed in recent years, it achieves comparable outcomes while being up to 35 times faster in the worst-case scenario. Additionally, it demonstrated a 20% improvement in compression speed and a 180% improvement in decompression speed in the best case. Furthermore, MLMCompress outperforms traditional compression methods like Gzip and specialized short text compression methods such as Smaz and Shoco, particularly in compressing short texts, even when using smaller models.

Implementing a Secure Cloud-Based System to Safeguard Sensitive Medical Data for Healthcare

In most developed countries, the medical healthcare system is experiencing rapid development from the stage of clinical information to the stage of information dissemination. In all of these countries, it is undeniable that the Internet of Medical Things (IoMT) technologies have contributed in order to develop information medical healthcare. In reality, the development of smart medical healthcare has been hindered by the protection of medical privacy, according to research and acceptance. This is especially true as telecommunications systems continue to expand and wireless sensor networks (WSN) develop, as well as ways to penetrate those checks that have become increasingly difficult. In the smart healthcare system, protecting users’ information remains an outstanding issue. IoMT features and the protection of privacy and security have led to the development of an extended privacy homomorphism algorithm based on scrambling matrixes, an encryption algorithm enhanced by Modified RSA (mRSA), and a method of encrypted data compression that ensures data confidentiality. For the above purpose, we built a prototype system on a demo temporary domain using both hardware and software. According to the results, the proposed scheme protects E-healthcare from potential threats by providing stakeholders with a secure interface and preventing unauthorized users from accessing the mCloud, thus ensuring privacy. E-healthcare services based on cloud technology are protected by the proposed scheme because it is simple and robust.

Improvement of Lossless Text Compression Methods using a Hybrid Method by the Integration of RLE, LZW and Huffman Coding Algorithms

In the field of computer science, data compression is essential in the process of data transfer because it reduces file size without losing information. Issues that depend on the characteristics of the text can greatly affect the effectiveness of any compression algorithm. Consequently, each traditional compression algorithm has its own strengths and limitations. Therefore, a highly efficient compression method that achieves higher compression ratio is needed. In this paper, based on the sequential implementation of Huffman, RLE and LZW algorithms, a hybrid data compression method has been proposed. The algorithms have been individually evaluated and compared in terms of compression ratio and compressed file size. We conclude that in terms of compression ratio and compressed file size, LZW is better than Huffman and RLE. The results show that the proposed hybrid algorithm achieves the highest average compression ratio is 2.42 and the lowest average compressed file size is 1138.5 or less compared to the individual implementations of the algorithms. Thus, it can be concluded that the proposed hybrid method enhances lossless text compression in terms of compression ratio and file size according to compression metrics.

Medium-adaptive compressive diffuse optical tomography.

The low spatial resolution of diffuse optical tomography (DOT) has motivated the development of high-density DOT systems utilizing spatially-encoded illumination and detection strategies. Data compression methods, through the application of Fourier or Hadamard patterns, have been commonly explored for both illumination and detection but were largely limited to pre-determined patterns regardless of imaging targets. Here, we show that target-optimized detection patterns can yield significantly improved DOT reconstructions in both in silico and experimental tests. Applying reciprocity, we can further iteratively optimize both illumination and detection patterns and show that these simultaneously optimized source/detection patterns outperform predetermined patterns in simulation settings. In addition, we show media-adaptive measurement data compression methods enable wide-field DOT systems to recover highly complex inclusions inside optically-thick media with reduced background artifacts. Furthermore, using truncated optimized patterns shows an improvement of 2-4× in increased speed of data acquisition and reconstruction without significantly losing image quality. The proposed method can be readily extended for additional data dimensions such as spectrum and time.

Wavefield Reconstruction of Distributed Acoustic Sensing: Lossy Compression, Wavefield Separation, and Edge Computing

AbstractDistributed acoustic sensing (DAS) presents challenges and opportunities for seismological research and data management. This study explores wavefield reconstruction using deep learning methods for data compression and wavefield separation. We test various architectures to treat DAS data as two‐dimensional arrays, such as the implicit neural representation (INR) models and the SHallow REcurrent Decoder (SHRED) model. The INR models present better data compression ability but do not generalize over space and time, a major practical limitation. On the other hand, SHRED generalizes over space and time for a single optical fiber with data from 20% decimated channels for the reconstruction. Despite good performance in reconstructing long‐wavelength features, the shallow recurrent decoder does not reconstruct transient earthquake wavefields at shorter wavelengths, limiting its usability for seismic data transmission. Nevertheless, we leverage wavefield reconstruction of ocean waves to separate them from the seismic wavefield and improve seismological use cases for earthquake detection and Earth imaging. In summary, as a lightweight deep learning model, SHRED is well suited for wavefield separation and lossy compression of the DAS data.

Compression of Battery X-Ray Tomography Data with Machine Learning

Abstract With the increasing demand for high-resolution x-ray tomography in battery characterization, the challenges of storing, transmitting, and analyzing substantial imaging data necessitate more efficient solutions. Traditional data compression methods struggle to balance reduction ratio and image quality, often failing to preserve critical details for accurate analysis. This study proposes a machine learning-assisted compression method tailored for battery x-ray imaging data. Leveraging physics-informed representation learning, our approach significantly reduces file sizes without sacrificing meaningful information. We validate the method on typical battery materials and different x-ray imaging techniques, demonstrating its effectiveness in preserving structural and chemical details. Experimental results show an up-to-95 compression ratio while maintaining high fidelity in the projection and reconstructed images. The proposed framework provides a promising solution for managing large-scale battery x-ray imaging datasets, facilitating significant advancements in battery research and development.

Advanced computer architecture optimization for machine learning/deep learning

Abstract The recent progress in Machine Learning (Géron, 2022) and particularly Deep Learning (Goodfellow, 2016) models exposed the limitations of traditional computer architectures. Modern algorithms demonstrate highly increased computational demands and data requirements that most existing architectures cannot handle efficiently. These demands result in training speed, inference latency, and power consumption bottlenecks, which is why advanced methods of computer architecture optimization are required to enable the development of ML/DL-dedicated efficient hardware platforms (Engineers, 2019). The optimization of computer architecture for applications of ML/DL becomes critical, due to the tremendous demand for efficient execution of complex computations by Neural Networks (Goodfellow, 2016). This paper reviewed the numerous approaches and methods utilized to optimize computer architecture for ML/DL workloads. The following sections contain substantial discussion concerning the hardware-level optimizations, enhancements of traditional software frameworks and their unique versions, and innovative explorations of architectures. In particular, we discussed hardware including specialized accelerators, which can improve the performance and efficiency of a computation system using various techniques, specifically describing accelerators like CPUs (multicore) (Hennessy, 2017), GPUs (Hwu, 2015) and TPUs (Contributors, 2017), parallelism in multicore architectures, data movement in hardware systems, especially techniques such as caching and sparsity, compression, and quantization, other special techniques and configurations, such as using specialized data formats, and measurement sparsity. Moreover, this paper provided a comprehensive analysis of current trends in software frameworks, Data Movement optimization strategies (A.Bienz, 2021), sparsity, quantization and compression methods, using ML for architecture exploration, and, DVFS (Hennessy, 2017),, which provides strategies for maximizing hardware utilization and power consumption during training, machine learning, dynamic voltage, and frequency scaling, runtime systems. Finally, the paper discussed research opportunity directions and the possibilities of computer architecture optimization influence in various industrial and academic areas of ML/DL technologies. The objective of implementing these optimization techniques is to largely minimize the current gap between the computational needs of ML/DL algorithms and the current hardware’s capability. This will lead to significant improvements in training times, enable real-time inference for various applications, and ultimately unlock the full potential of cutting-edge machine learning algorithms.

A signal-adaptive measurement matrix construction algorithm for compressed sensing of sEMG data

Abstract Surface electromyography (sEMG), serving as a pivotal wearable technology, is a promising tool to assess and monitor muscle function. Yet, the efficacy of a sEMG system faces inevitable constraints, primarily stemming from the challenges of transmission and energy consumption induced by big data. Compressed sensing (CS) is a promising data acquisition solution that takes advantage of the signal sparseness in a particular basis to significantly reduce the number of samples. Current CS methods usually employ random or deterministic measurement matrix to compress sEMG signal. However, these measurement matrices do not integrate the signal feature, which limits the performance of these CS methods. To address this problem, this paper proposes an improved CS method for sEMG data compression. This proposed method introduces a measurement matrix construction algorithm to produce a deterministic matrix tailored for processing sEMG signals. The deterministic measurement matrix integrates the characteristics of the magnitudes of sEMG signals. The sEMG signals acquired from the upper limb muscles of the stroke survivors were applied to evaluate the proposed CS method, with results showing that it achieves better reconstruction accuracy and robustness than the CS methods with other measurement matrices. The proposed method employing basis pursuit (BP) in the signal reconstruction presents better performance than that employing orthogonal matching pursuit (OMP). Hence, we can conclude that the proposed CS algorithm is of key importance for the popularization of sEMG in the wearable health monitoring devices.

Deep learning model for fiber optic distributed acoustic sensing seismic data compression based on autoencoder and recurrent neural networks

Because of the extremely high sampling rate of fiber optic distributed acoustic sensing (DAS) equipment, the amount of DAS seismic data collected is enormous, which poses great challenges to the transmission and storage of DAS seismic data. Therefore, it is essential to study the compression methods of DAS seismic data. Existing data compression methods such as wavelet transform, cosine transform, and convolutional autoencoder (CAE) still have room for improvement in the compression performance and compression ratio (CR). Thus, we have proposed what we believe to be is a novel deep learning model called the recurrent autoencoder (RAE) for high-performance compression of DAS seismic data. Under different CRs, we have designed performance evaluation experiments for RAE models based on different RNN modules with different loss functions. When the CR of the RAE model is 8, the signal-to-noise ratio (SNR) of the reconstructed DAS seismic data reaches 40.60 dB, which is better than that of the CAE’s 11.58 dB. The ultimate CR was increased to 512 without reducing the compression quality, which is 4.12 times higher than the CAE model. The LSTM with a weighted loss function improves the SNR to 43.66 dB at a CR of 8, which is 3.06 dB higher than the LSTM with additive loss function. The results show that the RAE model proposed with a weighted loss function in this paper has excellent DAS seismic data compression performance and provides a high CR, which can be widely applied in large-scale DAS seismic data compression.

Review of Modern Neural Network Compression Methods for the Task of Processing Measurement Data

In the article, the authors consider neural network methods of data compression, both lossy and lossless. The principles of operation and training of the following architectures are described in detail: transformer, recurrent neural networks with “long short-term memory”, bottleneck autoencoder, variational autoencoder and generative adversarial network. The efficiency of their work has been compared with classical compression algorithms and a conclusion has been made about the potential and prospects of their application.

Building Test Batteries Based on Analyzing Random Number Generator Tests within the Framework of Algorithmic Information Theory.

The problem of testing random number generators is considered and a new method for comparing the power of different statistical tests is proposed. It is based on the definitions of random sequence developed in the framework of algorithmic information theory and allows comparing the power of different tests in some cases when the available methods of mathematical statistics do not distinguish between tests. In particular, it is shown that tests based on data compression methods using dictionaries should be included in test batteries.

Meta-Meshing and Triangulating Lattice Structures at a Large Scale

Lattice structures have been widely used in applications due to their superior mechanical properties. To fabricate such structures, a geometric processing step called triangulation is often employed to transform them into the STL format before sending them to 3D printers. Because lattice structures tend to have high geometric complexity, this step usually generates a large amount of triangles, a memory and compute-intensive task. This problem manifests itself clearly through large-scale lattice structures that have millions or billions of struts. To address this problem, this paper proposes to transform a lattice structure into an intermediate model called meta-mesh before undergoing real triangulation. Compared to triangular meshes, meta-meshes are very lightweight and much less compute-demanding. The meta-mesh can also work as a base mesh reusable for conveniently and efficiently triangulating lattice structures with arbitrary resolutions. A CPU+GPU asynchronous meta-meshing pipeline has been developed to efficiently generate meta-meshes from lattice structures. It shifts from the thread-centric GPU algorithm design paradigm commonly used in CAD to the recent warp-centric design paradigm to achieve high performance. This is achieved by a new data compression method, a GPU cache-aware data structure, and a workload-balanced scheduling method that can significantly reduce memory divergence and branch divergence. Experimenting with various billion-scale lattice structures, the proposed method is seen to be two orders of magnitude faster than previously achievable.

ДОСЛІДЖЕННЯ МЕТОДУ ФРАКТАЛЬНОГО СТИСНЕННЯ ЗОБРАЖЕНЬ З МЕТОЮ ПОКРАЩЕННЯ ЯКОСТІ СТИСНЕННЯ

The development of the Internet, along with the availability of increasingly powerful computers and other digital devices, cameras, scanners and printers, has led to the wide-spread use of digital images. In this regard, interest in improving data compression algo-rithms, such as images, is growing. Data compression is important for both transfer speed and storage efficiency. In addition to many commercial uses, compression technologies are also of interest in the military industry, such as applications for processing telemetry data from missile inter-ceptors or for archiving terrain image data for defense simulations. Solving the problem of image compression, or, more generally, image coding, has used advances and stimulated the development of many fields of engineering and math-ematics. The article examines fractal image compression — a data compression method based on the use of self-similar patterns in an image. This method allows you to achieve a high degree of compression while preserving image details. Fractal image compression is a unique and efficient approach to data compression based on the mathematical theory of fractals. Nowadays, it has important applications and advantages that make it a valuable tool in image processing. The main advantages include: 1. Preservation of details during compression. One of the key advantages of fractal compression is its ability to preserve a high degree of image detail in a relatively small amount of storage. This is especially important in situations where image quality must be preserved with limited storage and data transfer resources. 2. Efficiency of transmission through the network. Fractal compression allows for compact images, making it suitable for image transmission over a low-bandwidth network. This is especially true for mobile devices, the Internet of Things, and other scenarios where high bandwidth is not always available. 3. Adaptive compression for different resolutions. Fractal compression allows you to adapt the level of compression depending on the resolution and details of the image. This means that it can be used to compress various image sizes without significant loss of quality. 4. Data archiving and storage. Fractal compression can be useful for archiving and long-term storage of images, as it allows you to effectively reduce the amount of data without losing important information. This is especially important for libraries, archives, research databases and other data repositories. Fractal image compression remains a relevant and valuable tool in today's envi-ronment, thanks to its ability to efficiently compress, preserve details, and adapt to dif-ferent usage scenarios. Therefore, the study of its efficiency, the optimization of the soft-ware code to obtain a faster and better compression result, is an urgent task.

A hybrid terrain data compression method in unity for deployment on resource-limited devices

With geographic information systems in general and simulation systems in particular, terrain data takes up most of the hard drive space and is often deployed on data servers. The higher the resolution of the terrain data, the more detailed it is, the larger the space occupied on the hard drive. If terrain data needs to be deployed offline on resource-limited devices such as minicomputers, it will face many difficulties due to hard drive space limitations. Terrain data compression is a solution that reduces terrain capacity to overcome that problem. This article presents an efficient hybrid approach based on Brotli and LZ4 compression algorithms to compress terrain data for deployment on resource-limited devices. Experimental results show that the proposed method significantly reduces the volume of terrain data compared to using each component algorithm independently while still ensuring quality.

Research on the effectiveness of data compression methods in relational and NoSQL DBMS

Research on the effectiveness of data compression methods in relational and NoSQL DBMS

Lossless Compression Method for the Magnetic and Helioseismic Imager (MHI) Payload

The Solar Polar-orbit Observatory (SPO), proposed by Chinese scientists, is designed to observe the solar polar regions in an unprecedented way with a spacecraft traveling in a large solar inclination angle and a small ellipticity. However, one of the most significant challenges lies in ultra-long-distance data transmission, particularly for the Magnetic and Helioseismic Imager (MHI), which is the most important payload and generates the largest volume of data in SPO. In this paper, we propose a tailored lossless data compression method based on the measurement mode and characteristics of MHI data. The background out of the solar disk is removed to decrease the pixel number of an image under compression. Multiple predictive coding methods are combined to eliminate the redundancy utilizing the correlation (space, spectrum, and polarization) in data set, improving the compression ratio. Experimental results demonstrate that our method achieves an average compression ratio of 3.67. The compression time is also less than the general observation period. The method exhibits strong feasibility and can be easily adapted to MHI.

Novel lossy compression method of noisy time series data with anomalies: Application to partial discharge monitoring in overhead power lines

In overhead power transmission lines, particularly in regions like natural parks where establishing a safe zone is difficult, the adoption of cross-linked polyethylene insulated covered conductors (CCs) helps prevent outages due to vegetation contact. However, these CCs are susceptible to partial discharge (PD) activity, which can degrade insulation and lead to system failures. Detecting and analyzing PD are essential for maintaining power system reliability and safety. A key challenge in PD monitoring is transmitting the large volumes of PD signal data over unreliable 2G networks, as existing compression methods either compromise on data integrity or are ineffective. This paper introduces a novel lossy compression technique utilizing an autoencoder with skip connections and correction data to address this issue. Unlike previous algorithms that struggle with noisy time series data and fail to preserve crucial anomaly information, our method reconstructs the signal without anomalies, which are subsequently restored using correction data. Achieving a compression factor of about 25 (reducing data to 4.1% of its original size), this approach maintains essential PD signal features for analysis. The effectiveness of our method is validated by three classification algorithms, showing promise for future fault detection, diagnosis, and memory space reduction. This innovative compression solution marks a significant advancement in PD data processing, offering a balanced trade-off between compression efficiency and data fidelity, and paving the way for enhanced remote monitoring in power transmission systems.

The Data Compression Method and FPGA Implementation in the Mars Rover Subsurface-Penetrating Radar on the Tianwen-1 Mission

Since Mars is far away from Earth, the propagation delay between Mars and Earth is very large. To ensure the effective use of the link transmission bandwidth, China’s first Mars exploration mission has put forward a demand for data compression for all scientific payloads. The on-board mature algorithms for data compression are mainly focused on optical images and microwave imaging radar applications. No articles have been published on data compression methods that are applied to subsurface-penetrating radar. Based on the background of this application, this paper proposes a logarithmic lossy compression algorithm which can meet the mission requirements for high compression ratios of 4:1 and 2.5:1. Its compression error is less than that of the block adaptive quantization (BAQ) algorithm. The algorithm is not only easy to implement on field-programmable gate array (FPGA) platforms, but also offers simple ground decompression and fast imaging. The experimental results show that high compression ratios of 4:1 and 2.5:1 can be realized, even if the data in and between traces do not have a strong correlation. And its relative error is less than 2%, which is a new type of high-efficiency data compression method that can be implemented on-board to meet with the demand of subsurface penetrating radar.

Data Compression and Damage Evaluation of Underground Pipeline With Musicalized Sonar GMM

The underground cement pipeline network is critical in urban infrastructure, such as sewage drainage, fluid transportation, etc. Pipelines are deeply buried, so the damage is difficult to be detected. It is of great significant to monitor the crack of pipeline in real time. In this research, a data compression and damage evaluation method of underground pipeline using musicalized sonar GMM (Gaussian Mixed Model) was proposed. First, the sonar echo signal was sampled and discretized into four musicalized indicators by musicalized model. Then, the distribution of multi parameters were analyzed by GMM. The Frechet similarity was constructed to compare the difference between each working condition and baseline condition, so that the damage state could be identified. The engineering experiment shown that this method could greatly compress the monitoring data and evaluate four crack levels.

Using convolutional neural networks to select the optimal wavelet for audio compression

The vast amount of data generated and stored across domains is overwhelming. Noteworthy examples include the Event Horizon Telescope using 3.5 petabytes for the first black hole image, LIGO amassing 4.5 petabytes, and CERN with 487 petabytes. This research addresses the challenge of data storage and transmission using machine learning and wavelet analysis. The goal is to create an efficient data compression method that significantly reduces storage needs without compromising data integrity. Wavelets, mathematical tools which break down signals into coefficients, show promise for compression. Choosing the right wavelet is crucial; our research demonstrates that machine learning can effectively solve this problem. By training a convolutional neural network on spectrograms of audio files, we identify the optimal wavelet for data compression, achieving up to 50% compression with minimal loss.