Lossless Compression Algorithm Research Articles

ACTF: An efficient lossless compression algorithm for time series floating point data

ACTF: An efficient lossless compression algorithm for time series floating point data

Parallel Lossless Compression of Raw Bayer Images on FPGA-Based High-Speed Camera

Digital image compression is applied to reduce camera bandwidth and storage requirements, but real-time lossless compression on a high-speed high-resolution camera is a challenging task. The article presents hardware implementation of a Bayer colour filter array lossless image compression algorithm on an FPGA-based camera. The compression algorithm reduces colour and spatial redundancy and employs Golomb–Rice entropy coding. A rule limiting the maximum code length is introduced for the edge cases. The proposed algorithm is based on integer operators for efficient hardware implementation. The algorithm is first verified as a C++ model and later implemented on AMD-Xilinx Zynq UltraScale+ device using VHDL. An effective tree-like pipeline structure is proposed to concatenate codes of compressed pixel data to generate a bitstream representing data of 16 parallel pixels. The proposed parallel compression achieves up to 56% reduction in image size for high-resolution images. Pipelined implementation without any state machine ensures operating frequencies up to 320 MHz. Parallelised operation on 16 pixels effectively increases data throughput to 40 Gbit/s while keeping the total memory requirements low due to real-time processing.

Performance Analysis of Compression Algorithms on Matrix Data: Data Transfer Optimization in Microservices Architectures

With the rapid proliferation of microservices architectures these days, the efficient and fast transfer of large matrix data between services has become a significant challenge. This study presents an analysis aimed at finding solutions to this challenge. The analysis addresses the compression and decompression of large matrix data, focusing on lossless compression algorithms to optimize data transfer without data loss. The study is implemented on an example scenario. This scenario is taken from a project with a microservice architecture. In the example scenario, an image processing service developed in Python programming language generates 640x480 matrix data. After going through a compression algorithm, this data is periodically transferred to a backend service developed in C# programming language. This data is then stored in a database. In the final stage, decompression operations are performed so that this data can be used for reporting. The performance of various compression algorithms in the data compression, database storage and report generation stages is extensively tested. Within the scope of the study, tests were performed using five different compression algorithms (Gzip, Zlib, Deflate, Brotli and Bz2). The results are obtained through performance tests aimed at determining the most optimized end-to-end solution. Analyzing the performance of the compression algorithms on the example scenario, the Brotli algorithm gives the most optimal result in terms of both speed and compression size. This work makes an important contribution to data transfer optimization in microservice architectures and provides a reference for research in this area by presenting the performance analysis of various compression algorithms.

A New Algorithm for Lossless Data Compression Based of Time Clock Mechanism

A New Algorithm for Lossless Data Compression Based of Time Clock Mechanism

Improved JPEG Lossless Compression for Compression of Intermediate Layers in Neural Networks Based on Compute-In-Memory

With the development of Convolutional Neural Networks (CNNs), there is a growing requirement for their deployment on edge devices. At the same time, Compute-In-Memory (CIM) technology has gained significant attention in edge CNN applications due to its ability to minimize data movement between memory and computing units. However, the deployment of complex deep neural network models on edge devices with restricted hardware resources continues to be challenged by a lack of adequate storage for intermediate layer data. In this article, we propose an optimized JPEG Lossless Compression (JPEG-LS) algorithm that implements serial context parameter updating alongside parallel encoding. This method is designed for the global prediction and efficient compression of intermediate data layers in neural networks employing CIM techniques. The results indicate average compression ratios of 6.44× for VGG16, 3.62× for ResNet34, 1.67× for MobileNetV2, and 2.31× for InceptionV3. Moreover, the implementation achieves a data throughput of 32 bits per cycle at 600 MHz on the TSMC 28 nm, with a hardware cost of 122 K Gate Count.

A Comparative Study of Text-Based Lossless Compression

Lossless data compression is a critical technique used to reduce file sizes without any loss of information during the encoding and decoding processes. This study presents a comparative analysis of two widely-used lossless compression algorithms: Huffman Encoding and Lempel-Ziv-Welch (LZW). The primary objective is to evaluate the performance of these algorithms in terms of compression ratio, compression time, decompression time, and space savings. The analysis was conducted on 100 files of varying sizes. The results demonstrate that the LZW algorithm outperforms Huffman Encoding, offering superior compression ratios, faster compression and decompression times, and greater disk space savings. These findings highlight the effectiveness of LZW for efficient data compression in practical applications.

Assembly Theory is an approximation to algorithmic complexity based on LZ compression that does not explain selection or evolution

We formally prove the equivalence between Assembly Theory (AT) and Shannon Entropy via a method based upon the principles of statistical compression that belongs to the LZ family of popular compression algorithms. Such popular lossless compression algorithms behind file formats such as ZIP and PNG have been shown to empirically reproduce the results that AT considers its cornerstone. The same results have also been reported before AT in successful application of other complexity measures in the areas covered by AT such as separating organic from non-organic molecules and in the context of the study of selection and evolution. We demonstrate that the assembly index is equivalent to the size of a minimal context-free grammar. The statistical compressibility of such a method is bounded by Shannon Entropy and other equivalent traditional LZ compression schemes, such as LZ77 and LZW. We also demonstrate that AT, and the algorithms supporting its pathway complexity, assembly index, and assembly number, define compression schemes and methods that are subsumed into algorithmic information theory. We conclude that the assembly index and the assembly number do not lead to an explanation or quantification of biases in generative (physical or biological) processes, including those brought about by (abiotic or biotic) selection and evolution, that could not have been arrived at using Shannon Entropy, or that have not been already reported before using classical information theory or algorithmic complexity.

A multi-predictor based lossless ARGB texture compression algorithm and FPGA implementation

. In the fields of GPU and AI chip design, the frequent read and write operations on the color buffer data (ARGB), which are intensive in graphical and image access, significantly impact performance. There is a need for applications that require random access and only read small images once. To address this situation, this paper proposes an algorithm with lower modeling complexity, yet achieving near-complex implementation results, along with its FPGA implementation method. Through actual testing on multiple images, the average lossless compression rate reached 40.3%. With hardware acceleration, the execution efficiency of the algorithm was further improved, ensuring both compression rate and speed, thus confirming the effectiveness of the algorithm.

Research and Implementation of FPGA Image Compression Parallel Algorithm Based on CCSDS

High-pixel, high-resolution spectral images are an important source of key graphical information obtained from space remote sensing, however, with the dramatically expanding amount of graphical information, the limited transmission bandwidth and hardware resources are difficult to meet the demand for massive data, which makes real-time image lossless compression and transmission technology an important solution. For this reason, this paper is based on the CCSDS image lossless compression algorithm, using FPGA for the construction of image compression system, and proposes a method strategy based on parallel acceleration algorithm to realize the real time processing of large-capacity hyperspectral images. The simulation results show that hyperspectral river and mountain range images can be compressed with greatly improved compression efficiency while maintaining real time performance and accuracy.

DeltaGzip: Computing Biopolymer-Ligand Binding Affinity via Kolmogorov Complexity and Lossless Compression.

The design of biosequences for biosensing and therapeutics is a challenging multistep search and optimization task. In principle, computational modeling may speed up the design process by virtual screening of sequences based on their binding affinities to target molecules. However, in practice, existing machine-learned models trained to predict binding affinities lack the flexibility with respect to reaction conditions, and molecular dynamics simulations that can incorporate reaction conditions suffer from high computational costs. Here, we describe a computational approach called DeltaGzip that evaluates the free energy of binding in biopolymer-ligand complexes from ultrashort equilibrium molecular dynamics simulations. The entropy of binding is evaluated using the Kolmogorov complexity definition of entropy and approximated using a lossless compression algorithm, Gzip. We benchmark the method on a well-studied data set of protein-ligand complexes comparing the predictions of DeltaGzip to the free energies of binding obtained using Jarzynski equality and experimental measurements.

DEVELOPMENT OF AEROSPACE IMAGES PRELIMINARY PROCESSING METHOD FOR SUBSEQUENT RECOGNITION AND IDENTIFICATION OF VARIOUS OBJECTS

Nowadays, the application of hyperspectral images is vital for every section of the humanity life such as agrotechnical research for the field condition state and water security. This article presents a new lossless data compression algorithm focused on the processing of hyperspectral aerospace images. The algorithm takes into account inter-band correlation and difference transformations to effectively reduce the range of initial values. correlation allows you to find the best reference channel that defines the sequence of operations in the algorithm, which contributes to a significant increase in the compression ratio while maintaining high data quality. The practical implementation of the algorithm lies in the process of the transfer the lower size file with high efficiency for unmanned aerial vehicle and satellites to save more computational resources. This method demonstrates high computational efficiency and can be applied to various tasks that require efficient storage and transmission of hyperspectral images. The importance of processing hyperspectral data and the problems associated with their volume and complexity of analysis were described. Current approaches to data compression are considered and their limitations are identified, which justifies the need to develop new methods. The relevance and necessity of effective compression algorithms for aerospace applications is emphasized. An analysis of existing methods and algorithms for compressing hyperspectral data was carried out. Particular attention is paid to approaches that use cross-channel correlation and difference transformations. The effectiveness of current methods is evaluated and their shortcomings are identified, which serves as a justification for the development of a new algorithm. A developed lossless data compression algorithm based on the use of inter-band correlation and difference transformations was described. The stages of forming groups of channels and the selection of optimal compression parameters are considered in detail. The method of determining the reference channel, which sets the sequence of operations in the algorithm, which provides more efficient data compression, is especially noted. The advantages and possible limitations of the new approach, as well as its potential for practical use, are discussed. It was noted that the developed method successfully solves the problems associated with the volume of hyperspectral data, providing a high compression ratio without quality loss. The prospects for further development of the algorithm and its application in various fields of science and technology are discussed.

A SoC-FPGA based readout platform for the free-running AMBER data acquisition system

Owing to a change in the scope of physics research, the AMBER spectrometer at CERN is undergoing an update on its instrumentation and trigger strategy for the data acquisition system. One of the key updates is the adoption of a free-running and trigger-less operation, which differs from the predecessor of AMBER, COMPASS, by working in a continuous mode but in an event-based triggered way.In this article, we present a multichannel data-acquisition platform developed in line with the next generation of trigger-less and free-running data acquisition systems. The platform is based on a pre-existing Mezzanine Sampling ADC board for the analog-to-digital conversion and a Xilinx Zynq Ultrascale+ System on Module for online real-time data processing. We also present the evaluation of the system operating in continuous mode, taking data from a 25-element electromagnetic calorimeter prototype with a muon beam, during the first AMBER pilot run. The acquired data were used for noise and pulse shape studies required for the design of the algorithms for lossless compression and data feature extraction needed for trigger-less operation.

High performance holographic video compression using spatio-temporal phase unwrapping

In this work, we present a high-performance holographic data compression technique. This approach is based in the temporal correlation found in a holographic video generated using an optimized random phase. This temporal correlation makes possible high levels of compression by using a spatio-temporal (ST) representation, partial phase unwrapping, and a lossless compression algorithm like DEFLATE. In this approach, two holographic videos are created using the optimized random phase (ORAP) technique and the Gerchberg-Saxton (GS) algorithm. The phase holograms corresponding to each frame of the videos are arranged into a ST representation, enabling spatial and temporal phase unwrapping. We test phase unwrapping along the spatial and temporal dimensions separately, followed by volume reduction after applying the DEFLATE lossless compression algorithm to the unwrapped phase in each case. The holographic video is then reconstructed from the compressed temporal and spatial unwrapped phase. The results demonstrate the high efficiency of the proposed method, with ORAP holograms achieving a compression percentage of 92.670 % after applying DEFLATE to a ST representation with partial phase unwrapping along the temporal dimension.

Investigation of fast and efficient lossless compression algorithms for macromolecular crystallography experiments.

Structural biology experiments benefit significantly from state-of-the-art synchrotron data collection. One can acquire macromolecular crystallography (MX) diffraction data on large-area photon-counting pixel-array detectors at framing rates exceeding 1000 frames per second, using 200 Gbps network connectivity, or higher when available. In extreme cases this represents a raw data throughput of about 25 GB s-1, which is nearly impossible to deliver at reasonable cost without compression. Our field has used lossless compression for decades to make such data collection manageable. Many MX beamlines are now fitted with DECTRIS Eiger detectors, all of which are delivered with optimized compression algorithms by default, and they perform well with current framing rates and typical diffraction data. However, better lossless compression algorithms have been developed and are now available to the research community. Here one of the latest and most promising lossless compression algorithms is investigated on a variety of diffraction data like those routinely acquired at state-of-the-art MX beamlines.

47‐4: A Novel Demura Compensation Data Compression Algorithm based on JPEG‐LS

JPEG‐LS is a lossless/near‐lossless image compression algorithm based on context modeling, which has the advantages of easy implementation, low resource consumption and high compression rate. It is widely used in the compression field of continuous‐tone still images. However, single image compression has significant latency and high hardware resource consumption issues, as the JPEG‐LS algorithm requires pixel by pixel prediction and real‐time context updates during the compression process, which is not conducive to algorithm IP implementation. Based on Demura compensation data compression requirements, this article has made changes to the pixel prediction method of the JPEG‐LS algorithm and delayed the update of the context to solve the compression latency and resource consumption issues. After algorithm optimization (NJPEGLS), the linebuffer resource occupation was reduced by 8/9, the clock frequency reached above 140 MHz, and the comprehensive compression rate loss was within 4.5%, meeting the demand for Demura compensation data compression/decompression. In this paper, we also conducted statistics and analysis on the distribution of values, and found that the distribution of values was very regular, and we propose an adaptive value scheme (KNJPEG‐LS) to further simplify the hardware circuit.

Leveraging Seed Generation for Efficient Hardware Acceleration of Lossless Compression of Remotely Sensed Hyperspectral Images

In the field of satellite imaging, effectively managing the enormous volumes of data from remotely sensed hyperspectral images presents significant challenges due to the limited bandwidth and power available in spaceborne systems. In this paper, we describe the hardware acceleration of a highly efficient lossless compression algorithm, specifically designed for real-time hyperspectral image processing on FPGA platforms. The algorithm utilizes an innovative seed generation method for square root calculations to significantly boost data throughput and reduce energy consumption, both of which represent key factors in satellite operations. When implemented on the Cyclone V FPGA, our method achieves a notable operational throughput of 1598.67 Mega Samples per second (MSps) and maintains a power requirement of under 1 Watt, leading to an efficiency rate of 1829.1 MSps/Watt. A comparative analysis with existing and related state-of-the-art implementations confirms that our system surpasses conventional performance standards, thus facilitating the efficient processing of large-scale hyperspectral datasets, especially in environments where throughput and low energy consumption are prioritized.

Triangular matrix-based lossless compression algorithm for 3D mesh connectivity

Triangular matrix-based lossless compression algorithm for 3D mesh connectivity

Lossless and Near-Lossless Compression Algorithms for Remotely Sensed Hyperspectral Images.

Rapid and continuous advancements in remote sensing technology have resulted in finer resolutions and higher acquisition rates of hyperspectral images (HSIs). These developments have triggered a need for new processing techniques brought about by the confined power and constrained hardware resources aboard satellites. This article proposes two novel lossless and near-lossless compression methods, employing our recent seed generation and quadrature-based square rooting algorithms, respectively. The main advantage of the former method lies in its acceptable complexity utilizing simple arithmetic operations, making it suitable for real-time onboard compression. In addition, this near-lossless compressor could be incorporated for hard-to-compress images offering a stabilized reduction at nearly 40% with a maximum relative error of 0.33 and a maximum absolute error of 30. Our results also show that a lossless compression performance, in terms of compression ratio, of up to 2.6 is achieved when testing with hyperspectral images from the Corpus dataset. Further, an improvement in the compression rate over the state-of-the-art k2-raster technique is realized for most of these HSIs by all four variations of our proposed lossless compression method. In particular, a data reduction enhancement of up to 29.89% is realized when comparing their respective geometric mean values.

A Compression Method for Power Quality Data

This paper presents a compression method for power quality (PQ) signals. This method consists of polynomial approximation algorithm and a lossless compression algorithm The proposed method uses the Deflate algorithm as lossless compression algorithm. The polynomial approximation is intended to decrease the entropy of the signal, thus increasing the compression ratio of the Deflate algorithm. The proposed method is characterized by high compression ratio and low computational cost, making feasible its implementation in embedded systems.

Evaluation on high-performance image compaction algorithms in spatio-temporal data processing

With the passage of time, the importance of spatio-temporal data (STD) is increasing day by day, but the spatiotemporal characteristics of STD bring huge challenges to data processing. Aiming at the problems of image information loss, limited compression ratio, slow compression speed and low compression efficiency, this method based on image compression. This article intended to focus on aircraft trajectory data, meteorological data, and remote sensing image data as the main research objects. The research results would provide more accurate and effective data support for research in related fields. The image compaction algorithm based on deep learning in this article consisted of two parts: encoder and decoder, and this method was compared with the JPEG (Joint Photographic Experts Group) method. When compressing meteorological data, the algorithm proposed in this paper can achieve a maximum compaction rate of 0.400, while the maximum compaction rate of the JPEG compaction algorithm was only 0.322. If a set of aircraft trajectory data containing 100 data points is compressed to 2:1, the storage space required for the algorithm in this paper is 4.2 MB, while the storage space required for the lossless compression algorithm is 5.6 MB, which increases the compression space by 33.33%. This article adopted an image compaction algorithm based on deep learning and data preprocessing, which can significantly improve the speed and quality of image compaction while maintaining the same compaction rate, and effectively compress spatial and temporal dimensional data.