Lossless Compression Algorithm Research Articles

Investigation of fast and efficient lossless compression algorithms for macromolecular crystallography experiments.

Structural biology experiments benefit significantly from state-of-the-art synchrotron data collection. One can acquire macromolecular crystallography (MX) diffraction data on large-area photon-counting pixel-array detectors at framing rates exceeding 1000 frames per second, using 200 Gbps network connectivity, or higher when available. In extreme cases this represents a raw data throughput of about 25 GB s-1, which is nearly impossible to deliver at reasonable cost without compression. Our field has used lossless compression for decades to make such data collection manageable. Many MX beamlines are now fitted with DECTRIS Eiger detectors, all of which are delivered with optimized compression algorithms by default, and they perform well with current framing rates and typical diffraction data. However, better lossless compression algorithms have been developed and are now available to the research community. Here one of the latest and most promising lossless compression algorithms is investigated on a variety of diffraction data like those routinely acquired at state-of-the-art MX beamlines.

Leveraging Seed Generation for Efficient Hardware Acceleration of Lossless Compression of Remotely Sensed Hyperspectral Images

In the field of satellite imaging, effectively managing the enormous volumes of data from remotely sensed hyperspectral images presents significant challenges due to the limited bandwidth and power available in spaceborne systems. In this paper, we describe the hardware acceleration of a highly efficient lossless compression algorithm, specifically designed for real-time hyperspectral image processing on FPGA platforms. The algorithm utilizes an innovative seed generation method for square root calculations to significantly boost data throughput and reduce energy consumption, both of which represent key factors in satellite operations. When implemented on the Cyclone V FPGA, our method achieves a notable operational throughput of 1598.67 Mega Samples per second (MSps) and maintains a power requirement of under 1 Watt, leading to an efficiency rate of 1829.1 MSps/Watt. A comparative analysis with existing and related state-of-the-art implementations confirms that our system surpasses conventional performance standards, thus facilitating the efficient processing of large-scale hyperspectral datasets, especially in environments where throughput and low energy consumption are prioritized.

Triangular matrix-based lossless compression algorithm for 3D mesh connectivity

Triangular matrix-based lossless compression algorithm for 3D mesh connectivity

Lossless and Near-Lossless Compression Algorithms for Remotely Sensed Hyperspectral Images.

Rapid and continuous advancements in remote sensing technology have resulted in finer resolutions and higher acquisition rates of hyperspectral images (HSIs). These developments have triggered a need for new processing techniques brought about by the confined power and constrained hardware resources aboard satellites. This article proposes two novel lossless and near-lossless compression methods, employing our recent seed generation and quadrature-based square rooting algorithms, respectively. The main advantage of the former method lies in its acceptable complexity utilizing simple arithmetic operations, making it suitable for real-time onboard compression. In addition, this near-lossless compressor could be incorporated for hard-to-compress images offering a stabilized reduction at nearly 40% with a maximum relative error of 0.33 and a maximum absolute error of 30. Our results also show that a lossless compression performance, in terms of compression ratio, of up to 2.6 is achieved when testing with hyperspectral images from the Corpus dataset. Further, an improvement in the compression rate over the state-of-the-art k2-raster technique is realized for most of these HSIs by all four variations of our proposed lossless compression method. In particular, a data reduction enhancement of up to 29.89% is realized when comparing their respective geometric mean values.

Evaluation on high-performance image compaction algorithms in spatio-temporal data processing

With the passage of time, the importance of spatio-temporal data (STD) is increasing day by day, but the spatiotemporal characteristics of STD bring huge challenges to data processing. Aiming at the problems of image information loss, limited compression ratio, slow compression speed and low compression efficiency, this method based on image compression. This article intended to focus on aircraft trajectory data, meteorological data, and remote sensing image data as the main research objects. The research results would provide more accurate and effective data support for research in related fields. The image compaction algorithm based on deep learning in this article consisted of two parts: encoder and decoder, and this method was compared with the JPEG (Joint Photographic Experts Group) method. When compressing meteorological data, the algorithm proposed in this paper can achieve a maximum compaction rate of 0.400, while the maximum compaction rate of the JPEG compaction algorithm was only 0.322. If a set of aircraft trajectory data containing 100 data points is compressed to 2:1, the storage space required for the algorithm in this paper is 4.2 MB, while the storage space required for the lossless compression algorithm is 5.6 MB, which increases the compression space by 33.33%. This article adopted an image compaction algorithm based on deep learning and data preprocessing, which can significantly improve the speed and quality of image compaction while maintaining the same compaction rate, and effectively compress spatial and temporal dimensional data.

Optimizing ATLAS data storage: The impact of compression algorithms on ATLAS physics analysis data formats

The increased footprint foreseen for Run-3 and HL-LHC data will soon expose the limits of currently available storage and CPU resources. Data formats are already optimized according to the processing chain for which they are designed. ATLAS events are stored in ROOT-based reconstruction output files called Analysis Object Data (AOD), which are then processed within the derivation framework to produce Derived AOD (DAOD) files. Numerous DAOD formats, tailored for specific physics and performance groups, have been in use throughout the ATLAS Run-2 phase. In view of Run-3, ATLAS has changed its analysis model, which entailed a significant reduction of the existing DAOD flavors. Two new formats, unfiltered, skimmable on read and designed to meet the requirements of the majority of the analysis workflows, have been proposed as replacements: DAOD_PHYS and DAOD_PHYSLITE, a smaller format containing already calibrated physics objects. As ROOT-based formats, they natively support four lossless compression algorithms: lzma, lz4, zlib and zstd. In this study, the effects of different compression settings on file size, compression time, compression factor and reading speed are investigated considering both DAOD_PHYS and DAOD_PHYSLITE formats. Moreover, the impact of the AutoFlush parameter controlling how in-memory data structures are serialized to ROOT files, has been evaluated. This study yields new quantitative results that can serve as a paradigm on how to make compression decisions for different ATLAS use cases. As an example, for both DAOD_PHYS and DAOD_PHYSLITE, the lz4 library exhibits the fastest reading speed, but results in the largest files, whereas the lzma algorithm provides larger compression factors at the cost of significantly slower reading speeds. In addition, guidelines for setting appropriate AutoFlush values are outlined.

Comprehensive evaluation of lossless compression algorithms in a real use case for smart grid applications

The so-called “energy digitalization” is pervading the power and energy industry by providing many state-of-the-art digital technologies to collect, store, and analyze the very heterogeneous information managed in modern power systems. Requirements in terms of sampling frequency, temporal and user aggregation, measured quantities, measurement aggregations, involved players, and applications can be very different. Compression strategies, removing redundancy and over-specification in collected samples, can play a relevant role in efficiently move and store such heterogeneous data. In this work, a reference dataset from a real-world use case has been collected for comparing the performance of 62 lossless compression algorithms derived from the LZ77/LZSS/LZMA strategies. Compression of the six-day long acquisition, including active power measurements of a prosumer’s system equipped with a Photovoltaic (PV) and a Battery Energy Storage System, highlights the different obtainable performance when the aggregation interval is changed from 15 min to 24 h. In particular, once the top performing algorithms have been selected, it has been possible to stress the impact of night–day cycle, mainly due to the different sparsity of PV-related data. The obtained results demonstrate that, globally, the compression ratio increases by increasing the aggregation interval, by reaching values close to 9.7. In particular, when offline operation is tolerated, optimal compression schemes can be easily applied, leading to consistent improvement of the compression ratio (up to 24%, depending on the actual algorithm and aggregation interval), which can be very significant when large number of data sources is considered.

AFC: An adaptive lossless floating-point compression algorithm in time series database

The time series database is a specialized type of database specifically designed for storing and analyzing time series data. Compression of time series data is crucial for its performance. However, efficiently compressing time series data, particularly floating-point data, remains challenging. Existing compression algorithms are efficient for only a limited range of data patterns, indicating a lack of self-adaptation. In this paper, we propose an effective and Adaptive lossless Floating-point Compression algorithm AFC for time series databases. We devise four unique compression strategies, and based on the data patterns, AFC dynamically selects the appropriate strategy. These strategies handle data compression for diverse data patterns, enhancing the compression ratio and efficiency. The most suitable strategy is employed to achieve an optimal compression ratio. We compared our AFC algorithm with four state-of-the-art compression algorithms, namely Gorilla, FPC, TSXor, and Chimp, as well as various general-purpose compression algorithms such as LZ4 and Snappy. Experimental results demonstrate that our algorithm achieves an improvement of at least 20% in compression ratio and even up to 100% on certain datasets.

Refactoring BZIP2 on the new‐generation sunway supercomputer

AbstractHigh‐performance computing is progressively assuming a fundamental role in advancing scientific research and engineering domains. However, the ever‐expanding scales of scientific simulations pose challenges for efficient data I/O and storage. The data compression technology has garnered significant attention as a solution to reduce data transmission and storage costs while enhancing performance. In particular, the BZIP2 lossless compression algorithm has been widely used due to its exceptional compression ratio, moderate compression speed, high reliability, and open‐source nature. This paper focuses on the design and realization of a parallelized BZIP2 algorithm tailored for deployment on the New‐Generation Sunway supercomputing platform. By leveraging the unique cache patterns of the New‐Generation Sunway processor, we propose the highly tuned multi‐threading and multi‐node implementations of the BZIP2 applications for different scenarios. Moreover, we also propose the efficient BZIP2 libraries based on the management processing element and computing processing element which support the commonly used high‐level (de)compression interfaces. The test results indicate that the our multi‐threading implementation achieves maximum speedup of 23.09 (8.57) in decompression(compression) compared to the sequential implementation. Furthermore, the multi‐node implementation achieves 50.81% (26.35%) parallel efficiency and peak performance of 16.6 GB/s (52.8 GB/s) for compression(decompression) when scaling up to 2048 processes.

Проєктування апаратного декомпресора

The common lossless compression algorithms were analyzed, and the LZW algorithm was selected for the hardware implementation. To express parallelism, this algorithm is represented as a cyclo-dynamic dataflow (CDDF). A hardware synthesis method for designing pipelined datapath is proposed, which optimizes CDDF considering the features of the FPGA primitives and maps it to hardware using VHDL language description. Using this method, an LZW de¬compressor is developed, which exhibits a high performance-to-hardware cost ratio. The de¬com¬¬¬pressor can be utilized in communication channels and other application-specific systems for data loading from memory, generating graphical stencils, and more.

TERSE/PROLIX (TRPX) - a new algorithm for fast and lossless compression and decompression of diffraction and cryo-EM data.

High-throughput data collection in crystallography poses significant challenges in handling massive amounts of data. Here, TERSE/PROLIX (or TRPX for short) is presented, a novel lossless compression algorithm specifically designed for diffraction data. The algorithm is compared with established lossless compression algorithms implemented in gzip, bzip2, CBF (crystallographic binary file), Zstandard(zstd), LZ4 and HDF5 with gzip, LZF and bitshuffle+LZ4 filters, in terms of compression efficiency and speed, using continuous-rotation electron diffraction data of an inorganic compound and raw cryo-EM data. The results show that TRPX significantly outperforms all these algorithms in terms of speed and compression rate. It was 60 times faster than bzip2 (which achieved a similar compression rate), and more than 3 times faster than LZ4, which was the runner-up in terms of speed, but had a much worse compression rate. TRPX files are byte-order independent and upon compilation the algorithm occupies very little memory. It can therefore be readily implemented in hardware. By providing a tailored solution for diffraction and raw cryo-EM data, TRPX facilitates more efficient data analysis and interpretation while mitigating storage and transmission concerns. The C++20 compression/decompression code, custom TIFF library and an ImageJ/Fiji Java plugin for reading TRPX files are open-sourced on GitHub under the permissive MIT license.

Concurrent encryption and lossless compression using inversion ranks

For secure and efficient transmission or storage, data files are commonly compressed and encrypted. In this work, we introduce a cost-effective encryption method of files as a built-in component of a lossless compression algorithm, thus avoiding the added cost of employing two separate processes. We have shown in earlier studies that preprocessing data with Burrows–Wheeler Transformation followed by Inversion Ranking transformation in advance of the utilization of an entropy coder resulted in an extremely effective general-purpose lossless compression technique. In our proposed algorithm, we isolate the inversion frequency vector obtained from the Inversion Ranking transformation and transmit the compressed file. Since the inversion frequency vector is necessary for decompression, we use it as our secret key and transmit it over a secure channel. Compared to the size of a data file, its frequency vector is rather small. Thus, encrypting only the inversion frequency vector, instead of the entire compressed file, results in substantial savings in computational cost. We demonstrate that the proposed technique is effective and resistant to common attacks on image and audio data sets.

EASC: An exception-aware semantic compression framework for real-world knowledge graphs

Knowledge graphs (KGs) have achieved great success in many real applications, and great efforts have been dedicated to constructing larger knowledge graphs. An obvious trend in KG construction is that the KGs become ever-increasingly bigger. However, we argue that constructing a KG by directly inserting more triples may harm the performance of the KG, and one possible solution is KG compression. In this paper, we propose an exception-aware semantic lossless compression framework EASC to compress a KG. Since many triples can be inferred from other triples with semantic rules, we remove the triples that can be inferred and store the rules and exception cases. Specifically, we formalize the lossless compression problem as a weighted set cover problem, which is NP-hard, and propose a semantic lossless compression algorithm to get an approximation result. We conduct extensive experiments on seven real-world large-scale KGs. The results show that EASC achieves state-of-the-art performance in semantic compression methods. Furthermore, by combining EASC as an independent module with syntactic compression methods, we achieve state-of-the-art performance in lossless compression methods.

Non-Markov models of single-molecule dynamics from information-theoretical analysis of trajectories.

Whether single-molecule trajectories, observed experimentally or in molecular simulations, can be described using simple models such as biased diffusion is a subject of considerable debate. Memory effects and anomalous diffusion have been reported in a number of studies, but directly inferring such effects from trajectories, especially given limited temporal and/or spatial resolution, has been a challenge. Recently, we proposed that this can be achieved with information-theoretical analysis of trajectories, which is based on the general observation that non-Markov effects make trajectories more predictable and, thus, more "compressible" by lossless compression algorithms. Toy models where discrete molecular states evolve in time were shown to be amenable to such analysis, but its application to continuous trajectories presents a challenge: the trajectories need to be digitized first, and digitization itself introduces non-Markov effects that depend on the specifics of how trajectories are sampled. Here we develop a milestoning-based method for information-theoretical analysis of continuous trajectories and show its utility in application to Markov and non-Markov models and to trajectories obtained from molecular simulations.

Lossless CFA image compression algorithm for wireless capsule endoscopy

Lossless CFA image compression algorithm for wireless capsule endoscopy

A lossless image compression and encryption algorithm combining JPEG-LS, neural network and hyperchaotic system

A lossless image compression and encryption algorithm combining JPEG-LS, neural network and hyperchaotic system

Common lossless compression algorithms and their error resiliency performance

With the development of communication technology and computer technology, many related industries, such as multimedia entertainment, put forward higher requirements for storing and transmitting information data. The research of data compression technology has attracted more and more attention. Therefore, the error resiliency ability of data compression algorithm is particularly important. How to enhance the error resiliency of data compression communication systems has been a hot topic for researchers. This paper mainly introduces the lossless data compression technology and its basic principle and performance index. Two typical lossless compression codes, Huffman and Arithmetic coding are deeply studied, including the principle of coding and the problem of error resiliency. Huffman coding and Arithmetic coding are two very important lossless compression codes widely used. The ability to resist channel error is an important index for data compression in communication. It is of great significance to further improve the channel adaptability of data compression to study the above two kinds of codes and their ability to resist channel error.

Almost lossless compression of noisy images

Abstract An almost lossless compression method for images is introduced adapted to the temporal noise of image sensors. In a first step, a non-linear gray value transform is applied to generate an image with a gray value independent temporal noise and less bits than the original image. The chosen value for the standard deviation of the temporal noise in the transformed image determines how accurately mean values and the standard deviation of temporal noise can be computed and to which extent the image can be compressed further by a lossless compression in a second step. Just a measurement of the noise characteristics according to the open and international EMVA standard 1288, a non-linear gray value transform for noise equalization, and an open source lossless compression algorithm are required to use this new compression method.

Image lossless compression algorithm optimization and FPGA implementation

In this thesis, a minimum redundant prefix coding with higher compression ratio and lower time complexity is proposed for lossless compression of HD images. The compression algorithm is based on Canonical Huffman coding, preprocesses the data source to be compressed according to the image data features, and then compresses the data in batches using the locally uneven features in the data, which improves the compression ratio by 1.678 times compared with the traditional canonical Huffman coding. During the implementation of the algorithm, the counting sorting method with lower time complexity and the code-length table construction method without relying on binary trees are used to reduce the complexity of the algorithm and achieve the purpose of real-time data processing by the system. Finally, the proposed compression algorithm is deployed in FPGA to improve the encoding rate by parallel hardware circuit and pipeline design.

Optimization of Lossless Compression Algorithms using Multithreading

The process of reducing the number of bits required to characterize data is referred to as compression. The advantages of compression include a reduction in the time taken to transfer data from one point to another, and a reduction in the cost required for the storage space and network bandwidth. There are two types of compression algorithms namely lossy compression algorithm and lossless compression algorithm. Lossy algorithms find utility in compressing audio and video signals whereas lossless algorithms are used in compressing text messages. The advent of the internet and its worldwide usage has not only raised the utility but also the storage of text, audio and video files. These multimedia files demand more storage space as compared to traditional files. This has given rise to the requirement for an efficient compression algorithm. There is a considerable improvement in the computing performance of the machines due to the advent of the multi-core processor. However, this multi-core architecture is not used by compression algorithms. This paper shows the implementation of lossless compression algorithms namely the Lempel-Ziv-Markov Algorithm, BZip2 and ZLIB algorithms using the concept of multithreading. The results obtained prove that the ZLIB algorithm proves to be more efficient in terms of the time taken to compress and decompress the text. The comparison is done for both compressions without multithreading and compression with multi-threading.