Compression Algorithm Research Articles

Parallel Lossless Compression of Raw Bayer Images on FPGA-Based High-Speed Camera

Digital image compression is applied to reduce camera bandwidth and storage requirements, but real-time lossless compression on a high-speed high-resolution camera is a challenging task. The article presents hardware implementation of a Bayer colour filter array lossless image compression algorithm on an FPGA-based camera. The compression algorithm reduces colour and spatial redundancy and employs Golomb–Rice entropy coding. A rule limiting the maximum code length is introduced for the edge cases. The proposed algorithm is based on integer operators for efficient hardware implementation. The algorithm is first verified as a C++ model and later implemented on AMD-Xilinx Zynq UltraScale+ device using VHDL. An effective tree-like pipeline structure is proposed to concatenate codes of compressed pixel data to generate a bitstream representing data of 16 parallel pixels. The proposed parallel compression achieves up to 56% reduction in image size for high-resolution images. Pipelined implementation without any state machine ensures operating frequencies up to 320 MHz. Parallelised operation on 16 pixels effectively increases data throughput to 40 Gbit/s while keeping the total memory requirements low due to real-time processing.

Compact lossy compression of tensors via neural tensor-train decomposition

AbstractMany real-world datasets are represented as tensors, i.e., multi-dimensional arrays of numerical values. Storing them without compression often requires substantial space, which grows exponentially with the order. While many tensor compression algorithms are available, many of them rely on strong data assumptions regarding its order, sparsity, rank, and smoothness. In this work, we propose TensorCodec, a lossy compression algorithm for general tensors that do not necessarily adhere to strong input data assumptions.TensorCodec incorporates three key ideas. The first idea is neural tensor-train decomposition (NTTD) where we integrate a recurrent neural network into Tensor-Train Decomposition to enhance its expressive power and alleviate the limitations imposed by the low-rank assumption. Another idea is to fold the input tensor into a higher-order tensor to reduce the space required by NTTD. Finally, the mode indices of the input tensor are reordered to reveal patterns that can be exploited by NTTD for improved approximation. In addition, we extend TensorCodec to enable the lossy compression of tensors with missing entries, often found in real-world datasets. Our analysis and experiments on 8 real-world datasets demonstrate that TensorCodec is (a) Concise: it gives up to $$7.38 \times $$ 7.38 × more compact compression than the best competitor with similar reconstruction error, (b) Accurate: given the same budget for compressed size, it yields up to $$3.33\times $$ 3.33 × more accurate reconstruction than the best competitor, (c) Scalable: Its empirical compression time is linear in the number of tensor entries, and it reconstructs each entry in logarithmic time. Our code and datasets are available at https://github.com/kbrother/TensorCodec.

Compression rates of microbial genomes are associated with genome size and base composition

BackgroundTo what degree a string of symbols can be compressed reveals important details about its complexity. For instance, strings that are not compressible are random and carry a low information potential while the opposite is true for highly compressible strings. We explore to what extent microbial genomes are amenable to compression as they vary considerably both with respect to size and base composition. For instance, microbial genome sizes vary from less than 100,000 base pairs in symbionts to more than 10 million in soil-dwellers. Genomic base composition, often summarized as genomic AT or GC content due to the similar frequencies of adenine and thymine on one hand and cytosine and guanine on the other, also vary substantially; the most extreme microbes can have genomes with AT content below 25% or above 85% AT. Base composition determines the frequency of DNA words, consisting of multiple nucleotides or oligonucleotides, and may therefore also influence compressibility. Using 4,713 RefSeq genomes, we examined the association between compressibility, using both a DNA based- (MBGC) and a general purpose (ZPAQ) compression algorithm, and genome size, AT content as well as genomic oligonucleotide usage variance (OUV) using generalized additive models.ResultsWe find that genome size (p < 0.001) and OUV (p < 0.001) are both strongly associated with genome redundancy for both type of file compressors. The DNA-based MBGC compressor managed to improve compression with approximately 3% on average with respect to ZPAQ. Moreover, MBGC detected a significant (p < 0.001) compression ratio difference between AT poor and AT rich genomes which was not detected with ZPAQ.ConclusionAs lack of compressibility is equivalent to randomness, our findings suggest that smaller and AT rich genomes may have accumulated more random mutations on average than larger and AT poor genomes which, in turn, were significantly more redundant. Moreover, we find that OUV is a strong proxy for genome compressibility in microbial genomes. The ZPAQ compressor was found to agree with the MBGC compressor, albeit with a poorer performance, except for the compressibility of AT-rich and AT-poor/GC-rich genomes.

High-Quality Image Compression Algorithm Design Based on Unsupervised Learning.

Increasingly massive image data is restricted by conditions such as information transmission and reconstruction, and it is increasingly difficult to meet the requirements of speed and integrity in the information age. To solve the urgent problems faced by massive image data in information transmission, this paper proposes a high-quality image compression algorithm based on unsupervised learning. Among them, a content-weighted autoencoder network is proposed to achieve image compression coding on the basis of a smaller bit rate to solve the entropy rate optimization problem. Binary quantizers are used for coding quantization, and importance maps are used to achieve better bit allocation. The compression rate is further controlled and optimized. A multi-scale discriminator suitable for the generative adversarial network image compression framework is designed to solve the problem that the generated compressed image is prone to blurring and distortion. Finally, through training with different weights, the distortion of each scale is minimized, so that the image compression can achieve a higher quality compression and reconstruction effect. The experimental results show that the algorithm model can save the details of the image and greatly compress the memory of the image. Its advantage is that it can expand and compress a large number of images quickly and efficiently and realize the efficient processing of image compression.

A Bridge-based Algorithm for Simultaneous Primal and Dual Defects Compression on Topologically Quantum-error-corrected Circuits

Topological quantum error correction (TQEC) using the surface code is among the most promising techniques for fault-tolerant quantum circuits. The required resource of a TQEC circuit can be modeled as a space-time volume of a three-dimensional diagram by describing the defect movement along the time axis. For large-scale complex problems, it is crucial to minimize the space-time volume for a quantum algorithm with a reasonable physical qubit number and computation time. Previous work proposed an automated tool for bridge compression on a large-scale TQEC circuit. However, the existing automated bridge compression is only for dual defects and not for primal defects. This paper presents an algorithm to simultaneously perform bridge compression on primal and dual defects. In addition, the automatic compression algorithm performs initialization/measurement simplification and flipping to improve the compression. Compared with the state-of-the-art work, experimental results show that our proposed algorithm can averagely reduce space-time volumes by 53%.

Performance Analysis of Compression Algorithms on Matrix Data: Data Transfer Optimization in Microservices Architectures

With the rapid proliferation of microservices architectures these days, the efficient and fast transfer of large matrix data between services has become a significant challenge. This study presents an analysis aimed at finding solutions to this challenge. The analysis addresses the compression and decompression of large matrix data, focusing on lossless compression algorithms to optimize data transfer without data loss. The study is implemented on an example scenario. This scenario is taken from a project with a microservice architecture. In the example scenario, an image processing service developed in Python programming language generates 640x480 matrix data. After going through a compression algorithm, this data is periodically transferred to a backend service developed in C# programming language. This data is then stored in a database. In the final stage, decompression operations are performed so that this data can be used for reporting. The performance of various compression algorithms in the data compression, database storage and report generation stages is extensively tested. Within the scope of the study, tests were performed using five different compression algorithms (Gzip, Zlib, Deflate, Brotli and Bz2). The results are obtained through performance tests aimed at determining the most optimized end-to-end solution. Analyzing the performance of the compression algorithms on the example scenario, the Brotli algorithm gives the most optimal result in terms of both speed and compression size. This work makes an important contribution to data transfer optimization in microservice architectures and provides a reference for research in this area by presenting the performance analysis of various compression algorithms.

Deblurring image compression algorithm using deep convolutional neural network

Deblurring image compression algorithm using deep convolutional neural network

Fast interactive simulations of cardiac electrical activity in anatomically accurate heart structures by compressing sparse uniform cartesian grids

Background and Objective:Numerical simulations are valuable tools for studying cardiac arrhythmias. Not only do they complement experimental studies, but there is also an increasing expectation for their use in clinical applications to guide patient-specific procedures. However, numerical studies that solve the reaction–diffusion equations describing cardiac electrical activity remain challenging to set up, are time-consuming, and in many cases, are prohibitively computationally expensive for long studies. The computational cost of cardiac simulations of complex models on anatomically accurate structures necessitates parallel computing. Graphics processing units (GPUs), which have thousands of cores, have been introduced as a viable technology for carrying out fast cardiac simulations, sometimes including real-time interactivity. Our main objective is to increase the performance and accuracy of such GPU implementations while conserving computational resources. Methods:In this work, we present a compression algorithm that can be used to conserve GPU memory and improve efficiency by managing the sparsity that is inherent in using Cartesian grids to represent cardiac structures directly obtained from high-resolution MRI and mCT scans. Furthermore, we present a discretization scheme that includes the cross-diagonal terms in the computational cell to increase numerical accuracy, which is especially important for simulating thin tissue sections without the need for costly mesh refinement. Results:Interactive WebGL simulations of atrial/ventricular structures (on PCs, laptops, tablets, and phones) demonstrate the algorithm’s ability to reduce memory demand by an order of magnitude and achieve calculations up to 20x faster. We further showcase its superiority in slender tissues and validate results against experiments performed in live explanted human hearts. Conclusions:In this work, we present a compression algorithm that accelerates electrical activity simulations on realistic anatomies by an order of magnitude (up to 20x), thereby allowing the use of finer grid resolutions while conserving GPU memory. Additionally, improved accuracy is achieved through cross-diagonal terms, which are essential for thin tissues, often found in heart structures such as pectinate muscles and trabeculae, as well as Purkinje fibers. Our method enables interactive simulations with even interactive domain boundary manipulation (unlike finite element/volume methods). Finally, agreement with experiments and ease of mesh import into WebGL paves the way for virtual cohorts and digital twins, aiding arrhythmia analysis and personalized therapies.

A New Algorithm for Lossless Data Compression Based of Time Clock Mechanism

A New Algorithm for Lossless Data Compression Based of Time Clock Mechanism

Improved JPEG Lossless Compression for Compression of Intermediate Layers in Neural Networks Based on Compute-In-Memory

With the development of Convolutional Neural Networks (CNNs), there is a growing requirement for their deployment on edge devices. At the same time, Compute-In-Memory (CIM) technology has gained significant attention in edge CNN applications due to its ability to minimize data movement between memory and computing units. However, the deployment of complex deep neural network models on edge devices with restricted hardware resources continues to be challenged by a lack of adequate storage for intermediate layer data. In this article, we propose an optimized JPEG Lossless Compression (JPEG-LS) algorithm that implements serial context parameter updating alongside parallel encoding. This method is designed for the global prediction and efficient compression of intermediate data layers in neural networks employing CIM techniques. The results indicate average compression ratios of 6.44× for VGG16, 3.62× for ResNet34, 1.67× for MobileNetV2, and 2.31× for InceptionV3. Moreover, the implementation achieves a data throughput of 32 bits per cycle at 600 MHz on the TSMC 28 nm, with a hardware cost of 122 K Gate Count.

Improvement of Lossless Text Compression Methods using a Hybrid Method by the Integration of RLE, LZW and Huffman Coding Algorithms

In the field of computer science, data compression is essential in the process of data transfer because it reduces file size without losing information. Issues that depend on the characteristics of the text can greatly affect the effectiveness of any compression algorithm. Consequently, each traditional compression algorithm has its own strengths and limitations. Therefore, a highly efficient compression method that achieves higher compression ratio is needed. In this paper, based on the sequential implementation of Huffman, RLE and LZW algorithms, a hybrid data compression method has been proposed. The algorithms have been individually evaluated and compared in terms of compression ratio and compressed file size. We conclude that in terms of compression ratio and compressed file size, LZW is better than Huffman and RLE. The results show that the proposed hybrid algorithm achieves the highest average compression ratio is 2.42 and the lowest average compressed file size is 1138.5 or less compared to the individual implementations of the algorithms. Thus, it can be concluded that the proposed hybrid method enhances lossless text compression in terms of compression ratio and file size according to compression metrics.

Two-dimensional hyperchaos-based encryption and compression algorithm for agricultural UAV-captured planar images.

This study presents an approach that integrates compressed sensing technology with two-dimensional hyperchaotic coupled Fourier oscillator systems (2D-HCFOS) to address the challenge of slow encryption speeds in agricultural unmanned aerial vehicles (UAVs). The primary challenge in enhancing encryption speed lies in the limited capacity inherent in traditional chaotic-based systems and the computational complexity of their processes. The 2D-HCFOS utilizes a complex two-dimensional hybrid chaotic system, which significantly enhances the security of agricultural UAV image data. Notably, the image encryption process is performed on a personal computer connected to the drone, ensuring efficient processing. By integrating advanced Fourier series and nonlinear coupled oscillators, the model surpasses existing chaotic-based methods, improving both the pseudo-randomness and robustness of encryption. Additionally, incorporating Bonouille functions into the discrete cosine transform (DCT) domain results in a sparser measurement matrix, which is essential for efficient encryption on personal computers. The effectiveness of 2D-HCFOS in securely encrypting agricultural drone images has been rigorously validated through simulations and analytical evaluations using sophisticated row, rotation, and matrix encryption techniques. The improved security performance is further verified by comparative analysis. Compared with other models, the Lyapunov index of 2D-HCFOS is 15.1039, and the sample entropy is 2.4987, indicating that it possesses superior chaotic performance and encryption reliability.

A color image compression and encryption algorithm combining compressed sensing, Sudoku matrix, and hyperchaotic map

A color image compression and encryption algorithm combining compressed sensing, Sudoku matrix, and hyperchaotic map

Lossy Image Compression with Stochastic Quantization

Introduction. Lossy image compression algorithms play a crucial role in various domains, including graphics, and image processing. As image information density increases, so do the resources required for processing and transmission. One of the most prominent approaches to address this challenge is color quantization, proposed by Orchard et al. (1991). This technique optimally maps each pixel of an image to a color from a limited palette, maintaining image resolution while significantly reducing information content. Color quantization can be interpreted as a clustering problem (Krishna et al. (1997), Wan (2019)), where image pixels are represented in a three-dimensional space, with each axis corresponding to the intensity of an RGB channel. The purpose of the paper. Scaling of traditional algorithms like K-Means can be challenging for large data, such as modern images with millions of colors. This paper reframes color quantization as a three-dimensional stochastic transportation problem between the set of image pixels and an optimal color palette, where the number of colors is a predefined hyperparameter. We employ Stochastic Quantization (SQ) with a seeding technique proposed by Arthur et al. (2007) to enhance the scalability of color quantization. This method introduces a probabilistic element to the quantization process, potentially improving efficiency and adaptability to diverse image characteristics. Results. To demonstrate the efficiency of our approach, we present experimental results using images from the ImageNet dataset. These experiments illustrate the performance of our Stochastic Quantization method in terms of compression quality, computational efficiency, and scalability compared to traditional color quantization techniques. Conclusions. This study introduces a scalable algorithm for solving the color quantization problem without memory constraints, demonstrating its efficiency on a subset of images from the ImageNet dataset. The convergence speed of the algorithm can be further enhanced by modifying the update rule with alternative methods to Stochastic Gradient Descent (SGD) that incorporate adaptive learning rates. Moreover, the stochastic nature of the proposed solution enables the utilization of parallelization techniques to simultaneously update the positions of multiple quants, potentially leading to significant performance improvements. This aspect of parallelization and its impact on algorithm efficiency presents a topic for future research. The proposed method not only addresses the limitations of existing color quantization techniques but also opens up new possibilities for optimizing image compression algorithms in resource-constrained environments. Keywords: non-convex optimization, stochastic optimization, stochastic quantization, color quantization, lossy compression.

A Comparative Study of Text-Based Lossless Compression

Lossless data compression is a critical technique used to reduce file sizes without any loss of information during the encoding and decoding processes. This study presents a comparative analysis of two widely-used lossless compression algorithms: Huffman Encoding and Lempel-Ziv-Welch (LZW). The primary objective is to evaluate the performance of these algorithms in terms of compression ratio, compression time, decompression time, and space savings. The analysis was conducted on 100 files of varying sizes. The results demonstrate that the LZW algorithm outperforms Huffman Encoding, offering superior compression ratios, faster compression and decompression times, and greater disk space savings. These findings highlight the effectiveness of LZW for efficient data compression in practical applications.

Assembly Theory is an approximation to algorithmic complexity based on LZ compression that does not explain selection or evolution

We formally prove the equivalence between Assembly Theory (AT) and Shannon Entropy via a method based upon the principles of statistical compression that belongs to the LZ family of popular compression algorithms. Such popular lossless compression algorithms behind file formats such as ZIP and PNG have been shown to empirically reproduce the results that AT considers its cornerstone. The same results have also been reported before AT in successful application of other complexity measures in the areas covered by AT such as separating organic from non-organic molecules and in the context of the study of selection and evolution. We demonstrate that the assembly index is equivalent to the size of a minimal context-free grammar. The statistical compressibility of such a method is bounded by Shannon Entropy and other equivalent traditional LZ compression schemes, such as LZ77 and LZW. We also demonstrate that AT, and the algorithms supporting its pathway complexity, assembly index, and assembly number, define compression schemes and methods that are subsumed into algorithmic information theory. We conclude that the assembly index and the assembly number do not lead to an explanation or quantification of biases in generative (physical or biological) processes, including those brought about by (abiotic or biotic) selection and evolution, that could not have been arrived at using Shannon Entropy, or that have not been already reported before using classical information theory or algorithmic complexity.

Compressed Point Cloud Quality Index by Combining Global Appearance and Local Details

In recent years, many standardized algorithms for point cloud compression (PCC) has been developed and achieved remarkable compression ratios. To provide guidance for rate-distortion optimization and codec evaluation, point cloud quality assessment (PCQA) has become a critical problem for PCC. Therefore, in order to achieve a more consistent correlation with human visual perception of a compressed point cloud, we propose a full-reference PCQA algorithm tailored for static point clouds in this article, which can jointly measure geometry and attribute deformations. Specifically, we assume that the quality decision of compressed point clouds is determined by both global appearance (e.g., density, contrast, and complexity) and local details (e.g., gradient and hole). Motivated by the nature of compression distortions and the properties of the human visual system, we derive perceptually effective features for the above two categories, such as content complexity, luminance/geometry gradient, and hole probability. Through systematically incorporating measurements of variations in the local and global characteristics, we derive an effective quality index for the input compressed point clouds. Extensive experiments and analyses conducted on popular PCQA databases show the superiority of the proposed method in evaluating compression distortions. Subsequent investigations validate the efficacy of different components within the model design.

Multilevel approximation of Gaussian random fields: Covariance compression, estimation, and spatial prediction

The distribution of centered Gaussian random fields (GRFs) indexed by compacta such as smooth, bounded Euclidean domains or smooth, compact and orientable manifolds is determined by their covariance operators. We consider centered GRFs given as variational solutions to coloring operator equations driven by spatial white noise, with an elliptic self-adjoint pseudodifferential coloring operator from the Hörmander class. This includes the Matérn class of GRFs as a special case. Using biorthogonal multiresolution analyses on the manifold, we prove that the precision and covariance operators, respectively, may be identified with bi-infinite matrices and finite sections may be diagonally preconditioned rendering the condition number independent of the dimension p of this section. We prove that a tapering strategy by thresholding applied on finite sections of the bi-infinite precision and covariance matrices results in optimally numerically sparse approximations. That is, asymptotically only linearly many nonzero matrix entries are sufficient to approximate the original section of the bi-infinite covariance or precision matrix using this tapering strategy to arbitrary precision. The locations of these nonzero matrix entries can be determined a priori. The tapered covariance or precision matrices may also be optimally diagonally preconditioned. Analysis of the relative size of the entries of the tapered covariance matrices motivates novel, multilevel Monte Carlo (MLMC) oracles for covariance estimation, in sample complexity that scales log-linearly with respect to the number p of parameters. In addition, we propose and analyze novel compressive algorithms for simulating and kriging of GRFs. The complexity (work and memory vs. accuracy) of these three algorithms scales near-optimally in terms of the number of parameters p of the sample-wise approximation of the GRF in Sobolev scales.

A method for compressing AIS trajectory based on the adaptive core threshold difference Douglas–Peucker algorithm

Traditional trajectory compression algorithms, such as the siliding window (SW) algorithm and the Douglas–Peucker (DP) algorithm, typically use static thresholds based on fixed parameters like ship dimensions or predetermined distances, which limits their adaptive capabilities. In this paper, the adaptive core threshold difference-DP (ACTD-DP) algorithm is proposed based on traditional DP algorithm. Firstly, according to the course value of automatic identification system (AIS) data, the original trajectory data is preprocessed and some redundant points are discarded. Then the number of compressed trajectory points corresponding to different thresholds is quantified. The function relationship between them is established by curve fitting method. The characteristics of the function curve are analyzed, and the core threshold and core threshold difference are solved. Finally, the compression factor is introduced to determine the optimal core threshold difference, which is the key parameter to control the accuracy and efficiency of the algorithm. Five different algorithms are used to compress the all ship trajectories in the experimental water area. The average compression ratio (ACR) of the ACTD-DP algorithm is 87.53%, the average length loss ratio (ALLR) is 23.20%, the AMSED (mean synchronous Euclidean distance of all trajectories) is 68.9747 mx, and the TIME is 25.6869 s. Compared with the other four algorithms, the ACTD-DP algorithm shows that the algorithm can not only achieve high compression ratio, but also maintain the integrity of trajectory shape. At the same time, the compression results of four different trajectories show that ACTD-DP algorithm has good robustness and applicability. Therefore, ACTD-DP algorithm has the best compression effect.

Massive Compression for High Data Rate Macromolecular Crystallography (HDRMX): Impact on Diffraction Data and Subsequent Structural Analysis.

New higher-count-rate, integrating, large area X-ray detectors with framing rates as high as 17,400 images per second are beginning to be available. These will soon be used for specialized MX experiments but will require optimal lossy compression algorithms to enable systems to keep up with data throughput. Some information may be lost. Can we minimize this loss with acceptable impact on structural information?