Lossless Research Articles

Quantization Step Estimation for JPEG Image Forensics

In image forensics, quantization step estimation plays a crucial role in revealing the JPEG compression history. This study focuses on the estimation for images that have been JPEG compressed and re-saved in lossless formats. Several effective methods have been developed, whereas the performance on small sized images still needs to be improved. In this paper, a novel JPEG quantization step estimation method with low complexity and high efficiency is proposed. First, for a given decompressed image, as the special shape of its DCT coefficients distribution is highly related to the quantization step, a function of the candidate step taking a similar shape is designed. Then, the quantization step is determined as the candidate leading to the maximum response of the designed function on the probability density function of DCT coefficients. The relation between the maximum response and the quantization step is verified mathematically, which provides a theoretical basis for the proposed method. In addition, to further improve the estimation performance, two fine adjustment procedures are adopted. Experimental results demonstrate that the proposed method outperforms some state-of-the-art works, especially on small sized images.

Context-Tree-Based Lossy Compression and Its Application to CSI Representation

We propose novel compression algorithms for time-varying channel state information (CSI) in wireless communications. The proposed scheme combines (lossy) vector quantisation and (lossless) compression. First, the new vector quantisation technique is based on a class of parametrised companders applied on each component of the normalised CSI vector. Our algorithm chooses a suitable compander in an intuitively simple way whenever empirical data are available. Then, the sequences of quantisation indices are compressed using a context-tree-based approach. Essentially, we update the estimate of the conditional distribution of the source at each instant and encode the current symbol with the estimated distribution. The algorithms have low complexity, are linear-time in both the spatial dimension and time duration, and can be implemented in an online fashion. We run simulations to demonstrate the effectiveness of the proposed algorithms in such scenarios.

Exploring Data Analytics Without Decompression on Embedded GPU Systems

With the development of computer architecture, even for embedded systems, GPU devices can be integrated, providing outstanding performance and energy efficiency to meet the requirements of different industries, applications, and deployment environments. Data analytics is an important application scenario for embedded systems. Unfortunately, due to the limitation of the capacity of the embedded device, the scale of problems handled by the embedded system is limited. In this paper, we propose a novel data analytics method, called G-TADOC, for efficient text analytics directly on compression on embedded GPU systems. A large amount of data can be compressed and stored in embedded systems, and can be processed directly in the compressed state, which greatly enhances the processing capabilities of the systems. Particularly, G-TADOC has three innovations. First, a novel fine-grained thread-level workload scheduling strategy for GPU threads has been developed, which partitions heavily-dependent loads adaptively in a fine-grained manner. Second, a GPU thread-safe memory pool has been developed to handle inconsistency with low synchronization overheads. Third, a sequence-support strategy is provided to maintain high GPU parallelism while ensuring sequence information for lossless compression. Moreover, G-TADOC involves special optimizations for embedded GPUs, such as utilizing the CPU-GPU shared unified memory. Experiments show that G-TADOC provides 13.2× average speedup compared to the state-of-the-art TADOC. G-TADOC also improves performance-per-cost by 2.6× and energy efficiency by 32.5× over TADOC.

A High Throughput and Energy Efficient Lepton Hardware Encoder With Hash-Based Memory Optimization

Although it has been surpassed by many subsequent coding standards, JPEG occupies a large storage share of the current data hosting service. To reduce the storage costs, DropBox proposed a secondary lossless compression algorithm, Lepton, to further improve the compression rate of JPEG images. However, the bloated probability models defined by Lepton severely restrict its throughput and energy efficiency. To solve this problem, we construct an access probability-based hash function for the probability models, and then propose a hardware-friendly memory optimization method by combining the proposed hash function with N-way Set-Associative unit. Besides, we also propose a synchronization mechanism for the serial accessing probability model, so that the syntax elements can be processed in parallel without changing the resulting bitstream. After that, we implement a high throughput, high energy efficiency, and low-cost Lepton hardware encoder. To the best of our knowledge, this is the first hardware implementation of transparent image recompression. The synthesis result shows that the proposed hardware structure reduces the total area of the probability models by 70.97%. Compared with DropBox’s software solution, the throughput and the energy efficiency of the proposed Lepton hardware encoder are increased by 63 and 1398 times on average. In terms of manufacturing cost, the proposed Lepton hardware encoder is also lower than the general-purpose CPU used by DropBox.

Electrocardiogram signal compression using tunable-Q wavelet transform and meta-heuristic optimization techniques

Electrocardiogram (ECG) signals are the biomedical signals commonly used in the prognosis of cardiovascular diseases. ECG recordings need to be stored and transferred when telemedicine-based healthcare systems are required. These data are stored in a digitized format at higher bits per sample that requires ample space for storage. Therefore, this motivated us to develop efficient compression methodologies for ECG signals. In this regard, this work proposes compression techniques using the optimized tunable-Q wavelet transform (TQWT). For this purpose, TQWT parameters are optimized using several meta-heuristic optimization algorithms such as variants of PSO, ABC and its hybrid with PSO, GWO and its hybrid with PSO, and Sparse PSO. These hybrid methods and Sparse PSO have been utilized for the first time to optimize TQWT. Subsequently, thresholding and quantization are performed by using a dead-zone quantizer (DZQ). The quantized coefficients are encoded by utilizing a lossless compression technique run-length encoding (RLE). The proposed algorithms have been examined on the MIT-BIH arrhythmia database. It is clear from the results that significant compression has been achieved when compared to existing techniques. The performance of the proposed algorithms has been evaluated in terms of various evaluation parameters that are compression ratio (CR), percent-root-mean square difference (PRD), signal-to-noise ratio (SNR), and quality score (QS).

Real-time and post-hoc compression for data from Distributed Acoustic Sensing

Distributed Acoustic Sensing (DAS) is an emerging sensing technology that records the strain-rate along fiber optic cables at high spatial and temporal resolution. This technique is becoming a popular tool in seismology, hydrology, and other subsurface monitoring applications. However, due to the large coverage (10’s of km) and high density of measurements (1m spacing at 100’s of Hz), a DAS installation could produce terabytes of data records per day. Because many DAS instruments are deployed in remote locations, this large data size poses significant challenges to its transfer and storage. In this paper, we explore lossless compression methods to reduce the storage requirement in both real-time and post-hoc scenarios. We propose a two-stage compression method to improve the compression ratio and compression speed. This two-stage compression method could reduce the storage requirement by 40%, which is 20% more than other lossless methods, such as ZSTD. We demonstrate that the compression method could complete its operation well before the DAS instrument needs to output the next file, making it suitable for real-time DAS acquisition. We also implement a parallel compression method for a post-hoc scenario and demonstrate that our method could effectively utilize a parallel computer. With 256 CPU cores, our parallel compression method achieves the speed of 26GB/second.

Binary hologram compression using context based Bayesian tree models with adaptive spatial segmentation

With holographic displays requiring giga- or terapixel resolutions, data compression is of utmost importance in making holography a viable technique in the near future. In addition, since the first-generation of holographic displays is expected to require binary holograms, associated compression algorithms are expected to be able to handle this binary format. In this work, the suitability of a context based Bayesian tree model is proposed as an extension to adaptive binary arithmetic coding to facilitate the efficient lossless compression of binary holograms. In addition, we propose a quadtree-based adaptive spatial segmentation strategy, as the scale dependent, quasi-stationary behavior of a hologram limits the applicability of the advocated modelling approach straightforwardly on the full hologram. On average, the proposed compression strategy produces files that are around 12% smaller than JBIG2, the reference binary image codec.

Lossless Compression Algorithm for Medical Images With High Precision Based on Discrete Wavelet Transform

In image distortion and low adaptive recognition after medical image compression, a high precision medical image lossless compression algorithm based on discrete wavelet transform is proposed. A 3D imaging model of multi-dimensional medical images is constructed, and adaptive information enhancement and image restoration processing are performed on the collected medical images. According to the results of high-dimensional segmentation and segmentation, discrete wavelet transform is used to achieve high-precision lossless compression of medical images. The results show that the medical image compression is better non-destructive and the image fidelity is higher, which improves the detection and adaptive recognition ability of medical images

Precision requirements and data compression in CryoEM/CryoET

With larger, higher speed detectors and improved automation, individual CryoEM instruments are capable of producing a prodigious amount of data each day, which must then be stored, processed and archived. While it has become routine to use lossless compression on raw counting-mode movies, the averages which result after correcting these movies no longer compress well. These averages could be considered sufficient for long term archival, yet they are conventionally stored with 32 bits of precision, despite high noise levels. Derived images are similarly stored with excess precision, providing an opportunity to decrease project sizes and improve processing speed. We present a simple argument based on propagation of uncertainty for safe bit truncation of flat-fielded images combined with lossless compression. The same method can be used for most derived images throughout the processing pipeline. We test the proposed strategy on two standard, data-limited CryoEM data sets, demonstrating that these limits are safe for real-world use. We find that 5 bits of precision is sufficient for virtually any raw CryoEM data and that 8–12 bits is sufficient for intermediate averages or final 3-D structures. Additionally, we detail and recommend specific rules for discretization of data as well as a practical compressed data representation that is tuned to the specific needs of CryoEM.

Retraction Note to: Lossless medical image compression algorithm using tetrolet transformation

Retraction Note to: Lossless medical image compression algorithm using tetrolet transformation

Efficient JPEG Encoding Using Bernoulli Shift Map for Secure Communication

To ensure confidentiality and efficient network bandwidth, digital data must be compressed and encrypted. In most communication systems, these two factors are critical for information processing. Image compression and encryption may result in lower restoration quality and performance. Secure-JPEG is an effort to create a compression and encryption technique for digital data. This approach is based on the JPEG compression standard, which is the most extensively used lossy compression scheme. It enhances the usual JPEG compression algorithm to encrypt data during compression. The Secure-JPEG approach encrypts the data while it is compressed, and it may be easily modified to offer near lossless compression. Lossless compression, on the other hand, has a lower compression ratio and is only useful in certain situations. The paper addresses the issue of insufficient security as a result of the usage of a simple random number generator that is not cryptographically safe. The enhanced security characteristics are provided via the Generalized Bernoulli Shift Map, which has a chaotic system with proven security. Several cryptographic tests are used to validate the algorithm's security, and the chaotic system's behavior is also examined.

ACO:lossless quality score compression based on adaptive coding order

BackgroundWith the rapid development of high-throughput sequencing technology, the cost of whole genome sequencing drops rapidly, which leads to an exponential growth of genome data. How to efficiently compress the DNA data generated by large-scale genome projects has become an important factor restricting the further development of the DNA sequencing industry. Although the compression of DNA bases has achieved significant improvement in recent years, the compression of quality score is still challenging.ResultsIn this paper, by reinvestigating the inherent correlations between the quality score and the sequencing process, we propose a novel lossless quality score compressor based on adaptive coding order (ACO). The main objective of ACO is to traverse the quality score adaptively in the most correlative trajectory according to the sequencing process. By cooperating with the adaptive arithmetic coding and an improved in-context strategy, ACO achieves the state-of-the-art quality score compression performances with moderate complexity for the next-generation sequencing (NGS) data.ConclusionsThe competence enables ACO to serve as a candidate tool for quality score compression, ACO has been employed by AVS(Audio Video coding Standard Workgroup of China) and is freely available at https://github.com/Yoniming/ACO.

Efficient ROI-based compression of mammography images

Medical imaging and telemedicine are progressively employed on a large scale these days. The utilization of lossless compression techniques for medical image storage and transmission may not provide significant benefits in terms of data reduction. On the other hand, the utilization of lossy compression techniques may lead to loss of essential data, which affects the diagnosis process. The critical data in the region of interest (ROI) of a medical image should be compressed with high-quality compression with no data lost or destroyed, regardless of the remaining parts of the medical image (non-ROI). Generally, mammography is used for investigating breast malignancies and localization of small tumors. Therefore, to increase the compression efficiency of mammogram images without losing any essential data, we present a hybrid technique based on lossless compression for the critical data in the ROI and lossy compression for the remaining parts of the medical image (non-ROI). In this work, edge-directed prediction lossless compression is adopted for the ROI, while fractal lossy compression is proposed for the non-ROI of mammogram images. Mammogram images from the mammographic image analysis society (MIAS) database are used to test the proposed hybrid technique. Encoding time, decoded image quality in terms of peak signal-to-noise ratio (PSNR), and compression ratio are the metrics used to assess the quality of compression. The obtained results prove that the proposed technique achieves a high compression ratio, while maintaining an acceptable quality of the reconstructed images compared with other recent mammogram image compression techniques.

Improving matrix-vector multiplication via lossless grammar-compressed matrices

As nowadays Machine Learning (ML) techniques are generating huge data collections, the problem of how to efficiently engineer their storage and operations is becoming of paramount importance. In this article we propose a new lossless compression scheme for real-valued matrices which achieves efficient performance in terms of compression ratio and time for linear-algebra operations. Experiments show that, as a compressor, our tool is clearly superior to gzip and it is usually within 20% of xz in terms of compression ratio. In addition, our compressed format supports matrix-vector multiplications in time and space proportional to the size of the compressed representation, unlike gzip and xz that require the full decompression of the compressed matrix. To our knowledge our lossless compressor is the first one achieving time and space complexities which match the theoretical limit expressed by the k -th order statistical entropy of the input. To achieve further time/space reductions, we propose column-reordering algorithms hinging on a novel column-similarity score. Our experiments on various data sets of ML matrices show that our column reordering can yield a further reduction of up to 16% in the peak memory usage during matrix-vector multiplication. Finally, we compare our proposal against the state-of-the-art Compressed Linear Algebra (CLA) approach showing that ours runs always at least twice faster (in a multi-thread setting), and achieves better compressed space occupancy and peak memory usage. This experimentally confirms the provably effective theoretical bounds we show for our compressed-matrix approach.

Combining 3D run-length encoding coding and searching techniques for medical image compression

<p><span>The field of image compression became a mandatory tool to face the increasing and advancing production of medical images, besides the inevitable need for smaller size of medical images in telemedicine systems. In spite of its simplicity, run-length encoding (RLE) technique is a considerably effective and practical tool in the field of lossless image compression. Such that, it is widely recommended for 2D space that utilizes common searching techniques like linear and zigzag. This paper adopts a new algorithm taking advantage of the potential simplicity of the run-length algorithm to contribute a volumetric RLE approach for binary medical data in the 3D form. The proposed volumetric-RLE (VRLE) algorithm differs from the 2D RLE approach utilizing correlations of intra-slice only, which is used for compressing binary medical data utilizing voxel-correlations of inter-slice. Furthermore, several forms of scanning are used to extending proposed technique like Hilbert and Perimeter, which determines the best possible procedure of scanning suitable for data morphology considering the segmented organ. This work employs proposed algorithm on four image datasets to get as sufficient as possible evaluation. Experimental results and benchmarking illustrate that the performance of the proposed technique surpasses other state-of-the-art techniques with 1:30 enhancement on average.</span></p>

Reversible integer approximation of color space transforms for lossless compression of big color raster data

Reversible integer transforms are or great importance for lossless compression algorithms. To perform reversible decorrelation of color channels we propose an algorithm for calculating parameters of a reversible integer transform, which approximates such continuous mappings as a discrete Karunen-Loeve transform. We propose a method for estimating the approximation errors, which makes it possible to choose an optimal approximation of the original transform that minimizes these errors. Using the MRG file format, intended for storing large amounts of integer raster data, as an example, we show that after applying decorrelation, it is possible to increase the compression ratio of multichannel raster images using lossless compression algorithms.

75‐2: The Effect of Chromatic Aberration Correction on Visually Lossless Compression

In augmented and virtual reality (AR/VR), magnifying display lenses typically introduce pincushion distortion and transverse chromatic aberration. These distortions are corrected by pre‐processing, one side‐effect of which is disruption of the spatial correlation between color channels. As a result, the standard practice of performing color space conversion prior to image compression may introduce undesirable, visible artefacts. To assess this, we evaluated the performance of two low impairment display stream codecs on distortion corrected stereoscopic images based on converting to YCoCg color space or bypassing the color conversion.

Ontology-Driven Data Cleaning Towards Lossless Data Compression.

With the available data in healthcare, healthcare organizations and practitioners require interoperable, efficient, and non-time-consuming data exchange. Currently, several cases aim to the exchanged data security, without considering the complexity of the data to be exchanged. This paper provides an Ontology-driven Data Cleaning mechanism, facilitating Lossless Healthcare Data Compression to efficiently compress healthcare data of different nature (textual, audio, image). The latter is being evaluated considering three datasets of different formats, concluding to the added value of the described mechanism.

Improving sign-algorithm convergence rate using natural gradient for lossless audio compression

In lossless audio compression, the predictive residuals must remain sparse when entropy coding is applied. The sign algorithm (SA) is a conventional method for minimizing the magnitudes of residuals; however, this approach yields poor convergence performance compared with the least mean square algorithm. To overcome this convergence performance degradation, we propose novel adaptive algorithms based on a natural gradient: the natural-gradient sign algorithm (NGSA) and normalized NGSA. We also propose an efficient natural-gradient update method based on the AR(p) model, which requires mathcal {O}(p) multiply–add operations at every adaptation step. In experiments conducted using toy and real music data, the proposed algorithms achieve superior convergence performance to the SA. Furthermore, we propose a novel lossless audio codec based on the NGSA, called the natural-gradient autoregressive unlossy audio compressor (NARU), which is open-source and implemented in C. In a comparative experiment with existing, well-known codecs, NARU exhibits superior compression performance. These results suggest that the proposed methods are appropriate for practical applications.

Polynomial Representation of Binary Trees of Entropy Binary Codes

An important component of streaming large amounts of information are algorithms for compressing information flow. Which in turn are divided into lossless compression algorithms (entropic) - Shannon, Huffman, arithmetic coding, conditional compression - LZW, and otherinformation cone injections and lossy compression algorithms - such as mp3, jpeg and others.
 It is important to follow a formal strategy when building a lossy compression algorithm. It can be formulated as follows. After describing the set of objects that are atomic elements of exchange in the information flow, it is necessary to build an abstract scheme of this description, which will determine the boundary for abstract sections of this scheme, which begins the allowable losses.
 Approaches to the detection of an abstract scheme that generates compression algorithms with allowable losses can be obtained from the context of the subject area. For example, an audio stream compression algorithm can divide a signal into simple harmonics and leave among them those that are within a certain range of perception. Thus, the output signal is a certain abstraction of the input, which contains important information in accordance with the context of auditory perception of the audio stream and is represented by less information. A similar approach is used in the mp3 format, which is a compressed representation.
 Unlike lossy compression algorithms, entropic compression algorithms do not require contextanalysis, but can be built according to the frequency picture. Among the known algorithms for constructing such codes are the Shannon-Fano algorithm, the Huffman algorithm and arithmetic coding.
 Finding the information entropy for a given Shannon code is a trivial task. The inverse problem, namely finding the appropriate Shannon codes that have a predetermined entropy and with probabilities that are negative integer powers of two, is quite complex. It can be solved by direct search, but a significant disadvantage of this approach is its computational complexity. This article offers an alternative technique for finding such codes.