Lossless Data Research Articles

Lossless Recompression of Vector Quantization Index Table for Texture Images Based on Adaptive Huffman Coding Through Multi-Type Processing

With the development of the information age, all walks of life are inseparable from the internet. Every day, huge amounts of data are transmitted and stored on the internet. Therefore, to improve transmission efficiency and reduce storage occupancy, compression technology is becoming increasingly important. Based on different application scenarios, it is divided into lossless data compression and lossy data compression, which allows a certain degree of compression. Vector quantization (VQ) is a widely used lossy compression technology. Building upon VQ compression technology, we propose a lossless compression scheme for the VQ index table. In other words, our work aims to recompress VQ compression technology and restore it to the VQ compression carrier without loss. It is worth noting that our method specifically targets texture images. By leveraging the spatial symmetry inherent in these images, our approach generates high-frequency symbols through difference calculations, which facilitates the use of adaptive Huffman coding for efficient compression. Experimental results show that our scheme has better compression performance than other schemes.

Where the Patterns Are: Repetition-Aware Compression for Colored de Bruijn Graphs.

We describe lossless compressed data structures for the colored de Bruijn graph (or c-dBG). Given a collection of reference sequences, a c-dBG can be essentially regarded as a map from k-mers to their color sets. The color set of a k-mer is the set of all identifiers, or colors, of the references that contain the k-mer. While these maps find countless applications in computational biology (e.g., basic query, reading mapping, abundance estimation, etc.), their memory usage represents a serious challenge for large-scale sequence indexing. Our solutions leverage on the intrinsic repetitiveness of the color sets when indexing large collections of related genomes. Hence, the described algorithms factorize the color sets into patterns that repeat across the entire collection and represent these patterns once instead of redundantly replicating their representation as would happen if the sets were encoded as atomic lists of integers. Experimental results across a range of datasets and query workloads show that these representations substantially improve over the space effectiveness of the best previous solutions (sometimes, even dramatically, yielding indexes that are smaller by an order of magnitude). Despite the space reduction, these indexes only moderately impact the efficiency of the queries compared to the fastest indexes.

Enhancing Load Balancing With In-Network Recirculation to Prevent Packet Reordering in Lossless Data Centers

Enhancing Load Balancing With In-Network Recirculation to Prevent Packet Reordering in Lossless Data Centers

A New Algorithm for Lossless Data Compression Based of Time Clock Mechanism

A New Algorithm for Lossless Data Compression Based of Time Clock Mechanism

Hardware optimization for effective switching power reduction during data compression in GOLOMB rice coding.

Loss-less data compression becomes the need of the hour for effective data compression and computation in VLSI test vector generation and testing in addition to hardware AI/ML computations. Golomb code is one of the effective technique for lossless data compression and it becomes valid only when the divisor can be expressed as power of two. This work aims to increase compression ratio by further encoding the unary part of the Golomb Rice (GR) code so as to decrease the amount of bits used, it mainly focuses on optimizing the hardware for encoding side. The algorithm was developed and coded in Verilog and simulated using Modelsim. This code was then synthesised in Cadence Encounter RTL Synthesiser. The modifications carried out show around 6% to 19% reduction in bits used for a linearly distributed data set. Worst-case delays have been reduced by 3% to 8%. Area reduction varies from 22% to 36% for different methods. Simulation for Power consumption shows nearly 7% reduction in switching power. This ideally suggest the usage of Golomb Rice coding technique for test vector compression and data computation for multiple data types, which should ideally have a geometrical distribution.

A Comparative Study of Text-Based Lossless Compression

Lossless data compression is a critical technique used to reduce file sizes without any loss of information during the encoding and decoding processes. This study presents a comparative analysis of two widely-used lossless compression algorithms: Huffman Encoding and Lempel-Ziv-Welch (LZW). The primary objective is to evaluate the performance of these algorithms in terms of compression ratio, compression time, decompression time, and space savings. The analysis was conducted on 100 files of varying sizes. The results demonstrate that the LZW algorithm outperforms Huffman Encoding, offering superior compression ratios, faster compression and decompression times, and greater disk space savings. These findings highlight the effectiveness of LZW for efficient data compression in practical applications.

DEVELOPMENT OF AEROSPACE IMAGES PRELIMINARY PROCESSING METHOD FOR SUBSEQUENT RECOGNITION AND IDENTIFICATION OF VARIOUS OBJECTS

Nowadays, the application of hyperspectral images is vital for every section of the humanity life such as agrotechnical research for the field condition state and water security. This article presents a new lossless data compression algorithm focused on the processing of hyperspectral aerospace images. The algorithm takes into account inter-band correlation and difference transformations to effectively reduce the range of initial values. correlation allows you to find the best reference channel that defines the sequence of operations in the algorithm, which contributes to a significant increase in the compression ratio while maintaining high data quality. The practical implementation of the algorithm lies in the process of the transfer the lower size file with high efficiency for unmanned aerial vehicle and satellites to save more computational resources. This method demonstrates high computational efficiency and can be applied to various tasks that require efficient storage and transmission of hyperspectral images. The importance of processing hyperspectral data and the problems associated with their volume and complexity of analysis were described. Current approaches to data compression are considered and their limitations are identified, which justifies the need to develop new methods. The relevance and necessity of effective compression algorithms for aerospace applications is emphasized. An analysis of existing methods and algorithms for compressing hyperspectral data was carried out. Particular attention is paid to approaches that use cross-channel correlation and difference transformations. The effectiveness of current methods is evaluated and their shortcomings are identified, which serves as a justification for the development of a new algorithm. A developed lossless data compression algorithm based on the use of inter-band correlation and difference transformations was described. The stages of forming groups of channels and the selection of optimal compression parameters are considered in detail. The method of determining the reference channel, which sets the sequence of operations in the algorithm, which provides more efficient data compression, is especially noted. The advantages and possible limitations of the new approach, as well as its potential for practical use, are discussed. It was noted that the developed method successfully solves the problems associated with the volume of hyperspectral data, providing a high compression ratio without quality loss. The prospects for further development of the algorithm and its application in various fields of science and technology are discussed.

A SoC-FPGA based readout platform for the free-running AMBER data acquisition system

Owing to a change in the scope of physics research, the AMBER spectrometer at CERN is undergoing an update on its instrumentation and trigger strategy for the data acquisition system. One of the key updates is the adoption of a free-running and trigger-less operation, which differs from the predecessor of AMBER, COMPASS, by working in a continuous mode but in an event-based triggered way.In this article, we present a multichannel data-acquisition platform developed in line with the next generation of trigger-less and free-running data acquisition systems. The platform is based on a pre-existing Mezzanine Sampling ADC board for the analog-to-digital conversion and a Xilinx Zynq Ultrascale+ System on Module for online real-time data processing. We also present the evaluation of the system operating in continuous mode, taking data from a 25-element electromagnetic calorimeter prototype with a muon beam, during the first AMBER pilot run. The acquired data were used for noise and pulse shape studies required for the design of the algorithms for lossless compression and data feature extraction needed for trigger-less operation.

Predictive Queue-Based Rate Control for Low Latency in Lossless Data Center Networks

Predictive Queue-Based Rate Control for Low Latency in Lossless Data Center Networks

Mapping the CityGML Energy ADE to CityGML 3.0 Using a Model-Driven Approach

With the increasing adoption of semantic 3D city models, the relevance of applications in the field of Urban Building Energy Modelling (UBEM) has rapidly grown, as the building sector accounts for a large part of the total energy consumption. UBEM allows us to better understand the energy performance of the building stock and can contribute to defining refurbishment strategies. However, UBEM applications require lots of heterogeneous data, eventually advocating for standards for data interoperability. The Energy Application Domain Extension has been created to cope with UBEM data requirements and offers a standardised data model that builds upon the CityGML standard. The Energy ADE 1.0, released in 2018, creates new classes and adds new properties to existing classes of the CityGML 2.0 Core and Building modules. CityGML 3.0, released in 2021, has introduced several changes to the data model and its ADE mechanism. These changes render the Energy ADE incompatible with CityGML 3.0. This article presents how the Energy ADE has been ported to CityGML 3.0 to allow, on the one hand, for a lossless data conversion and, on the other hand, to exploit the new characteristics of CityGML 3.0 while keeping a logical symmetry between the ADE classes of both CityGML versions. The article describes the chosen methodology, the mapping strategies, the implementation steps, as well as the data conversion tests to check the validity of the “new” Energy ADE for CityGML 3.0.

Lossless Compression Method for the Magnetic and Helioseismic Imager (MHI) Payload

The Solar Polar-orbit Observatory (SPO), proposed by Chinese scientists, is designed to observe the solar polar regions in an unprecedented way with a spacecraft traveling in a large solar inclination angle and a small ellipticity. However, one of the most significant challenges lies in ultra-long-distance data transmission, particularly for the Magnetic and Helioseismic Imager (MHI), which is the most important payload and generates the largest volume of data in SPO. In this paper, we propose a tailored lossless data compression method based on the measurement mode and characteristics of MHI data. The background out of the solar disk is removed to decrease the pixel number of an image under compression. Multiple predictive coding methods are combined to eliminate the redundancy utilizing the correlation (space, spectrum, and polarization) in data set, improving the compression ratio. Experimental results demonstrate that our method achieves an average compression ratio of 3.67. The compression time is also less than the general observation period. The method exhibits strong feasibility and can be easily adapted to MHI.

Low-cost ANS encoder for lossless data compression in FPGAs

We present the implementation of the hardware ANS compressor in FPGAs. The main goal of the design was to propose a solution suitable to low-cost, low-energy embedded systems. We propose the streaming-rANS algorithm of the ANS family as a target for the implementation. Also, we propose a set of algorithm parameters that substantially reduce the use of FPGA resources, and we examine what is the influence of the chosen parameters on compression performance. Further, we compare our design to the lossless codecs found in literature, and to the streaming-rANS codecs with arbitrary parameters.

Lossless text compression by means of binary-coded ternary number representation

A lossless data compression code based on a binary-coded ternary number representation is investigated. The code is complete, universal, synchronizable, and supports a direct search in a compressed file. A simple monotonous encoding and very fast decoding algorithms are constructed owing to code properties. Experiments show that in natural language text compression, the new code outperforms byte-aligned codes SCDC and RPBC either in compression ratio or in decoding speed.

A Proposed IOT-based Smart Healthcare Management Framework for Performing Lossless Data Compression using Concept of Ontology

A Proposed IOT-based Smart Healthcare Management Framework for Performing Lossless Data Compression using Concept of Ontology

Lossless data hiding technique reducing cover data size for compressed videos

Lossless data hiding technique reducing cover data size for compressed videos

基于多头注意力与双向门控循环单元的数据无损压缩方法

基于多头注意力与双向门控循环单元的数据无损压缩方法

Lossless Text Compression Using Recurrent Neural Networks

Lossless Data compression is the process of reducing the size or the number of bits required to represent data, and Arithmetic coding is one of the popular lossless text compression techniques. This project focuses on lossless data compression using an innovative approach that combines arithmetic coding and Recurrent Neural Networks (RNN) with the help of encoder-decoder architecture. The proposed method aims to improve compression ratios compared to standard techniques. By leveraging the power of RNNs and arithmetic coding, encoder-decoder architecture effectively captures patterns and dependencies in the input data, enabling more efficient compression. The proposed approach achieved a remarkable improvement, achieving an impressive compression ratio that is, on average, 3.5 times smaller than that achieved by traditional methods. These results showcase the potential of developing improved general-purpose compressors based on neural networks and hybrid modeling. This demonstrates the potential of the proposed method for significantly enhancing lossless data compression performance.

Lossless Data Hiding in NTRU Cryptosystem by Polynomial Encoding and Modulation

Lossless Data Hiding in NTRU Cryptosystem by Polynomial Encoding and Modulation

An Event-Based Neural Compressive Telemetry With >11× Loss-Less Data Reduction for High-Bandwidth Intracortical Brain Computer Interfaces.

Intracortical brain-computer interfaces offer superior spatial and temporal resolutions, but face challenges as the increasing number of recording channels introduces high amounts of data to be transferred. This requires power-hungry data serialization and telemetry, leading to potential tissue damage risks. To address this challenge, this paper introduces an event-based neural compressive telemetry (NCT) consisting of 8 channel-rotating Δ-ADCs, an event-driven serializer supporting a proposed ternary address event representation protocol, and an event-based LVDS driver. Leveraging a high sparsity of extracellular spikes and high spatial correlation of the high-density recordings, the proposed NCT achieves a compression ratio of >11.4×, while consumes only 1 µW per channel, which is 127× more efficient than state of the art. The NCT well preserves the spike waveform fidelity, and has a low normalized RMS error <23% even with a spike amplitude down to only 31 µV.

Enhancing Fault Detection in Wireless Sensor Networks Through Support Vector Machines: A Comprehensive Study

The Wireless Sensor Network (WSN) consists of many sensors that are distributed in a specific area for the purpose of monitoring physical conditions. Factors such as hardware limitations, limited resources, unfavourable WSN deployment environment, and the presence of various attacks on nodes can lead to the presence of Faulty Nodes in a WSN. This raises the problem of detecting Faulty Nodes and avoiding Data loss. Detecting Faulty Nodes in real-world scenarios will improve the quality of a WSN. The research was aimed at developing an algorithm to determine the location of Faulty Nodes in a WSN. The algorithm uses characteristics of Machine Learning and Support Vector Machines (SVM), which use the classification of Data into true and false. A Mathematical Model for determining Faulty Nodes using the SVM is considered. A methodology for detecting a Faulty Node is demonstrated, which consists of Data Collection, Feature Extraction, Training, and Testing the Model. The Results of simulated experiments that were conducted with different numbers of nodes from 50 to 320 are shown. The Model is tested on Data very similar to real-world sensing Data to evaluate the ability of the Model to detect failed nodes and calculate training and testing errors. As a result, the training error is 4.6667%, the accuracy of detecting faulty nodes was 97%. The simulation results demonstrate the high stability of the proposed algorithm and are suitable for network environments with irregular node distribution or coverage gaps. In real scenarios, it can provide a high level of uninterrupted operation of the WSN and lossless data transmission. Shortcomings and prospects in research on fault detection in WSN, such as studying real-world hardware issues and its security, were presented.