Data Compression Research Articles

Large-FoV 3D imaging of single-photon LiDAR at up to 12 km

Single-photon light detection and ranging (LiDAR) has emerged as a valuable technology for active imaging applications. The growing demand for wider applications has driven advancements in measurement range, imaging resolution, and field of view (FoV). Here, we present a high-resolution, large-FoV single-photon LiDAR system capable of panoramic imaging. The system employs continuous raster scanning with a single-photon detector array to achieve rapid measurements, while timing-based pixel segmentation ensures precise data organization. A histogram-folding-based data compression and reconstruction method was developed, resulting in a two-order-of-magnitude increase in processing speed during simulations while maintaining high image reconstruction quality. The experimental results demonstrated the system’s capacity for rapid, high-resolution, and large-FoV imaging, achieving a resolution of 7.2′′ across a 1.6∘×2.1∘ FoV (1052 × 803 pixels) for buildings up to 12 km, all captured and processed in just 10 min. This approach has great potential for use in environmental mapping and related fields.

Linear prediction coefficients correction method for digital speech processing systems with data compression based on the autoregressive model of a voice signal

The problem of distortion of the autoregressive model of the voice signal under the influence of additive background noise in digital speech processing systems with data compression based on linear prediction is considered. In the frequency domain, these distortions are observed in the weakening of the main formants responsible for the intelligibility of the speaker’s speech. To compensate for formant attenuation, it is proposed to modify the parameters of the autoregressive model (linear prediction coefficients) using the impulse response of a recursive shaping filter. Along with the amplitude amplification of the formants, their frequencies remain unchanged to make the speaker’s voice recognizable. The effectiveness of the method was studied experimentally using specially developed software. Based on the experimental results, conclusions were drawn about a significant increase in the relative level of formants in the power spectrum of the corrected voice signal.

Efficient Data Compression of Acoustic Signals in Rail Using Sparse Decomposition and Kurtosis-Guided Resampling

Efficient Data Compression of Acoustic Signals in Rail Using Sparse Decomposition and Kurtosis-Guided Resampling

File Compression System Using Huffman Coding

Abstract—File compression, which minimizes file sizes without significantly compromising the quality of information contained in the file, is among the most critical aspects of data management since it facilitates fast flow and storage of data. Storage manage- ment has become more challenging due to the exponential growth of digital data; hence, efficient file compression has been an issue of great importance to help in decreasing storage expenses and increasing data transfer rates. This paper presents a web interface for the file compression system that is based on lossless data compression known as Huffman coding to reduce the size of the files without changing their content. A standard internet portal allows the users to upload their files to the system and get back resized files within the shortest time possible. The technique employed in the compression is astute in that it uses Huffman codes which give characters codes depending with their frequencies in the file. It then follows that since most of the file will have short codes; the file will take up less space. To safeguard the compressed information, the system generates a data structure called a Huffman tree in which no code is a sub- string of any other. This application does seize to be usable from a few selected devices such as desktops and laptop computers but can also be accessed from projectors and cell phones because of the effective design. The file compression system is also versatile and accommodates various file categories such text, image and binary among others providing users with convenience in a number of sectors. Index Terms—Data Compression, Hoffman Coding, HTML( Hypertext Markup Language), CSS ( Cascading Styling Sheet ), JavaScript.

The Logic and Application of Greedy Algorithms

Abstract. Being easy to implement and offering faster solutions, the greedy algorithms have widely been used in solving combinatorial optimization problems in computer science and many practical applications, such as resource allocation and data compression. This paper aims at analyzing the definitions, theory, and application of greedy algorithms in computer science as well as their logic and efficacy in numerous situations. In this analysis, concepts such as the greedy choice property and optimal substructure are examined as essential to understanding how greedy algorithms operate. The provided analysis mainly assumes prior knowledge of concepts and problems like the Fractional Knapsack problem and the Shortest Path problem solution as well as some practical problems, for example, resource allocation and the exact coin change problem. Analytical data and real applications are used to analyze the performance of greedy algorithms vis-a-vis other categories, such as the dynamic programming paradigm. The results reveal that greedy algorithms have the potential for great efficiency while also being restricted by certain parameters. They are ideal in the following aspects pertaining to computational aspects, that is, speed, ease of implementation, and applicability to large datasets. But they do not necessarily promise strong optimality in certain situations. The conclusion notes that greedy algorithms deserve further applications when the specified conditions are met and states that the effectiveness of these conditions needs to be studied.

Design and FPGA Implementation of a Hyper-Chaotic System for Real-time Secure Image Transmission

Recently, chaos theory has been widely used in multimedia and digital communications due to its unique properties that can enhance security, data compression, and signal processing. It plays a significant role in securing digital images and protecting sensitive visual information from unauthorized access, tampering, and interception. In this regard, chaotic signals are used in image encryption to empower the security; that's because chaotic systems are characterized by their sensitivity to initial conditions, and their unpredictable and seemingly random behavior. In particular, hyper-chaotic systems involve multiple chaotic systems interacting with each other. These systems can introduce more randomness and complexity, leading to stronger encryption techniques. In this paper, Hyper-chaotic Lorenz system is considered to design robust image encryption/ decryption system based on master-slave synchronization. Firstly, the rich dynamic characteristics of this system is studied using analytical and numerical nonlinear analysis tools. Next, the image secure system has been implemented through Field-Programmable Gate Arrays (FPGAs) Zedboard Zynq xc7z020-1clg484 to verify the image encryption/decryption directly on programmable hardware Kit. Numerical simulations, hardware implementation, and cryptanalysis tools are conducted to validate the effectiveness and robustness of the proposed system.

Three‐Way Data Reduction Based on Essential Information

ABSTRACTIn this article, the idea of essential information‐based compression is extended to trilinear datasets. This basically boils down to identifying and labelling the essential rows (ERs), columns (ECs) and tubes (ETs) of such three‐dimensional datasets that allow by themselves to reconstruct in a linear way the entire space of the original measurements. ERs, ECs and ETs can be determined by exploiting convex geometry computational approaches such as convex hull or convex polytope estimations and can be used to generate a reduced version of the data at hand. These compressed data and their uncompressed counterpart share the same multilinear properties and their factorisation (carried out by means of, for example, parallel factor analysis–alternating least squares [PARAFAC‐ALS]) yield, in principle, indistinguishable results. More in detail, an algorithm for the assessment and extraction of the essential information encoded in trilinear data structures is here proposed. Its performance was evaluated in both real‐world and simulated scenarios which permitted to highlight the benefits that this novel data reduction strategy can bring in domains like multiway fluorescence spectroscopy and imaging.

RGB-D Data Compression via Bi-Directional Cross-Modal Prior Transfer and Enhanced Entropy Modeling

RGB-D data, being homogeneous cross-modal data, demonstrates significant correlations among data elements. However, current research focuses only on a unidirectional pattern of cross-modal contextual information, neglecting the exploration of bidirectional relationships in the compression field. Thus, we propose a joint RGB-D compression scheme, which is combined with Bi-directional Cross-modal Prior Transfer (Bi-CPT) modules and a Bi-directional Cross-modal Enhanced Entropy (Bi-CEE) model. The Bi-CPT module is designed for compact representations of cross-modal features, effectively eliminating spatial and modality redundancies at different granularity levels. In contrast to the traditional entropy models, our proposed Bi-CEE model not only achieves spatial-channel contextual adaptation through partitioning RGB and depth features but also incorporates information from other modalities as prior to enhance the accuracy of probability estimation for latent variables. Furthermore, this model enables parallel multi-stage processing to accelerate coding. Experimental results demonstrate the superiority of our proposed framework over the current compression scheme, outperforming both rate-distortion performance and downstream tasks, including surface reconstruction and semantic segmentation. The source code will be available at https://github.com/xyy7/Learning-based-RGB-D-Image-Compression .

Dynamic mode decomposition of GRACE satellite data

Advancements in satellite technology yield environmental data with ever improving spatial coverage and temporal resolution. This necessitates the development of techniques to discern actionable information from large amounts of such data. We explore the potential of dynamic mode decomposition (DMD) to discover the dynamics of spatially correlated structures present in global-scale data, specifically in observations of total water storage anomalies provided by GRACE satellite missions. Our results demonstrate that DMD enables data compression and extrapolation from a reduced set of dominant spatiotemporal structures. The accuracy of its predictions of global system dynamics is preserved in its reconstruction of local time series. These findings suggest potential uses of DMD in analysis of remote-sensing data for hydrologic applications.

Lossy Compression of Integer Astronomical Images Preserving Photometric Properties* *This work was supported in part by the Spanish Ministry of Science and Innovation (MICINN) and by the European Regional Development Fund (FEDER), funded by MCIN/AEI/10.13039/501100011033/FEDER, UE, under grant PID2021-125258OB-I00; also by the Catalan Government under grant SGR2021-00643; and also by the “PhotSat” project under the Institute for Space Studies of

Observatories are producing astronomical image data at quickly increasing rates. As a result, the efficiency of the compression methods employed is critical to meet the storage and distribution requirements of both observatories and scientists. This paper presents a novel lossy compression technique that is able to preserve the results of photometry analysis with high fidelity while improving upon the state of the art in terms of compression performance. The proposed compression pipeline combines a flexible bi-region quantization scheme with the lossless, dictionary-based, LPAQ9M encoder. The quantization process allows compression performance and photometric fidelity to be precisely tailored to different scientific requirements. A representative data set of 16-bit integer astronomical images produced by telescopes from all around the world has been employed to empirically assess its compression-fidelity trade-offs, and compare them to those of the de facto standard Fpack compressor. In these experiments, the widespread SExtractor software is employed as the ground truth for photometric analysis. Results indicate that after lossy compression with our proposed method, the decompressed data allows consistent detection of over 99% of all astronomical objects for all tested telescopes, maintaining the highest photometric fidelity (as compared to state of the art lossy techniques). When compared to the best configuration of Fpack (Hcompress lossy using 1 quantization parameter) at similar compression rates, our proposed method provides better photometry precision: 7.15% more objects are detected with magnitude errors below 0.01, and 9.13% more objects with magnitudes below SExtractor’s estimated measurement error. Compared to the best lossless compression results, the proposed pipeline allows us to reduce the compressed data set volume by up to 38.75% and 27.94% while maintaining 90% and 95%, respectively, of the detected objects with magnitude differences lower than 0.01 mag; and up to 18.93% while maintaining 90% of the detected objects with magnitude differences lower than the photometric measure error.

Image Data Compression and Decompression Unit Considering Brightness Sensitivity

Image Data Compression and Decompression Unit Considering Brightness Sensitivity

Comparative Analysis of Huffman Coding Implementations for Efficient Data Communication Using Greedy and Divide-and-Conquer Techniques

Efficient data compression techniques are required to minimize storage and processing overhead due to modern systems' growing amount of data. Huffman Coding is a lossless compression technique that maintains data integrity by assigning shorter bit codes to characters appearing frequently, reducing size. Our analysis focuses on two implementation methodologies: greedy technique and divide and conquer. To find efficient solutions, divide-and-conquer algorithms partition problems into smaller components. In contrast, greedy algorithms strive to attain the utmost attainable result at each level. Our extensive investigation centers on the timing and space intricacies of diverse methodologies, enabling a comparative analysis that underscores their respective merits and drawbacks.

Using the Discrete Wavelet Transform for Lossy On‐the‐Fly Compression of GPU Fluid Simulations

ABSTRACTHigh‐performance computing in fluid dynamics frequently confronts substantial memory demands, especially in large‐scale applications. Data compression techniques can alleviate these memory constraints, but introduce new challenges. This paper introduces an innovative on‐the‐fly low‐overhead lossy compression technique tailored for GPU‐based fluid simulations, utilizing the discrete wavelet transform (DWT). The technique is applicable to any numerical scheme where the data is stored on a regular grid and the time step is computed using a stencil. Our approach significantly diminishes memory requirements, achieving up to a 10‐fold long‐term reduction on a D3Q27 simulation, while minimally impacting the simulation accuracy. The methodology is built around careful design choices to achieve a satisfactory compression ratio/speed trade‐off. It effectively maintains mass conservation and accurately preserves essential discontinuities in simulations. Extensive testing with a D3Q27 Lattice‐Boltzmann method (LBM) simulation on a single GPU has shown that large‐scale grids can be processed with minimal impact on the simulation accuracy and acceptable compression times. This compression technique demonstrates a robust capability to handle memory limitations in fluid simulations, opening the door to more complex and larger‐scale simulations.

Mechanics-informed autoencoder enables automated detection and localization of unforeseen structural damage

Structural health monitoring ensures the safety and longevity of structures like buildings and bridges. As the volume and scale of structures and the impact of their failure continue to grow, there is a dire need for SHM techniques that are scalable, inexpensive, can operate passively without human intervention, and are customized for each mechanical structure without the need for complex baseline models. We present a novel “deploy-and-forget” approach for automated detection and localization of damage in structures. It is a synergistic integration of entirely passive measurements from inexpensive sensors, data compression, and a mechanics-informed autoencoder. Once deployed, the model continuously learns and adapts a bespoke baseline model for each structure, learning from its undamaged state’s response characteristics. After learning from just 3 hours of data, it can autonomously detect and localize different types of unforeseen damage. Results from numerical simulations and experiments indicate that incorporating the mechanical characteristics into the autoencoder allows for up to a 35% improvement in the detection and localization of minor damage over a standard autoencoder. Our approach holds significant promise for reducing human intervention and inspection costs while enabling proactive and preventive maintenance strategies. This will extend the lifespan, reliability, and sustainability of civil infrastructures.

Computing MEMs and Relatives on Repetitive Text Collections

We consider the problem of computing the Maximal Exact Matches (MEMs) of a given pattern \(P[1\mathinner{.. }m]\) on a large repetitive text collection \(T[1\mathinner{.. }n]\) over an alphabet of size \(\sigma\) , which is represented as a (hopefully much smaller) run-length context-free grammar of size \(g_{rl}\) . We show that the problem can be solved in time \(O(m^{2}\log^{\epsilon}n)\) , for any constant \(\epsilon\gt0\) , on a data structure of size \(O(g_{rl})\) . Further, on a locally consistent grammar of size \(O(\delta\log\frac{n\log\sigma}{\delta\log n})\) , the time decreases to \(O(m\log m(\log m+\log^{\epsilon}n))\) . The value \(\delta\) is a function of the substring complexity of \(T\) and \(\Omega(\delta\log\frac{n\log\sigma}{\delta\log n})\) is a tight lower bound on the compressibility of repetitive texts \(T\) , so our structure has optimal size in terms of \(n\) , \(\sigma\) , and \(\delta\) . We extend our results to several related problems, such as finding \(k\) -MEMs, MUMs, rare MEMs, and applications. Categories and Subject Descriptors: E.1 [Data structures] ; E.2 [Data storage representations] ; E.4 [Coding and information theory]: Data compaction and compression; F.2.2 [Analysis of algorithms and problem complexity] : Nonnumerical algorithms and problems— Pattern matching, Computations on discrete structures, Sorting and searching

Model uncertainty quantification of a degradation model of miter gates using normalizing flow-based likelihood-free inference

Degradation modeling is essential for failure prognostics of miter gates and subsequent risk-informed maintenance or operational decision-making. Typically, degradation models are formulated based on certain assumptions and simplifications and may not fully capture the complexity of underlying degradation mechanisms. Even minor errors in such models may lead to significant discrepancies in failure prognostics due to error accumulation over time. Aiming to improve the accuracy of the degradation model and ensure reliable failure prognostics, this article proposes a degradation model updating framework for miter gates using normalizing flow-based likelihood-free inference, accounting for both model parameter uncertainty and model form uncertainty (i.e., model bias). The proposed work consists of two phases: the offline training phase and the online updating phase. During the offline training phase, the unknown model bias term is modeled as a random noise with uncertain standard deviation. Synthetic data are then generated using a physical model based on prior knowledge of the uncertain model parameters and model bias. The synthetic data are utilized to train an inference model, which has a summary network for data compression and a conditional invertible neural network for parameter estimation. In the online updating phase, the trained inference model uses strain observations to obtain posterior samples. The Gaussian mixture model and dual particle filter techniques are utilized to recursively update posterior samples, thereby improving the estimation accuracy. Subsequently, model bias is inversely estimated using the degradation model, which uses the updated model parameters and the gap length from the inference model. Following that, a regression model is constructed to correct the biases inherent in the simplified degradation model. This process is implemented iteratively over time to perform continuous model updating and failure prognostics. Results of a case study demonstrate the efficacy of the proposed framework in reducing both model parameter uncertainty and recovering model biases and thereby improving the accuracy of failure prognostics of miter gates.

Lossless Recompression of Vector Quantization Index Table for Texture Images Based on Adaptive Huffman Coding Through Multi-Type Processing

With the development of the information age, all walks of life are inseparable from the internet. Every day, huge amounts of data are transmitted and stored on the internet. Therefore, to improve transmission efficiency and reduce storage occupancy, compression technology is becoming increasingly important. Based on different application scenarios, it is divided into lossless data compression and lossy data compression, which allows a certain degree of compression. Vector quantization (VQ) is a widely used lossy compression technology. Building upon VQ compression technology, we propose a lossless compression scheme for the VQ index table. In other words, our work aims to recompress VQ compression technology and restore it to the VQ compression carrier without loss. It is worth noting that our method specifically targets texture images. By leveraging the spatial symmetry inherent in these images, our approach generates high-frequency symbols through difference calculations, which facilitates the use of adaptive Huffman coding for efficient compression. Experimental results show that our scheme has better compression performance than other schemes.

Quantum Machine Learning—Quo Vadis?

The book Quantum Machine Learning: What Quantum Computing Means to Data Mining, by Peter Wittek, made quantum machine learning popular to a wider audience. The promise of quantum machine learning for big data is that it will lead to new applications due to the exponential speed-up and the possibility of compressed data representation. However, can we really apply quantum machine learning for real-world applications? What are the advantages of quantum machine learning algorithms in addition to some proposed artificial problems? Is the promised exponential or quadratic speed-up realistic, assuming that real quantum computers exist? Quantum machine learning is based on statistical machine learning. We cannot port the classical algorithms directly into quantum algorithms due to quantum physical constraints, like the input–output problem or the normalized representation of vectors. Theoretical speed-ups of quantum machine learning are usually analyzed in the literature by ignoring the input destruction problem, which is the main bottleneck for data encoding. The dilemma results from the following question: should we ignore or marginalize those constraints or not?

Broadband and parallel multiple-order optical spatial differentiation enabled by Bessel vortex modulated metalens

Optical analog image processing technology is expected to provide an effective solution for high-throughput and real-time data processing with low power consumption. In various operations, optical spatial differential operations are essential in edge extraction, data compression, and feature classification. Unfortunately, existing methods can only perform low-order or selectively perform a particular high-order differential operation. Here, we propose and experimentally demonstrate a Bessel vortex modulated metalens composed of a single complex amplitude metasurface, which can perform multiple-order radial differential operations over a wide band by presetting the order of the corresponding Bessel vortex. This architecture further enables angle multiplexing to create multiple information channels that synchronously perform multi-order spatial differential operations, indicating the superiority of the proposed devices in parallel processing. Our approach may find various applications in artificial intelligence, machine vision, autonomous driving, and advanced biomedical imaging.

Data Pipeline Integrating Apache Kafka and Rabbit MQ

Amidst the dynamic realm of big data, the convergence of Apache Kafka and RabbitMQ into a cohesive symphony of data orchestration represents a seminal undertaking. This ambitious research project endeavors to harmonize these potent technologies, employing sophisticated methodologies such as data batching and compression to maximize their collaborative potential. The resulting amalgamation, a vanguard in big data, pledges to redefine the frontiers of data management and messaging systems. This integration affords unparalleled flexibility in selecting the most apt tool for a given task without compromising on performance or reliability. Furthermore, a unified management and monitoring interface empowers administrators and developers with comprehensive insights, simplifying the intricate orchestration of these two distinct but complementary platforms [1]. Ultimately, this research seeks not merely to unify Apache Kafka with RabbitMQ but to usher forth a paradigm where their union transcends the individual, revolutionizing the landscape of big data processing and transmission.