Data Compression Scheme Research Articles

Modular-Based Compression Scheme for Address Data in the Blockchain System for IoV Applications

Modular-Based Compression Scheme for Address Data in the Blockchain System for IoV Applications

Improving Cloud Database Performance Using Storage Technologies

Cloud databases have become indispensable for modern applications due to their scalability, flexibility, and cost-effectiveness. However, optimizing performance in cloud environments can be challenging due to factors such as network latency, data distribution, and resource contention. This paper explores various storage technologies and techniques that can significantly enhance cloud database performance. We delve into the characteristics of different storage types, including solid-state drives (SSDs), hard disk drives (HDDs), and object storage, and discuss their suitability for various workloads. Additionally, we examine caching mechanisms, data compression, and deduplication strategies that can optimize data access and reduce storage costs. Furthermore, we explore the impact of database architecture choices, such as distributed databases and inmemory databases, on performance. We also discuss the importance of effective query optimization and indexing techniques. By understanding and implementing these storage technologies and techniques, organizations can achieve substantial improvements in cloud database performance, leading to better user experiences and increased application efficiency

A Novel intelligent SAV oriented QL-based task offloading in mobile edge environments

Edge computing is a novel and potential computing model which moves storage and computing capabilities to the network edge, substantially decreasing service latency and network traffic. The existing Internet of Things (IoT) network offloading algorithm faces limitations such as a fixed number of applications, high edge-to-edge delay, and reliance on a single Mobile Edge Computing (MEC) server, posing security and privacy concerns. Moreover, resource-constrained mobile devices need more effective data integration and compression strategies. Addressing these challenges, this study suggests an approach based on deep reinforcement learning (DRL), specifically the SAV (State Action Value) Oriented QL (Q-Learning) based Task Offloading method, to optimise task offloading and resource allocation in edge-cloud computing. The model aims to empower Mobile Devices (MDs) to develop optimal offloading decisions for long-term Quality Perception, utilising a neural network to establish the relationship between MD state and action value. The paper introduces a Recurrent Extended Memory Network (REMN) to capture dynamic workload behaviour at Edge nodes (ENs). It incorporates Quality Mapping, Quality Estimation, and a Quality-Aware DRL Task Offloading Algorithm to improve the accuracy and efficiency of the offloading procedure in MEC systems. This systematic approach improves overall system performance and enables MDs to leverage ENs for neural network training, reducing computational burdens. As a result, it can accomplish a more significant number of tasks, reducing latency from 0.74 ms to 7.168 ms and decreasing energy consumption from 270 J to 1820.39 J for tasks ranging from 10 to 50, respectively.

Study on the combination of virtual machine tools and wearable vibration devices for operators experiencing cutting forces in the milling process

The primary goal of this study is to develop a wearable system for providing CNC machine operators with visual and tactile perception of triaxial cutting forces, thereby assisting operators in industrial environments to enhance work efficiency and prevent mechanical failures. To achieve this goal, we successfully integrated a virtual machining tool simulator with the remote-control wearable system (RCWS). Using the ‘King Path’ milling parameters, we employed the simulation software developed by the AIM-HI team to calculate static and dynamic cutting forces, converting this data into vibrational commands for the RCWS to generate corresponding tactile feedback. Furthermore, we conducted extensive experiments, testing various data conversion methods, including three sampling techniques and two data compression strategies, aiming to provide accurate tactile feedback related to cutting forces under different operating conditions.

Neural-Based Compression Scheme for Solar Image Data

Neural-Based Compression Scheme for Solar Image Data

Multifocal multiview imaging and data compression based on angular-focal-spatial representation.

Multifocal multiview (MFMV) is an emerging high-dimensional optical data that allows to record richer scene information but yields huge volumes of data. To unveil its imaging mechanism, we present an angular-focal-spatial representation model, which decomposes high-dimensional MFMV data into angular, spatial, and focal dimensions. To construct a comprehensive MFMV dataset, we leverage representative imaging prototypes, including digital camera imaging, emerging plenoptic refocusing, and synthesized Blender 3D creation. It is believed to be the first-of-its-kind MFMV dataset in multiple acquisition ways. To efficiently compress MFMV data, we propose the first, to our knowledge, MFMV data compression scheme based on angular-focal-spatial representation. It exploits inter-view, inter-stack, and intra-frame predictions to eliminate data redundancy in angular, focal, and spatial dimensions, respectively. Experiments demonstrate the proposed scheme outperforms the standard HEVC and MV-HEVC coding methods. As high as 3.693 dB PSNR gains and 64.22% bitrate savings can be achieved.

12 × 2 pt combined probes: pipeline, neutrino mass, and data compression

With the rapid advance of wide-field surveys it is increasingly important to perform combined cosmological probe analyses. We present a new pipeline for simulation-based multi-probe analyses, which combines tomographic large-scale structure (LSS) probes (weak lensing and galaxy clustering) with cosmic microwave background (CMB) primary and lensing data. These are combined at the C ℓ-level, yielding 12 distinct auto- and cross-correlations. The pipeline is based on UFalconv2, a framework to generate fast, self-consistent map-level realizations of cosmological probes from input lightcones, which is applied to the CosmoGridV1 N-body simulation suite. It includes a non-Gaussian simulation-based covariance for the LSS tracers, several data compression schemes, and a neural network emulator for accelerated theoretical predictions. We validate the pipeline by comparing the simulations to these predictions, and our derived constraints to earlier analyses. We apply our framework to a simulated 12×2 pt tomographic analysis of KiDS, BOSS, and Planck, and forecast constraints for a ΛCDM model with a variable neutrino mass. We find that, while the neutrino mass constraints are driven by the CMB data, the addition of LSS data helps to break degeneracies and improves the constraint by up to 35%. For a fiducial Mν = 0.15 eV, a full combination of the above CMB+LSS data would enable a 3σ constraint on the neutrino mass. We explore data compression schemes and find that MOPED outperforms PCA and is made robust using the derivatives afforded by our automatically differentiable emulator. We also study the impact of an internal lensing tension in the CMB data, parametrized by AL , on the neutrino mass constraint, finding that the addition of LSS to CMB data including all cross-correlations is able to mitigate the impact of this systematic. UFalconv2 and a MOPED compressed Planck CMB primary + CMB lensing likelihood are made publicly available.[UFalconv2: https://cosmology.ethz.ch/research/software-lab/UFalcon.html, compressed Planck CMB primary + CMB lensing likelihood: https://github.com/alexreevesy/planck_compressed.]

Cocv: A compression algorithm for time-series data with continuous constant values in IoT-based monitoring systems

Sensor-generated time-series data now constitutes a significant and growing portion of the world’s data due to the rapid proliferation of the Internet of Things (IoT). The transmission and storage of such voluminous data have emerged as enormous challenges. Data compression and reduction strategies have been instrumental in mitigating these challenges to some extent. However, they have exhibited limitations when applied to real-time IoT-based monitoring systems. This stems from their failure to adequately consider the stringent requirements of real-time data transmission and the continuous constant-value redundancy within periodic monitoring data. Consequently, we introduce a dedicated compression algorithm tailored specifically for time-series data within periodic IoT-based monitoring systems, namely Cocv. It takes advantage of the continuous constant-value repetition of the time-series data to compress data by discarding redundant data points. It can not only compress static batches of data but also dynamically compress data streams to improve system performance in real-time IoT-based monitoring systems. The offline Cocv outperforms traditional compressors on gas-leak monitoring data with a compression ratio of 98.5%, maintaining a decent speed for both compression and decompression. In an actual IoT-based gas-leak monitoring system, the online Cocv improves handling capacity by 255%, reading speed by 728%, reduces bandwidth consumption by 94%, and storage space consumption by 98% compared to the original scheme.

VxH: A Systematic Determination of Efficient Hierarchical Voxel Structures

Three-dimensional (3D) maps with many millions to billions of points are now used in an increasing number of applications, with processing rates in the hundreds of thousands to millions of points per second. In mobile applications, power and energy consumption for managing such data and extracting useful information thereof are critical concerns. We have developed structures and methodologies with the purpose of minimizing memory usage and associated energy consumption for indexing and serialization of voxelized point-clouds. The primary source of points in our case is airborne laser scanning, but our methodology is not restricted to only this setting. Our emulated results show a memory usage reduction factor of roughly up to 200× that of Octree/Octomap and a file size reduction factor of up to 1.65× compared with the predominating compression scheme for airborne Lidar data, LASzip. In addition, our structures enable significantly more efficient processing since they are included in a hierarchical structure that captures geometric aspects.

Energy-efficient algorithms for lossless data compression schemes in wireless sensor networks

Wireless sensor networks (WSNs) are reliant on limited power resources, primarily provided by small batteries within sensor nodes. Inefficient energy management within these networks can lead to premature battery depletion during data transmission between sensor nodes, significantly impacting network longevity. Data compression emerges as a viable strategy to mitigate energy consumption by reducing data size before transmission and employing various compression and decompression techniques. This work presents a comparative analysis of data compression algorithms tailored for WSNs. It studies and enhances two adaptive lossless data compression techniques, namely ‘Adaptive Lossless Data Compression’ (ALDC) and ‘Fast and Efficient, Lossless Adaptive Compression System’ (FELACS), as means to effectively manage energy consumption in wireless sensor networks. ALDC and FELACS algorithms encode differences between consecutive data readings, thereby reducing the number of bits required for encoding. ALDC employs Huffman coding, while FELACS leverages the Golomb-Rice coding method. Encoding data samples by using three Huffman tables interchangeably as an enhancement of the ALDC algorithm, resulted in an improvement in energy saving from 73 % to 77 %. Analysis of FELACS unveiled the impact of natural phenomena-induced anomalies on measured data, identified as outliers. The outliers disrupt data patterns and ranges, subsequently altering the optimal coding parameters for data samples, resulting in encoding and decoding errors. This study proposes a robust method for identifying and replacing outliers within sensor data, significantly enhancing compression performance. A reduction of variations in dataset patterns facilitated more accurate sampling and encoding of data. Consequently, fewer bits are required to encode data samples, rendering the algorithm energy-efficient and suitable for applications demanding error-free data recovery or meticulous error analysis. The proposed method was successfully applied to the modified ALDC algorithm, exhibiting efficient performance. An optimum block size of sampled data was discovered for Fishnet relative humidity deployment ensuring efficient transmission of environmental data real-world sensor network deployments like Fishnet, Lucerne, and Le Genepi. These findings underscore the potential for significant energy savings and improved data accuracy through adaptive lossless data compression techniques, making them valuable assets for applications with stringent energy constraints or demanding data integrity.

Multi-channel LFP recording data compression scheme using Cooperative PCA and Kalman Filter

The power consumption and wireless communication bandwidth restrict the channel count and amount of high-resolution intracortical Brain–Computer Interfaces(iBCIs). So, it is necessary to compress the captured data before transmission. Here, we propose a multi-channel data compression method on the correlation between channels for Local Field Potential(LFPs) recorded through Utah microelectrode arrays. The method improves the compression performance of the Stepwise Expectation-Maximization Principal Component Analysis(SEM-PCA) algorithm by implementing the Kalman filter. The multi-channel correlation is sampled with high resolution while the uniqueness of each channel is retained at a low rate. Then, the two parts of information are fused by the Kalman filter, and finally, the data is reconstructed with high precision and high rate. Our testing results verified the effectiveness of the proposed compression scheme. Compared to the traditional SEM-PCA, this algorithm significantly improved neural LFP reconstruction accuracy from 79.61% to 88%.

Reversible adversarial steganography for security enhancement

In adaptive steganography, existing universal stego post-processing methods can enhance security, but suffer from incorrect extraction of messages and undesired visual defects, especially in flat regions. To address this issue, a reversible adversarial steganography method is proposed by modifying the LSB or 2ndLSB of stego-images, which has promising visual quality and security. To that end, the content-adaptive adversarial perturbations are first generated, which consider image texture with noise residual features to counter deep learning-based steganalyzers. Then, a data compression strategy of adversarial perturbations is designed by applying lossless run-length encoding based on the sparse nature of non-zero elements in the perturbations to reduce the perturbation’s quantity. Finally, reversible data hiding based on ternary coding is applied to embed and extract stego images with compressed adversarial perturbations. Extensive experimental results demonstrate that the proposed method can effectively enhance security and visual quality compared with state-of-the-art methods.

Optimizing GPU-Based Graph Sampling and Random Walk for Efficiency and Scalability

Graph sampling and random walk algorithms are playing increasingly important roles today because they can significantly reduce graph size while preserving structural information, thus enabling computationally intensive tasks on large-scale graphs. Current frameworks designed for graph sampling and random walk tasks are generally not efficient in terms of memory requirement and throughput. Not to mention that some of them result in biased results. To solve the above problems, we introduce Skywalker+, a high-performance graph sampling and random walk framework on multiple GPUs supporting multiple algorithms. Skywalker+ makes four key contributions: First, it realizes highly paralleled alias method on GPUs. Second, it applies finely adjusted workload-balancing techniques and locality-aware execution modes to present a highly efficient execution engine. Third, it optimizes the GPU memory usage with efficient buffering and data compression schemes. Last, it scales to multi-GPU to further enhance the system throughput. Abundant experiments show that Skywalker+ exhibits significant advantage over the baselines both in performance and utility.

A data compression and optimal galaxy weights scheme for Dark Energy Spectroscopic Instrument and weak lensing data sets

ABSTRACT Combining different observational probes, such as galaxy clustering and weak lensing, is a promising technique for unveiling the physics of the Universe with upcoming dark energy experiments. The galaxy redshift sample from the Dark Energy Spectroscopic Instrument (DESI) will have a significant overlap with major ongoing imaging surveys specifically designed for weak lensing measurements: the Kilo-Degree Survey (KiDS), the Dark Energy Survey (DES), and the Hyper Suprime-Cam (HSC) survey. In this work, we analyse simulated redshift and lensing catalogues to establish a new strategy for combining high-quality cosmological imaging and spectroscopic data, in view of the first-year data assembly analysis of DESI. In a test case fitting for a reduced parameter set, we employ an optimal data compression scheme able to identify those aspects of the data that are most sensitive to cosmological information and amplify them with respect to other aspects of the data. We find this optimal compression approach is able to preserve all the information related to the growth of structures.

φ-OTDR signal compression scheme based on the compressed sensing theory.

In this paper, based on the compressed sensing theory and the orthogonal matching pursuit algorithm, we have designed a data compression scheme, taking the Space-Temporal graph, time domain curve, and its time-frequency spectrum of phase-sensitive optical time-domain reflectometer as the target signals. The compression rates of the three signals were 40%, 35%, and 20%, while the average reconstruction times were 0.74 s, 0.49 s, and 0.32 s. The reconstructed samples effectively retained the characteristic blocks, response pulses, and energy distribution that symbolize the presence of vibrations. The average correlation coefficients of the three kinds of reconstructed signals with the original samples were 0.88, 0.85, and 0.86, respectively, and then a series of quantitative metrics were designed to evaluate the reconstructing efficiency. We have utilized the neural network trained by the original data to identify the reconstructed samples with an accuracy of over 70%, indicating that the reconstructed samples accurately present the vibration characteristics.

Mathematical Preliminaries in the Case of Lossless Compression Markov Models

Compression schemes can be divided into two categories, lossy and lossless, but this paper presents lossless data compression models and the original data can be correctly recovered from the data compressed material. Some mathematical results are assumed; the results of probability tests are assumed and used to evaluate the compression techniques we will discuss. To learn more about math concepts for some of the topics in this article, see [2, 3]. First, we look at several ideas in information theory that provide a standard for the development of lossless data compression schemes are briefly reviewed. We next look at several ways to model data that lead to efficient data compression encryption schemes.

Field programmable gate array compression for large array multispeckle diffuse correlation spectroscopy.

Diffuse correlation spectroscopy (DCS) is an indispensable tool for quantifying cerebral blood flow noninvasively by measuring the autocorrelation function (ACF) of the diffused light. Recently, a multispeckle DCS approach was proposed to scale up the sensitivity with the number of independent speckle measurements, leveraging the rapid development of single-photon avalanche diode (SPAD) cameras. However, the extremely high data rate from advanced SPAD cameras is beyond the data transfer rate commonly available and requires specialized high-performance computation to calculate large number of autocorrelators (ACs) for real-time measurements. We aim to demonstrate a data compression scheme in the readout field-programmable gate array (FPGA) of a large-pixel-count SPAD camera. On-FPGA, data compression should democratize SPAD cameras and streamline system integration for multispeckle DCS. We present a SPAD array with 128 linear ACs embedded on an FPGA to calculate 12,288 ACFs in real time. We achieved a signal-to-noise ratio (SNR) gain of 110 over a single-pixel DCS system and more than threefold increase in SNR with respect to the state-of-the-art multispeckle DCS. The FPGA-embedded autocorrelation algorithm offers a scalable data compression method to large SPAD array, which can improve the sensitivity and usability of multispeckle DCS instruments.

The Typical Set and Entropy in Stochastic Systems with Arbitrary Phase Space Growth

The existence of the typical set is key for data compression strategies and for the emergence of robust statistical observables in macroscopic physical systems. Standard approaches derive its existence from a restricted set of dynamical constraints. However, given its central role underlying the emergence of stable, almost deterministic statistical patterns, a question arises whether typical sets exist in much more general scenarios. We demonstrate here that the typical set can be defined and characterized from general forms of entropy for a much wider class of stochastic processes than was previously thought. This includes processes showing arbitrary path dependence, long range correlations or dynamic sampling spaces, suggesting that typicality is a generic property of stochastic processes, regardless of their complexity. We argue that the potential emergence of robust properties in complex stochastic systems provided by the existence of typical sets has special relevance to biological systems.

Maximum Likelihood Estimation in Data-Driven Modeling and Control

Recently, various algorithms for data-driven simulation and control have been proposed based on the Willems’ fundamental lemma. However, when collected data are noisy, these methods lead to ill-conditioned data-driven model structures. In this article, we present a maximum likelihood framework to obtain an optimal data-driven model, the signal matrix model, in the presence of output noise. Data compression and noise-level estimation schemes are also proposed to apply the algorithm efficiently to large datasets and unknown noise-level scenarios. Two approaches in system identification and receding horizon control are developed based on the derived optimal estimator. The first one identifies a finite impulse response model. This approach improves the least-squares estimator with less restrictive assumptions. The second one applies the signal matrix model as the predictor in predictive control. The control performance is shown to be better than existing data-driven predictive control algorithms, especially under high noise levels. Both approaches demonstrate that the derived estimator provides a promising framework to apply data-driven algorithms to noisy data.

Column storage enables edge computation of biological big data on 5G networks.

With the continuous improvement of biological detection technology, the scale of biological data is also increasing, which overloads the central-computing server. The use of edge computing in 5G networks can provide higher processing performance for large biological data analysis, reduce bandwidth consumption and improve data security. Appropriate data compression and reading strategy becomes the key technology to implement edge computing. We introduce the column storage strategy into mass spectrum data so that part of the analysis scenario can be completed by edge computing. Data produced by mass spectrometry is a typical biological big data based. A blood sample analysed by mass spectrometry can produce a 10 gigabytes digital file. By introducing the column storage strategy and combining the related prior knowledge of mass spectrometry, the structure of the mass spectrum data is reorganized, and the result file is effectively compressed. Data can be processed immediately near the scientific instrument, reducing the bandwidth requirements and the pressure of the central server. Here, we present Aird-Slice, a mass spectrum data format using the column storage strategy. Aird-Slice reduces volume by 48% compared to vendor files and speeds up the critical computational step of ion chromatography extraction by an average of 116 times over the test dataset. Aird-Slice provides the ability to analyze biological data using an edge computing architecture on 5G networks.