Data Compression Research Articles

Improvement of Lossless Text Compression Methods using a Hybrid Method by the Integration of RLE, LZW and Huffman Coding Algorithms

In the field of computer science, data compression is essential in the process of data transfer because it reduces file size without losing information. Issues that depend on the characteristics of the text can greatly affect the effectiveness of any compression algorithm. Consequently, each traditional compression algorithm has its own strengths and limitations. Therefore, a highly efficient compression method that achieves higher compression ratio is needed. In this paper, based on the sequential implementation of Huffman, RLE and LZW algorithms, a hybrid data compression method has been proposed. The algorithms have been individually evaluated and compared in terms of compression ratio and compressed file size. We conclude that in terms of compression ratio and compressed file size, LZW is better than Huffman and RLE. The results show that the proposed hybrid algorithm achieves the highest average compression ratio is 2.42 and the lowest average compressed file size is 1138.5 or less compared to the individual implementations of the algorithms. Thus, it can be concluded that the proposed hybrid method enhances lossless text compression in terms of compression ratio and file size according to compression metrics.

Hardware optimization for effective switching power reduction during data compression in GOLOMB rice coding.

Loss-less data compression becomes the need of the hour for effective data compression and computation in VLSI test vector generation and testing in addition to hardware AI/ML computations. Golomb code is one of the effective technique for lossless data compression and it becomes valid only when the divisor can be expressed as power of two. This work aims to increase compression ratio by further encoding the unary part of the Golomb Rice (GR) code so as to decrease the amount of bits used, it mainly focuses on optimizing the hardware for encoding side. The algorithm was developed and coded in Verilog and simulated using Modelsim. This code was then synthesised in Cadence Encounter RTL Synthesiser. The modifications carried out show around 6% to 19% reduction in bits used for a linearly distributed data set. Worst-case delays have been reduced by 3% to 8%. Area reduction varies from 22% to 36% for different methods. Simulation for Power consumption shows nearly 7% reduction in switching power. This ideally suggest the usage of Golomb Rice coding technique for test vector compression and data computation for multiple data types, which should ideally have a geometrical distribution.

MicroGravity Explorer Kit (MGX): An Open-Source Platform for Accessible Space Science Experiments

The study of microgravity, a condition in which an object experiences near-zero weight, is a critical area of research with far-reaching implications for various scientific disciplines. Microgravity allows scientists to investigate fundamental physical phenomena influenced by Earth’s gravitational forces, opening up new possibilities in fields such as materials science, fluid dynamics, and biology. However, the complexity and cost of developing and conducting microgravity missions have historically limited the field to well-funded space agencies, universities with dedicated government funding, and large research institutions, creating a significant barrier to entry. This paper presents the MicroGravity Explorer Kit’s (MGX) design, a multifunctional platform for conducting microgravity experiments aboard suborbital rocket flights. The MGX aims to democratize access to microgravity research, making it accessible to high school students, undergraduates, and researchers. To ensure that the tool is versatile across different scenarios, the authors conducted a comprehensive literature review on microgravity experiments, and specific requirements for the MGX were established. The MGX is designed as an open-source platform that supports various experiments, reducing costs and accelerating development. The multipurpose experiment consists of a Jetson Nano computer with multiple sensors, such as inertial sensors, temperature and pressure, and two cameras with up to 4k resolution. The project also presents examples of codes for data acquisition and compression and the ability to process images and run machine learning algorithms to interpret results. The MGX seeks to promote greater participation and innovation in space sciences by simplifying the process and reducing barriers to entry. The design of a platform that can democratize access to space and research related to space sciences has the potential to lead to groundbreaking discoveries and advancements in materials science, fluid dynamics, and biology, with significant practical applications such as more efficient propulsion systems and novel materials with unique properties.

A Comparative Study of Text-Based Lossless Compression

Lossless data compression is a critical technique used to reduce file sizes without any loss of information during the encoding and decoding processes. This study presents a comparative analysis of two widely-used lossless compression algorithms: Huffman Encoding and Lempel-Ziv-Welch (LZW). The primary objective is to evaluate the performance of these algorithms in terms of compression ratio, compression time, decompression time, and space savings. The analysis was conducted on 100 files of varying sizes. The results demonstrate that the LZW algorithm outperforms Huffman Encoding, offering superior compression ratios, faster compression and decompression times, and greater disk space savings. These findings highlight the effectiveness of LZW for efficient data compression in practical applications.

Sustainable energy: Advancing wind power forecasting with grey wolf optimization and GRU models

Wind power forecasting is critical for optimizing energy use and ensuring the reliability of wind power systems in renewable energy. This paper introduces a novel method that combines the Grey Wolf Optimization (GWO) algorithm with data compression techniques to enhance feature selection and reduce redundancy in wind speed prediction. By employing GWO, essential features were identified by grouping the dataset into intervals and analyzing their frequencies. Performance evaluation was conducted using various compression measures, including Rate DC-Miss, Rate DC-MEF, and Rate DC-BDG, compared with other models such as extreme gradient boosting, space-time graph neural networks, and deep learning models. The study's results show significant improvements in accuracy and efficiency for predicting wind speed compared to existing techniques. The proposed approach addresses both larger datasets and the impact of noise samples on prediction errors. Additionally, an MLDDR model was introduced to predict DC power generated from wind datasets, encompassing five stages: Data Preparation, Feature Selection, Data Compression, GRU-Based Predictions, and Rate of Reduction. Data reduction results are notable. The original wind dataset (104857613) was reduced to 1093913 after processing missing values, achieving a reduction rate of 0.136. Applying the MEF-GWO algorithm further reduced the dataset to 109395, with a reduction rate of 0.385. The BDG dataset was compressed to 1805, with a reduction rate of 0.607. In terms of prediction performance, the GRU model was evaluated on three datasets: the original, MEF, and BDG-GWO datasets. The GRU model demonstrated the highest accuracy (99.20 %) with the BDG-GWO dataset, with precision (0.9965), recall (0.9978), and F1 scores (0.9897) indicating superior performance. Training and testing times varied significantly, highlighting the computational challenges associated with deep learning techniques. This research addresses both programming and application challenges. Programming challenges include high computational demands and the trial-and-error nature of parameter determination in deep learning, mitigated by using GWO. Application challenges involve reducing large datasets, grouping them into intervals, and evaluating performance using different compression measures. The main research questions addressed include the suitability of the GWO algorithm for dataset reduction in terms of dimensions (features and records) and the effectiveness of combining GWO with deep learning, specifically GRU, for enhanced prediction results. The study concludes that the GWO-PCA and GRU combination significantly improves prediction accuracy and reduces implementation time.

A Modified and Customized Wingsuit Flying Search (MCWFS)-based Approach for the Circular Scan Architecture

The circular scan architecture (CSA) is a widely used scan design technique in digital circuit testing that involves scanning test data through a circuit in a circular pattern. Several optimization approaches have been suggested so far to lessen the Test Data Volume (TDV), but no one has shown satisfactory results. Therefore, this paper proposes a Modified and Customized Wingsuit Flying Search algorithm to decrease the test data volume time. The proposed MCWFS-SBA-CSA introduces some novel features that distinguish it from traditional circular scan approaches. The concept of wingsuit flying is an innovative approach in the field of scan systems. This allows the scan that moves more dynamically and efficiently, enabling it to cover a larger area in a shorter time than traditional circular-scan approaches. After this phase, only scan-chosen inputs are required to obtain the next test pattern from the captured response. Therefore, required less Automatic test equipment (ATE) memory in this research so there is a Decreased cost in system-on-chip (SOC). The proposed method attains fast global searching capacity. The efficiency of the proposed technique attains 26.42%, 32.45%, and 29.34% higher compression ratio (CR), Test Application Time (TAT) reduction of SC-128 attains 29.45%, 34.67%, and 40.22% higher, TDV of SC-256 attains 32.23%, 36.34%, and 39.23% higher than existing methods, such as TDV for Circular Scan Approaches utilizing Modified Shuffled Shepard Optimization (TDV-CSA-MSSO), Effectual Very-large-scale integration Test data compression system for CSA under modified and colony metaheuristic (ETDC-CSA-MACMH), and Test Data Compression Utilizing Tetrad state Skip System (TDC-DC-TSSC) for Digital Circuits, respectively.

Reducing Bandwidth Usage in CVSLAM: A Novel Approach to Map Point Selection and Efficient Data Compression

In the field of collaborative visual simultaneous localization and mapping (CVSLAM), efficient data communication poses a significant challenge, particularly in environments with limited bandwidth. To address this issue, we introduce a method aimed at reducing communication consumption. Our approach starts with a strategic culling of map points, aiming at maximizing pose-visibility and expanding spatial diversity to effectively eliminate redundant data in CVSLAM. We achieve this by formulating the problem of maximizing pose-visibility and spatial diversity as a minimum-cost maximum-flow graph optimization problem. Subsequently, we apply finite state entropy encoding for the compression of visual information, further alleviating bandwidth constraints. To verify the proposed method, we implement it within a centralized collaborative monocular simultaneous localization and mapping (SLAM) system. Our approach has been tested on publicly available datasets and in real-world scene. The results show a prominent reduction in bandwidth usage by 49% while maintaining mapping accuracy and without introducing additional latency, confirming its effectiveness in a multi-agent system setting.

Shadow Simulation of Quantum Processes.

We introduce the task of shadow process simulation, where the goal is to simulate the estimation of the expectation values of arbitrary quantum observables at the output of a target physical process. When the sender and receiver share random bits or other no-signaling resources, we show that the performance of shadow process simulation exceeds that of conventional process simulation protocols in a variety of scenarios including communication, noise simulation, and data compression. Remarkably, we find that there exist scenarios where shadow simulation provides increased statistical accuracy without any increase in the number of required samples.

MEDCO: an efficient protocol for data compression in wireless body sensor network

MEDCO: an efficient protocol for data compression in wireless body sensor network

Adaptive stepsize forward–backward pursuit and acoustic emission-based health state assessment of high-speed train bearings

Compressed sensing (CS) is a promising tool for data compression reconstruction. However, fault diagnosis methods for high-speed train bearings based on CS and acoustic emission (AE) technologies have not been reported yet. Notably, the accuracy and speed of CS two-stage reconstruction methods are affected and restricted by prior initial conditions. Therefore, this article proposes adaptive dynamic thresholds applicable to adaptive stepsize forward–backward pursuit (ASFBP), and bearing health state assessment method. First, the adaptive dynamic thresholds for atom selection and deletion are constructed based on the residual feedback mechanism and the atom quality distribution law, which enables ASFBP to realize high-precision rapid reconstruction of signal without any atom priori initial conditions. Second, the initial dictionary length is improved based on the AE hit characteristics. Furthermore, a damage state comprehensive evaluation index (DSCEI) is established using principal component analysis based on AE time-domain hit parameters and compression-domain energy parameter. Compared with the kurtosis index and permutation entropy index, the DSCEI demonstrates better monotonicity and stability in the quantitative evaluation of high-speed train bearing condition. Finally, the validity and stability of the method are verified by testing under complex test conditions resembling actual high-speed train lines, providing valuable insights for the CS-based data-driven bearing fault diagnosis.

One-index vector quantization based adversarial attack on image classification

To improve storage and transmission, images are generally compressed. Vector quantization (VQ) is a popular compression method as it has a high compression ratio that suppresses other compression techniques. Despite this, existing adversarial attack methods on image classification are mostly performed in the pixel domain with few exceptions in the compressed domain, making them less applicable in real-world scenarios. In this paper, we propose a novel one-index attack method in the VQ domain to generate adversarial images by a differential evolution algorithm, successfully resulting in image misclassification in victim models. The one-index attack method modifies a single index in the compressed data stream so that the decompressed image is misclassified. It only needs to modify a single VQ index to realize an attack, which limits the number of perturbed indexes. The proposed method belongs to a semi-black-box attack, which is more in line with the actual attack scenario. We apply our method to attack three popular image classification models, i.e., Resnet, NIN, and VGG16. On average, 55.9 % and 77.4 % of the images in CIFAR-10 and Fashion MNIST, respectively, are successfully attacked, with a high level of misclassification confidence and a low level of image perturbation.

A new representation learning based maximum power operation towards improved energy management integration with DG controllers for photovoltaic generators using online deep exponentially expanded RVFLN algorithm

To incorporate Local Energy Management System (LEMS) based Tertiary Controller (TC) operation with multiple Photovoltaic based Distributed Generations (PV-DGs) level Primary Controller/ PC: Independent DG Controllers (IDGC) more adequately, a new representation learning (Symmetrical Input-Output weight Encoded Data Compression/ SIODC, with Exponential Expansion based Random Vector Functional Link Network towards Softmax layer’s Control Reference Estimation/ ExRVFLN-SoCRE) based MPPT controller is proposed in this paper in terms of Secondary Controllers (SCs). A new Hybrid LEMS (HyLEMS) is presented towards the proposed Deep Neural Network based SCs (i.e. DNN-SCs) in a distributed (SCdist), as well as centralized (SCcent) manner, to ease the computational, and communication requirements. To investigate that multiple PV-DGs based duty cycle (kth instant Control References/CRs estimation) controllers are considered here with auxiliary Battery Energy Storage Systems (BESS), and AC utility, integrated to Common DC feeder in terms of DC-DC converter dynamics. To avoid Control Reference Estimation Error (CREE) due to initial randomization, optimization subroutines are incorporated for SIODC by generalized semi-supervised learning, and for ExRVFLN-SoCRE by Lagrange multiplier weighted, rms based cost function. The proposed control performance is verified in MATLAB Simulink® based average modeling, and validated through dSPACE DS1104 based RTI with multi-PV (emulators) test-bench as well.

Prime number factorization and degree of coherence of speckled light beams.

We discover a connection between a Gauss sum of number theory and the degree of coherence (DOC) of the field in a transverse plane of structured speckled light beams. We theoretically demonstrate and experimentally validate that prime number factorization can be achieved by manipulating the source beam's DOC in Young's double-slit experiment. The determination of whether a number can be factored is based solely on the visibility of the resulting interference patterns. Our findings offer new insights into information encryption and decryption, data compression, etc.

The Effect of Unsteady Inlet Boundary Conditions on the Aero-Thermal Behavior of High-Pressure Turbine Vanes – A Numerical Study Using Scale-Resolving Simulations

Abstract This paper presents the application of a novel method to prescribe unsteady boundary conditions to transient, scale-resolving computational fluid dynamics simulations of the high-pressure turbine in modern jet engines. The methodology is based on the compression of the interface data at the combustor–turbine interface, using proper orthogonal decomposition and Fourier series (PODFS). Doing so can reduce the stored data at the interface drastically. The capability of the PODFS method to produce realistic inlet boundary conditions was demonstrated in previous work. Here, the method is applied to a turbine case. The outlet data of a combustor simulation is used to create the PODFS boundary conditions for a scale-resolving simulation of a simplified first nozzle guide vane of the high-pressure turbine. This simulation is compared with simulations with steady-state boundary conditions to show the effect of unsteadiness in the inlet boundary condition on the aerodynamic and thermal behaviors of the turbine. While the aerodynamics show minor sensitivity against the way of applying the inlet boundary conditions, the thermal behavior of the vanes is strongly affected by the modeling of combustor unsteadiness.

Optimized Tiny Machine Learning and Explainable AI for Trustable and Energy-Efficient Fog-Enabled Healthcare Decision Support System

The Internet of things (IoT)-based healthcare decision support system plays a crucial role in modern medicine, especially with the rise in chronic illnesses and an aging population necessitating continuous remote health monitoring. Current healthcare decision support systems struggle to deliver timely and accurate decisions with minimal latency due to limited real-time healthcare data and inefficient computational resources. There is a critical need for an optimized, energy-efficient machine learning model that reliably supports remote health monitoring within IoT and fog computing environments. Our study proposes an Optimized Tiny Machine Learning (TinyML) and Explainable AI (XAI) binary classification model for a trustable and energy-efficient healthcare decision support system, leveraging fog computing to optimize performance. The fog-based approach improves response times and enhances bandwidth usage, addressing critical needs such as reduced latency, higher bandwidth utilization, and decreased packet loss. To further improve efficiency, we incorporate the innovative mLZW data compression technique, significantly enhancing data communication efficiency and reducing response time to critical health alerts. However, limited real-time healthcare data records challenge machine learning classification performance. By implementing a TinyML algorithm, our system demonstrates superior performance to other machine learning models. The proposed optimized TinyML model achieves an impressive F1 score of 0.93 for health abnormalities detection, emphasizing its robustness and effectiveness. This paper highlights the potential of TinyML and XAI in delivering robust, trustworthy, and energy-aware healthcare solutions, making significant contributions toward effective remote health monitoring and decision support in fog-enabled IoT networks.Graphical abstract

An efficient data compression and storage technique with key management authentication in cloud space

Cloud computing is one of the promising technologies that offers cost-effective choices for processing and storing the huge volumes of data. In today’s world, data is the most important asset that one can have but it needs to be handled and protected properly. Portability of data can be increased by reducing the size of the data to be stored because of the limited storage space. As a result, data compression has arisen significantly. Data compression is a useful technique for reducing data size and increasing the effectiveness of data transit and storage. Data compression reduces the size of a data file while using lossy or lossless compression. One of the newest techniques for data compression is data duplication, which can reduce the amount of data saved while removing unnecessary data and maintaining an exact copy of the data. This analysis presents an Efficient data compression and storage technique with key management authentication in cloud space. This approach uses Regressive probabilistic key encryption (RPKE) to encrypt the cloud data and Lempel-Ziv-77-Huffman coding (LZ77-HM) is used to compress the huge amounts of cloud data. The Performance of presented approach is evaluated in terms of compression ratio and compression rate.

Wavefield Reconstruction of Distributed Acoustic Sensing: Lossy Compression, Wavefield Separation, and Edge Computing

AbstractDistributed acoustic sensing (DAS) presents challenges and opportunities for seismological research and data management. This study explores wavefield reconstruction using deep learning methods for data compression and wavefield separation. We test various architectures to treat DAS data as two‐dimensional arrays, such as the implicit neural representation (INR) models and the SHallow REcurrent Decoder (SHRED) model. The INR models present better data compression ability but do not generalize over space and time, a major practical limitation. On the other hand, SHRED generalizes over space and time for a single optical fiber with data from 20% decimated channels for the reconstruction. Despite good performance in reconstructing long‐wavelength features, the shallow recurrent decoder does not reconstruct transient earthquake wavefields at shorter wavelengths, limiting its usability for seismic data transmission. Nevertheless, we leverage wavefield reconstruction of ocean waves to separate them from the seismic wavefield and improve seismological use cases for earthquake detection and Earth imaging. In summary, as a lightweight deep learning model, SHRED is well suited for wavefield separation and lossy compression of the DAS data.

T-product based [formula omitted]-norm tensor principal component analysis and a finite-step convergence algorithm

T-product based tensor principal component analysis (ℓ2-tPCA) was used for dimensionality reduction, data preprocessing, compression, and visualization of multivariate data. However, ℓ2-tPCA may amplify the influence of outliers and large-magnitude noise. To explore robustness against heavily corrupted third-order data, we consider the ℓ1-norm tPCA model (ℓ1-tPCA). We develop an effective proximal alternating maximization method and prove that within finitely many steps, the algorithm stops at a point satisfying certain optimality conditions. Numerical experiments on color face reconstruction and recognition demonstrate the efficiency of the proposed algorithms, confirming that ℓ1-tPCA is more resilient to outliers compared to ℓ2-tPCA.

Compression of Battery X-Ray Tomography Data with Machine Learning

Abstract With the increasing demand for high-resolution x-ray tomography in battery characterization, the challenges of storing, transmitting, and analyzing substantial imaging data necessitate more efficient solutions. Traditional data compression methods struggle to balance reduction ratio and image quality, often failing to preserve critical details for accurate analysis. This study proposes a machine learning-assisted compression method tailored for battery x-ray imaging data. Leveraging physics-informed representation learning, our approach significantly reduces file sizes without sacrificing meaningful information. We validate the method on typical battery materials and different x-ray imaging techniques, demonstrating its effectiveness in preserving structural and chemical details. Experimental results show an up-to-95 compression ratio while maintaining high fidelity in the projection and reconstructed images. The proposed framework provides a promising solution for managing large-scale battery x-ray imaging datasets, facilitating significant advancements in battery research and development.

Design of real-time & historical database for photovoltaic power plant

Abstract The real-time & historical database of photovoltaic power plant monitoring system is used to store and manage the real-time and historical data, including power plant parameters, environmental data, and photovoltaic unit data and so on. Analyzing of this data is helpful for monitoring the status of the plant equipment, performing maintenance, and enhancing the operational efficiency of the power plant. The data models and database function of the database is designed. The data models include tags, data processing, and data storage models. Database function include the modules of the collection, storage, compression, and index query. The data collection module utilizes dynamic link libraries to access database data and configures parameters through a comprehensive data collection framework. The storage strategies involve in a combination of in-memory databases and disk-based historical databases, incorporating secondary caching technology and cloud server storage techniques. The data compression module is achieved through secondary compression methods. The index query module encompass time index files and the secondary index files to improve query speed and real-time system performance. The designed elements collectively establish an efficient monitoring system database.