Lempel-Ziv-Markov Chain Algorithm Research Articles

Synchro-waveform data compression using multi-stage hybrid coding algorithm

To augment situational awareness capabilities for high-frequency disturbances, the modern power system necessitates the implementation of high-sampling rates for real-time Synchro-Waveform (SW) measurements. Nevertheless, the nonlinear nature of data collected from SW measurement devices poses significant challenges to the communication and storage capacities of data centers. To effectively tackle this issue, this paper proposes a Multi-stage Hybrid Coding (MHC) method for the lossless compression of SW data. The MHC method utilizes a well-structured two-stage approach to achieve efficient compression. In the first stage, a variable frame-based method is designed to reduce redundant information and achieve initial compression using multiple periodic recursive compression. Subsequently, the dictionary-based method, specifically the Lempel–Ziv–Markov chain algorithm, is seamlessly integrated to further compress the SW data. Experimental and numerical analyses are conducted using both simulated and real-world source data, with real-time performance validation in SW measurement units. The obtained results convincingly demonstrate that the proposed method enables online compression of both event and non-event SW data, achieving an impressive reduction of over 73.4% in data space, while maintaining a data loss ratio lower than 0.4% for high-density SW data.

Modified Playfair for Text File Encryption and Meticulous Decryption with Arbitrary Fillers by Septenary Quadrate Pattern

AbstractCryptography secures data and serves to ensure the confidentiality of records. Playfair is a cryptographic symmetrical algorithm that encrypts statistics based on key costs. This secret is shared with an authorized person to retrieve data. In the conventional pattern, there is an area complexity and deficiency in letters, numbers, and special characters. This hassle has been overcome in previous studies by editing pattern dimensions. The fillers used throughout the enciphering were not eliminated during the retrieval process, which resulted in the indiscrimination of the retrieved statistics. The proposed method uses a separate quadrate pattern that strengthens the Playfair cipher and guarantees that the fillers are eliminated to ensure the authentic retrieval of records. The fillers indiscriminate and strengthen the set of rules in opposition to brute force and avalanche impact. The proposed algorithm was evaluated with a minimal change in the key, and was found to have an avalanche effect between 65% and 93.7%. The encrypted document is further encoded using the Lempel–Ziv–Markov chain algorithm (LZMA) to provide compressed second-level secured text with a compression ratio of 0.75 for a file size of 100 KB. The pattern was designed to subsidize the integrated characters found on the keyboard.

A Hybrid Data-Differencing and Compression Algorithm for the Automotive Industry.

We propose an innovative delta-differencing algorithm that combines software-updating methods with LZ77 data compression. This software-updating method relates to server-side software that creates binary delta files and to client-side software that performs software-update installations. The proposed algorithm creates binary-differencing streams already compressed from an initial phase. We present a software-updating method suitable for OTA software updates and the method’s basic strategies to achieve a better performance in terms of speed, compression ratio or a combination of both. A comparison with publicly available solutions is provided. Our test results show our method, Keops, can outperform an LZMA (Lempel–Ziv–Markov chain-algorithm) based binary differencing solution in terms of compression ratio in two cases by more than 3% while being two to five times faster in decompression. We also prove experimentally that the difference between Keops and other competing delta-creator software increases when larger history buffers are used. In one case, we achieve a three times better performance for a delta rate compared to other competing delta rates.

Class Imbalance Handling with Deep Learning Enabled IoT Healthcare Diagnosis Model

The rapid advancements in the field of big data, wearables, Internet of Things (IoT), connected devices, and cloud environment find useful to improve the quality of healthcare services. Medical data classification using the data collected by the wearables and IoT devices can be used to determine the presence or absence of disease. The recently developed deep learning (DL) models can be used for several processes such as classification, natural language processing, etc. This study presents a bacterial foraging optimization (BFO) based convolutional neural network-gated recurrent unit (CNN-GRU) with class imbalance handling (CIH) model, named BFO-CNN-GRU-CIH for medical data classification in IoT enabled cloud environment. The proposed BFO-CNN-GRU-CIH model initially enables the IoT devices to gather healthcare data and preprocess it for further processing. In addition, Lempel Ziv Markov chain Algorithm (LZMA) is employed for the compression of healthcare data to reduce the amount of data being communicated. Besides, Synthetic Minority Over-sampling Technique (SMOTE) is applied to handle class imbalance data problems. Moreover, BFO with CNN-GRU model is utilized to perform the classification process in which the hyperparameters of the CNN-GRU model are optimally adjusted by the use of BFO algorithm. In order to showcase the better performance of the BFO-CNN-GRU-CIH model, a wide range of simulations take place on three benchmark datasets and the results portrayed the betterment of the BFO-CNN-GRU-CIH model over the recent state of art approaches.

Optimal Compression of Remote Sensing Images Using Deep Learning during Transmission of Data

Industrial internet of things has grown quite popular in recent years and involves a large number of intelligent devices linked together to build a system that can investigate, communicate, gather and observe information. Due to this requirement, there is more demand for compression techniques which compresses data, leading to less usage of resources and low complexity. This is where Convolutional Neural Networks (CNN) play a large role in the field of computer vision, especially in places where high applications such as interpretation coupled with detection is required. Similarly, low-level applications such as image compression cannot be resolved using this methodology. In this paper, a compression technique for remote sensing images using CNN is proposed. This methodology incorporates CNN in a compact learning environment wherein the actual image that consists of structural data is coded using Lempel Ziv Markov chain algorithm. This process is followed by image reconstruction in order to obtain the actual image in high quality. Other methodologies such as optimized trunctiona, JPEG2000, JPEC and binary tree were compared using a large number of experiments in terms of space saving, reconstructed image quality and efficiency. The output obtained indicates that the proposed methodology shows effective improvement, attaining a 50 dB signal to noise ratio and space saving of 90%.

Semi-lossless Fractal MRI Image Compression Based on Fixed length Technique

Medical image compression plays an essential role to handle large amounts of data for communication and storage purposes. Fractal image compression is a potential lossy compression models with a resulting image that loses some of its information. However, health data communication usually cannot afford any lose for patients visual information. This paper proposes a new high efficiency semi-lossless fractal image compression method (SLFIC) based on fractal theory and fixed length technique. Technically, the resultant lossy fractals compressed image is analyzed and error in comparison with the original image is detected. Then, Fixed-length is developed to compress the detected errors and attached to the compressed image. In practice, a potential performance by the new developed model has been obtained in comparison with two other lossless models: ( Lion optimization algorithm (LOA) and Lempel Ziv Markov chain Algorithm (LZMA) with Linde–Buzo–Gray (LBG) (L2-LBG)) and(Neural Network Radial Basis Function (NNRBF)). Moreover, a high quality that has been obtained by the proposed system based on Peak Signal to Noise Ratio (PSNR) and Compression Ratio (CR).

Model-Free Lossless Data Compression for Real-Time Low-Latency Transmission in Smart Grids

This article proposes a model-free lossless data compression method for time series in smart grids (SGs), namely, Lossless Coding considering Precision (LCP) method. The LCP method encodes the current datapoint only using the immediate previous datapoint by differential coding, XOR coding, and variable length coding and transmits the encoded data once generated. It does not use the dynamics (e.g., many previous datapoints) or prior knowledge (e.g., mathematical models) of the time series. It considers the patterns, potential applications, and associated precision to preprocess the time series and especially suits high-resolution time series with long steady periods. The LCP method features low-latency and generalizability which enables real-time data communication for different time-critical tasks. Sub-metered load profiles in REDD dataset, high-resolution LIFTED dataset, AMPds dataset and PMU dataset are used to evaluate the performance of the LCP method. The results show that the LCP method demonstrates high compression ratio, low latency, and low complexity compared to state-of-the-art Resumable Data Compression (RDC) method, DEFLATE based on LZ77 & Huffman coding, and Lempel-Ziv-Markov Chain Algorithm (LZMA).

Optimal deep learning based image compression technique for data transmission on industrial Internet of things applications

AbstractRecently, industrial Internet of things becomes more popular and it involves a group of intelligent devices linked to create systems which observe, gather, communicate, and investigate data. In this view, the demand for compression techniques in remote sensing images is increasing since low complexity technique is required in spacecraft. Deep learning, for instance, convolutional neural network (CNN) has gained more attention in the domain of computer vision, particularly for high‐level applications like detection along with interpretation. At the same time, it is difficult to resolve the low‐level applications like image compression and it is investigated in this article. This article presents an optimal compression technique using CNNs for remote sensing images. The proposed method uses CNN for learning the compact representation of the original image which held the structural data and was then coded by Lempel Ziv Markov chain algorithm. Next, the encoded image was reconstructed to retrieve the original image with high reconstructed image quality. The proposed optimal compression technique is compatible with the available image codec standards. Wide range of experiments was carried out and the results were compared with binary tree and optimized truncation, JPEG, and JPEG2000 in terms of compression efficiency, reconstructed image quality, and space saving (SS). The obtained results apparently proved the effectiveness of the presented method, which attains an average peak signal to noise ratio of 49.90 dB and SS of 89.38%.

Highly Reliable and Low-Complexity Image Compression Scheme Using Neighborhood Correlation Sequence Algorithm in WSN

Recently, the advancements in the field of wireless technologies and micro-electro-mechanical systems lead to the development of potential applications in wireless sensor networks (WSNs). The visual sensors in WSN create a significant impact on computer vision based applications such as pattern recognition and image restoration. generate a massive quantity of multimedia data. Since transmission of images consumes more computational resources, various image compression techniques have been proposed. But, most of the existing image compression techniques are not applicable for sensor nodes due to its limitations on energy, bandwidth, memory, and processing capabilities. In this article, we introduce a highly reliable and low-complexity image compression scheme using neighborhood correlation sequence (NCS) algorithm. The NCS algorithm performs the bit reduction operation and then encoded by a codec (such as PPM, Deflate, and Lempel Ziv Markov chain algorithm.) to further compress the image. The proposed NCS algorithm increases the compression performance and decreases the energy utilization of the sensor nodes with high fidelity. Moreover, it achieved a minimum end to end delay of 1074.46 ms at the average bit rate of 4.40 bpp and peak signal to noise ratio of 48.06 on the applied test images. On comparing with state-of-art methods, the proposed method maintains a better tradeoff between compression efficiency and reconstructed image quality.

An evolutionary lion optimization algorithm‐based image compression technique for biomedical applications

AbstractRecently, medical image compression becomes essential to effectively handle large amounts of medical data for storage and communication purposes. Vector quantization (VQ) is a popular image compression technique, and the commonly used VQ model is Linde–Buzo–Gray (LBG) that constructs a local optimal codebook to compress images. The codebook construction was considered as an optimization problem, and a bioinspired algorithm was employed to solve it. This article proposed a VQ codebook construction approach called the L2‐LBG method utilizing the Lion optimization algorithm (LOA) and Lempel Ziv Markov chain Algorithm (LZMA). Once LOA constructed the codebook, LZMA was applied to compress the index table and further increase the compression performance of the LOA. A set of experimentation has been carried out using the benchmark medical images, and a comparative analysis was conducted with Cuckoo Search‐based LBG (CS‐LBG), Firefly‐based LBG (FF‐LBG) and JPEG2000. The compression efficiency of the presented model was validated in terms of compression ratio (CR), compression factor (CF), bit rate, and peak signal to noise ratio (PSNR). The proposed L2‐LBG method obtained a higher CR of 0.3425375 and PSNR value of 52.62459 compared to CS‐LBG, FA‐LBG, and JPEG2000 methods. The experimental values revealed that the L2‐LBG process yielded effective compression performance with a better‐quality reconstructed image.

Lossless Compression of Data From Static and Mobile Dynamic Vision Sensors-Performance and Trade-Offs

Dynamic Vision Sensors (DVS) are emerging retinomorphic visual capturing devices, with great advantages over conventional vision sensors in terms of wide dynamic range, low power consumption, and high temporal resolution. The bio-inspired approach of the DVS results in lower data rates than conventional vision sensors. Still, such data can be further compressed. Compression of DVS data is an emerging research area and a detailed performance comparison of different compression strategies for these data is still missing. This paper addresses lossless compression strategies for data output by neuromorphic visual sensors. We compare the performance of a number of strategies, including the only strategy developed specifically for such data and other more generic data compression strategies, tailored here to the case of neuromorphic data. We perform the comparison in terms of compression ratio, as well as compression and decompression speed and latency. Moreover, the compression performance analysis is performed under diverse scenarios including stationary and mobile DVS. According to the detailed experimental analysis, Lempel-Ziv-Markov chain algorithm (LZMA) achieves the best compression ratios among all the considered strategies for the case when the DVS is static. On the other hand, Spike coding achieves the best compression ratios under the scenario when spike events are produced by a sensor in motion. However, both strategies result in low compression speed and high latency which restrict the applications of these strategies in real-time scenarios. The Brotli strategy achieves the best trade-off between compression ratio, speed and latency under static as well as mobile scenarios. We also observe a significant decrease in compression and decompression performance (in terms of ratio, speed and latency) of all the strategies under mobile DVS scenarios.

Polynomial Data Compression for Large-Scale Physics Experiments

The next generation of research experiments will introduce a huge data surge to continuously increasing data production by current experiments. This data surge necessitates efficient compression techniques. These compression techniques must guarantee an optimum trade-off between compression rate and the corresponding ratio of compression to decompression speed without affecting the data integrity. This work presents a lossless compression algorithm to compress physics data generated by astronomy, astrophysics and particle physics experiments. The developed algorithms have been tuned and tested on a real-use case: the next-generation, ground-based, high-energy gamma ray observatory, Cherenkov Telescope Array, requiring important compression performance. As a stand-alone method, the proposed compression method is very fast and reasonably efficient. Alternatively, applied as a pre-compression algorithm, it can accelerate common methods like the Lempel–Ziv–Markov chain algorithm (LZMA), keeping close performance.

An improved lossless group compression algorithm for seismic data in SEG-Y and MiniSEED file formats

An improved lossless group compression algorithm is proposed for decreasing the size of SEG-Y files to relieve the enormous burden associated with the transmission and storage of large amounts of seismic exploration data. Because each data point is represented by 4 bytes in SEG-Y files, the file is broken down into 4 subgroups, and the Gini coefficient is employed to analyze the distribution of the overall data and each of the 4 data subgroups within the range [0,255]. The results show that each subgroup comprises characteristic frequency distributions suited to distinct compression algorithms. Therefore, the data of each subgroup was compressed using its best suited algorithm. After comparing the compression ratios obtained for each data subgroup using different algorithms, the Lempel-Ziv-Markov chain algorithm (LZMA) was selected for the compression of the first two subgroups and the Deflate algorithm for the latter two subgroups. The compression ratios and decompression times obtained with the improved algorithm were compared with those obtained with commonly employed compression algorithms for SEG-Y files with different sizes. The experimental results show that the improved algorithm provides a compression ratio of 75–80%, which is more effective than compression algorithms presently applied to SEG-Y files. In addition, the proposed algorithm is applied to the miniSEED format used in natural earthquake monitoring, and the results compared with those obtained using the Steim2 compression algorithm, the results again show that the proposed algorithm provides better data compression.

FPGA hardware implementation of the LZMA compression algorithm

Software-based LZMA( Lempel Ziv Markov-chain algorithm) nondestructive compression algorithm is very slow and consumes too much CPU( central processing unit) resources during the data compression process,as a result it can not meet the requirements of real-time systems. On the basis of the improved algorithm,a hardware accelerator for LZMA was designed with FPGA( field programmable gate array) implementation. The hardware accelerator is composed of LZ77 compressor,range encoder and send out controller.Ping-pong operation,parallel matching method with high performance,pipeline processing structure and other acceleration techniques were used to speed up LZMA compression algorithm. While at the same time,data compressed by the circuit is still compatible with standard LZMA file format. The compression rate of the circuit is speeded up to 125 Mb / s,nearly 10 times faster than that of the software based LZMA. The processing relative data of each clock is speeded up nearly 200 times. Results from the experiments on ML605 basing on a Virtex-6 FPGA development kit,show the accelerator is correct and feasible.

Implementation of LZMA compression algorithm on FPGA

In the era of big data, compression techniques are needed to improve data processing bandwidth and data storage efficiency. Compared with software-based compression techniques, hardware-based compression techniques can improve speed and reduce power consumption. LZMA (Lempel-Ziv-Markov chain algorithm) is a lossless compression technology, and its hardware implementation has broad application prospects. A novel high-performance implementation of the LZMA compression algorithm capable of processing up to 125 Mbit/s on a Virtex-6 field programmable gate array (FPGA) chip is proposed. A typical application and its compression performance for a specific data sample are then presented.

Backup and recovery mechanism of Web anti-tamper system based on LZMA and multi-version

Because the functions of backup and recovery are usually ignored in the Web anti-tamper system,a Web anti-tamper system model was proposed in this paper,based on which a safe and efficient remote backup and recovery system was given.The multi-version management technique of backup data was used for restoring the data of different versions and different periods.The Lempel-Ziv-Markov chain Algorithm(LZMA) was used to compress the backup data for improving the disk utilization.The Data Encryption Standard(DES) and File Transfer Protocol(FTP) were used for secure remote data storage and transmission.Finally,the system performance test shows it can effectively backup and recover the Web server data without affecting the Web server load and recover a single page within 100 ms.It is an effective means to solve the Web tampering problem.