Data Compression Research Articles

Multifocal multiview imaging and data compression based on angular-focal-spatial representation.

Multifocal multiview (MFMV) is an emerging high-dimensional optical data that allows to record richer scene information but yields huge volumes of data. To unveil its imaging mechanism, we present an angular-focal-spatial representation model, which decomposes high-dimensional MFMV data into angular, spatial, and focal dimensions. To construct a comprehensive MFMV dataset, we leverage representative imaging prototypes, including digital camera imaging, emerging plenoptic refocusing, and synthesized Blender 3D creation. It is believed to be the first-of-its-kind MFMV dataset in multiple acquisition ways. To efficiently compress MFMV data, we propose the first, to our knowledge, MFMV data compression scheme based on angular-focal-spatial representation. It exploits inter-view, inter-stack, and intra-frame predictions to eliminate data redundancy in angular, focal, and spatial dimensions, respectively. Experiments demonstrate the proposed scheme outperforms the standard HEVC and MV-HEVC coding methods. As high as 3.693 dB PSNR gains and 64.22% bitrate savings can be achieved.

Prediction-based coding with rate control for lossless region of interest in pathology imaging

Online collaborative tools for medical diagnosis produced from digital pathology images have experimented an increase in demand in recent years. Due to the large sizes of pathology images, rate control (RC) techniques that allow an accurate control of compressed file sizes are critical to meet existing bandwidth restrictions while maximizing retrieved image quality. Recently, some RC contributions to Region of Interest (RoI) coding for pathology imaging have been presented. These encode the RoI without loss and the background with some loss, and focus on providing high RC accuracy for the background area. However, none of these RC contributions deal efficiently with arbitrary RoI shapes, which hinders the accuracy of background definition and rate control. This manuscript presents a novel coding system based on prediction with a novel RC algorithm for RoI coding that allows arbitrary RoIs shapes. Compared to other methods of the state of the art, our proposed algorithm significantly improves upon their RC accuracy, while reducing the compressed data rate for the RoI by 30%. Furthermore, it offers higher quality in the reconstructed background areas, which has been linked to better clinical performance by expert pathologists. Finally, the proposed method also allows lossless compression of both the RoI and the background, producing data volumes 14% lower than coding techniques included in DICOM, such as HEVC and JPEG-LS.

Scale perception data compression method with blockchain technology for UAV communication

The demand for high data throughput and security in Unmanned Aerial Vehicle (UAV) communication is rising. Efficient data transmission remains challenging due to limited wireless resources. This paper introduces a Scale Perception Compression Method that leverages compression and blockchain technology to optimize UAV communication. The method involves normalizing data into matrices, generating compression and decompression matrix codebook pairs, and selecting suitable pairs for small or large-scale matrices. Simulation results show that the error rate is associated with the distributions of the data and compression matrix. It remains relatively low when the data and compression matrix subjected to Uniform distribution compared to Gaussian distribution, regardless of the values of the number of rows m and columns r in compression matrix. Bubble method is more suitable to find out the proper matrix pair to compress and decompress transmitted data for small-scale codebook because it is stable, fast and results in lower difference between original and recovered data matrices. The same applies to Merge method for the large-scale codebook. The ER is always lower based on a small-scale codebook searching by Bubble method than that of on a large-scale codebook searching by Merge method, which indicates the former is sufficient for compression.

ПРИНЦИП СУБЪЕКТИВНОЙ ИЗБЫТОЧНОСТИ И МЕТОД СЖАТИЯ ДАННЫХ, ПОЛУЧАЕМЫХ ОТ ЦИФРОВЫХ АНТЕННЫХ РЕШЕТОК

The problem of limiting the capacity of the diagram-forming circuit of the digital antenna array is considered, due to the capabilities of existing communication lines with serial data transmission from the receiving channels to the computer. This problem limits the possibilities of implementing various methods of signal processing in digital antenna arrays. The subject of the study is the possibility of increasing the capacity of the diagram-forming circuit of the digital antenna array due to software compression of radar data. The purpose of the work is to create a scientific and methodological apparatus that will increase the capcity of the diagram-forming scheme by using the principle of subjective redundancy in relation to radar data processing tasks. The results of the work include: justification and formulation of the principle of subjective redundancy of radar data and the method of compression of radar data, as well as the results of numerical studies confirming the operability of the proposed scientific and methodological apparatus. The principle of subjective redundancy is based on the independence of the separation of signal processing processes in the spatial and temporal domains when receiving a single echo signal, and in the case of receiving several echo signals, on the principle of superposition of fields from various sources in the far antenna zone. The method of radar data compression is based on the principle of subjective redundancy of radar data and differs from the known iterative procedure for approximating the matrix of complex samples in the form of a superposition of products of one-dimensional cosine spectra. It is shown that the application of the radar data compression method makes it possible to increase the transmission capacity of the digital antenna array transmission lines several times.

Implementasi Kompresi File MP3 dengan Menerapkan Algoritma Levenstein

MP3 is an audio format that is often used because the data stored in MP3 format resembles the actual data when recorded.Storage space requirements will also rise if you keep a large number of MP3 files. Because of this, MP3 files must be compressed in order to prevent the storage space from filling up too quickly. This is because the compressed data will be lower in size. The Lossless compression method can be used to compress audio. The author uses the Levenstein Algorithm to implement audio compression in MP3 format. Levenstein's algorithm compresses data without any loss. When a file is compressed using lossless compression, the resulting file has the exact same size as the original file upon decompression. The author will be able to assess the effectiveness of MP3 file compression by using the Levenstein Algorithm to compress MP3 files. This will allow for faster transmission times and less data storage space consumption because the compressed MP3 file will shrink from its initial large size. After compression, the MP3 file's size—which was originally 128 bits—was reduced to 104 bits, yielding an 81.25% compression ratio. The purpose of this computation is to determine whether the compressed files are similar enough to be either decompressed or restored to their original size.

Communication Management and Data Compression Algorithm Design of BeiDou Transparent Transmission Terminal for Argo Buoy

The Argo buoy detects marine environmental data by making profile movements in the ocean and transmits the profile detection data to the shore base through the communication terminal. However, due to the large volume of data collected from profile detections and the continuous operation of the terminal, the remote communication of buoys is characterized by lengthy communication times and significant power consumption. A low-power Beidou transparent transmission terminal is designed to solve these problems in this paper. The terminal performs low-power operation management and power management for terminal remote communication. After the end of a communication process, the microcontroller turns the Beidou module power off and enters STOP mode. Before the next communication process begins, the serial port wakes up the microcontroller, which powers up the Beidou module. Before the remote communication, the microcontroller compacts the profile detection data collected by the buoy to reduce the quantity of remote communication data. In this paper, a variety of data compression methods are used to compare the compression rate, and the best compression method is selected according to the format characteristics of the data. The results show that the Beidou transparent transmission terminal of the Argo buoy can realize low power consumption for remote communication in ocean exploration. The terminal reduces power consumption by 77.282% per communication, and the average number of remote communications for each profile detection data are reduced by 55 times. The low-power Beidou transparent transmission terminal improves the battery life and is conducive to the long-term operation of the buoy.

NeuralBeds: Neural embeddings for efficient DNA data compression and optimized similarity search

The availability of high throughput sequencing tools coupled with the declining costs in the production of DNA sequences has led to the generation of enormous amounts of omics data curated in several databases such as NCBI and EMBL. Identification of similar DNA sequences from these databases is one of the fundamental tasks in bioinformatics. It is essential for discovering homologous sequences in organisms, phylogenetic studies of evolutionary relationships among several biological entities, or detection of pathogens. Improving DNA similarity search is of outmost importance because of the increased complexity of the evergrowing repositories of sequences. Therefore, instead of using the conventional approach of comparing raw sequences, e.g., in fasta format, a numerical representation of the sequences can be used to calculate their similarities and optimize the search process. In this study, we analyzed different approaches for numerical embeddings, including Chaos Game Representation, hashing, and neural networks, and compared them with classical approaches such as principal component analysis. It turned out that neural networks generate embeddings that are able to capture the similarity between DNA sequences as a distance measure and outperform the other approaches on DNA similarity search, significantly.

Reconstructing Disease Histories in Huge Discrete State Spaces

AbstractMany progressive diseases develop unnoticed and insidiously at the beginning. This leads to an observational gap, since the first data on the disease can only be obtained after diagnosis. Mutual Hazard Networks address this gap by reconstructing latent disease dynamics. They model the disease as a Markov chain on the space of all possible combinations of progression events. This space can be huge: Given a set of $$n\ge 266$$ n ≥ 266 events, its size exceeds the number of atoms in the universe. Mutual Hazard Networks combine time-to-event modeling with generalized probabilistic graphical models, regularization, and modern numerical tensor formats to enable efficient calculations in large state spaces using compressed data formats. Here we review Mutual Hazard Networks and put them in the context of machine learning theory. We describe how the Mutual Hazard assumption leads to a compact parameterization of the models and show how modern tensor formats allow for efficient computations in large state spaces. Finally, we show how Mutual Hazard Networks reconstruct the most likely history of glioblastomas.

JPEG-compatible Joint Image Compression and Encryption Algorithm with File Size Preservation

Joint image compression and encryption algorithms are intensively investigated due to their powerful capability of simultaneous image data compression and sensitive information protection. Unfortunately, most of the existing algorithms suffered from either poor compression efficiency or weak encryption strength, making them vulnerable to cryptanalysis. To address these limitations, we propose a chaos-based JPEG-compatible joint image compression and encryption algorithm. We separate the luminance and chrominance coefficients to preserve file size and encrypt the discrete cosine transform (DCT) coefficients in parallel. The proposed inter-block DC encryption strategy achieves high encryption intensity based on the permutation-substitution structure. In addition, we apply both inter- and intra-block permutations to AC coefficients and strengthen the encryption using an inter-block substitution for non-zero AC coefficients. The results of security and performance analyses demonstrate that the proposed algorithm offers robust encryption of image data while maintaining compression efficiency for real-time transmission.

Penerapan Teknologi Kompresi Data untuk Optimalisasi Manajemen Bahan Ajar Digital Pada Aplikasi Respository Menggunakan Personal Extreme Programming (PXP)

This research aims to create an educational environment integrated with technology through the implementation of educational technology, particularly by managing and storing digital learning device documents, known as a repository. The developed Repository System aims to optimize the management of storage for learning device documents within an educational institution. A case study was conducted at a Vocational High School (SMK). The management and storage system for learning device documents was developed using the Personal Extreme Programming (PXP) method, adopting an adaptive and flexible Agile approach. The repository system also applies data compression technology to optimize the storage capacity of learning device documents. Based on the results of data compression testing, the developed system proves to be suitable, achieving an average storage efficiency of 40.04% for each uploaded file. This research contributes to practical insights into the implementation of a repository system in an educational environment, particularly in Vocational High Schools, with a focus on curriculum transitions and the efficiency of managing learning device documents.

Investigation of virtual reality multimedia interaction technology based on wireless network

<p>In recent years, with the continuous development of wireless communication and virtual reality technology, multimedia interaction technology has received more and more attention. However, due to the limitations of bandwidth, delay, packet loss, and other problems in wireless networks, multimedia interaction technology also faces many challenges. In this paper, the virtual reality multimedia interaction technology will be studied in combination with a wireless network, in which the virtual environment is first constructed. Then, the interaction is carried out through data transmission and user interaction, and then the existing virtual reality multimedia interaction is optimized by using a data compression algorithm. In order to test the effectiveness and performance of virtual reality multimedia interaction under different network environments and determine the optimal network environment, this paper compares the multimedia interaction effect under three network environments using a wireless network, mobile network, and Bluetooth network as the research object, with download speed, loading speed, image quality, smoothness and real-time as the test variables, and 10 VR software as the constraints. The research results indicated that, under the same other conditions, taking real-time performance as an example, the delay time and feedback speed of wireless networks were between 26 ms–37 ms and 105 KB/s–115 KB/s. The delay time and feedback speed of mobile networks (3G/4G/5G, generation) were between 66 ms–75 ms and 46 KB/s–55 KB/s, while the delay time and feedback speed of Bluetooth networks were between 120 ms–130 ms and 25 KB/s–35 KB/s; this indicated that VR multimedia interaction technology in the wireless network had better performance. Virtual reality multimedia interaction technology based on wireless networks is a promising high technology that can bring users a more realistic, vivid, and intuitive communication and entertainment experience. At the same time, it will expand the application scenarios and promote the development and progress of the technology.</p>

Optimal data compression for Lyman-α forest cosmology

ABSTRACT The Lyman-α three-dimensional correlation functions have been widely used to perform cosmological inference using the baryon acoustic oscillation scale. While the traditional inference approach employs a data vector with several thousand data points, we apply near-maximal score compression down to tens of compressed data elements. We show that carefully constructed additional data beyond those linked to each inferred model parameter are required to preserve meaningful goodness of fit tests that guard against unknown systematics, and to avoid information loss due to non-linear parameter dependences. We demonstrate, on suites of realistic mocks and Data Release 16 data from the Extended Baryon Oscillation Spectroscopic Survey, that our compression approach is lossless and unbiased, yielding a posterior that is indistinguishable from that of the traditional analysis. As an early application, we investigate the impact of a covariance matrix estimated from a limited number of mocks, which is only well conditioned in compressed space.

Intrusion Detection in Internet of Things Systems: A Feature Extraction with Naive Bayes Classifier Approach

The Internet of Things (IoT) has proliferated, transitioning from modest home automation to encompass sectors like healthcare, agriculture, transportation, and manufacturing. This evolution is characterized by devices' ability to autonomously gather, disseminate, and analyze data, leading to improved real-time decision-making, predictive insights, and customized user experiences. The ubiquity of IoT, while promising, introduces significant data security concerns. The vast number of interlinked devices and diverse and often insufficient security features make them vulnerable to cyber threats, emphasizing the need for robust security mechanisms. Intrusion Detection Systems (IDS) have traditionally acted as vital guards against such threats; however, with the ever-increasing data in the IoT, traditional IDS models, such as Naive Bayes, face processing speed and accuracy challenges. This paper introduces a novel model, "FE+NB," which merges advanced Feature Extraction (FE) with the Naive Bayes (NB) classifier. Central to this model is the "Temporal-Structural Synthesis" technique tailored for IoT traffic data, focusing on data compression, temporal and structural analyses, and Feature Selection (FS) using mutual information. Consequently, the model enhances efficiency and accuracy in Intrusion Detection (ID) in complex IoT networks.

An anomaly detection method based on ship behavior trajectory

An anomalous trajectory detection method based on ship trajectory clustering and prediction is proposed. The method consists of two modules, namely, trajectory clustering based on improved DBSCAN and Trajectory prediction by ProbSparse Attention-based Transformer. we propose the concept of ship behavior similarity for the first time in this paper, by decomposing the velocity according to the ship's heading to get ‘velocity coordinates' to replace latitude and longitude coordinates, on the basis of which the ship behavior trajectories are clustered and the core trajectories are extracted. By comparing the behavioral similarity between the predicted trajectory and the core trajectory with the preset threshold, it is judged whether the abnormal trajectory or the ship has abnormal operation. AF-DP is used in this method for compression of trajectory data, and Fast-DTW is used to calculate the behavioral similarity between trajectories. Notably, this is the first application of ProbSparse Attention-based Transformer in the maritime domain. The proposed method is applicable to inland river traffic near Wusongkou, Shanghai. The final experimental results show that the method we proposed can effectively identify the abnormal behaviors in the trajectory, and can provide technical support for the analysis of ship track data and the risk management of maritime transport in this region.

GSC: efficient lossless compression of VCF files with fast query.

With the rise of large-scale genome sequencing projects, genotyping of thousands of samples has produced immense variant call format (VCF) files. It is becoming increasingly challenging to store, transfer, and analyze these voluminous files. Compression methods have been used to tackle these issues, aiming for both high compression ratio and fast random access. However, existing methods have not yet achieved a satisfactory compromise between these 2 objectives. To address the aforementioned issue, we introduce GSC (Genotype Sparse Compression), a specialized and refined lossless compression tool for VCF files. In benchmark tests conducted across various open-source datasets, GSC showcased exceptional performance in genotype data compression. Compared with the industry's most advanced tools (namely, GBC and GTC), GSC achieved compression ratios that were higher by 26.9% to 82.4% over GBC and GTC on the datasets, respectively. In lossless compression scenarios, GSC also demonstrated robust performance, with compression ratios 1.5× to 6.5× greater than general-purpose tools like gzip, zstd, and BCFtools-a mode not supported by either GBC or GTC. Achieving such high compression ratios did require some reasonable trade-offs, including longer decompression times, with GSC being 1.2× to 2× slower than GBC, yet 1.1× to 1.4× faster than GTC. Moreover, GSC maintained decompression query speeds that were equivalent to its competitors. In terms of RAM usage, GSC outperformed both counterparts. Overall, GSC's comprehensive performance surpasses that of the most advanced technologies. GSC balances high compression ratios with rapid data access, enhancing genomic data management. It supports seamless PLINK binary format conversion, simplifying downstream analysis.

A high-efficiency material deployment design method assisted with multiscale analysis

Efficient deployment of the right materials at the right place is a nontrivial problem. Structural components can achieve superior mechanical properties through tunable material deployment. However, such a structure-property-performance relationship is not explicit and requires the trial-and-error procedure. Discovering this relationship is extremely challenging due to a large number of possible material microstructures and deployed position combinations, and the search for the optimal design is computationally expensive. In this study, a data-driven approach is proposed to efficiently discover the underlying relationship between the deployment of a specific material microstructure and the final structural performance. The uniaxial tension of a perforated composite plate was used as a demonstrator. A high-fidelity representative volume element of a unidirectional fiber reinforced composite was constructed and utilized to calibrate the phenomenological constitutive model. A data compression algorithm was applied to the structure geometry in a finite element simulation to mechanically group similar regions and reduce dimensionality. After optimization, the calculated scenarios were selected based on the target performance. High-efficiency multiscale analysis was subsequently conducted on the selected conceptual designs to reveal the micro-deformation mechanism and interpret the interaction between the fibers and the matrix.

Compression of electrical code violation recognition data using the improved swinging door trending algorithm

Abstract Aiming at the challenge of storing massive power grid data, this paper proposes an improved swing gate trend algorithm to effectively compress 5G data. The algorithm first performs least squares smoothing on the original data to reduce noise interference on the SDT algorithm, which enables the data compression process to more accurately determine the data trend. Further, the shortcomings of the original SDT algorithm are improved, including adaptive frequency conversion data processing, dynamic threshold adjustment, and anomaly recording strategy, to enhance the practicality and efficiency of the algorithm. Through simulation analysis and example data validation, the study shows that the data compression ratio can be stabilized at about 23.98 when the data compression time reaches 1.6 minutes, and the actual error is very close to the desired error. The time overhead of the improved SDT algorithm is only 0.225 seconds, indicating that the algorithm is efficient and reliable. Combined with different data compression storage strategies, the algorithm can further reduce the data compression time. This study provides an adequate data compression method for electric code violation identification, which offers a practical solution for processing and storing large-scale grid data.

Information about the book by Joseph Dubrovkin “Data Compression IN Spectroscopy”, Cambridge Scholars Publishing. 2022, 355 PP. ISBN (10): 1-5275-8620-0 ISBN (13)

Widespread use of instrumental methods of analysis, in particular analytical spectroscopy, in recent years has led to increasing volume of information about the objects being studied. Experimental results are stored on a computer in the form of multidimensional digital data arrays that need to be processed in a special way, for example, to remove noise, visualize, analyze, and compress for efficient storage. For these purposes, there are theoretical methods and algorithms of data compression. At the same time compression (decompression) algorithms must ensure minimum distortion of the original signals for processing hyperspectral data. Large volume of data implies a comprehensive analysis of information. When working with multidimensional arrays, it is important to use special methods and technologies for processing, analyzing and interpreting hyperspectral data. This article provides information about the monograph “Data Compression in Spectroscopy” by Joseph Dubrovkin. The book was published by Cambridge Scholars Publishing. The book was written by an expert with extensive experience in the field of multivariate data analysis and chemometrics. The book consists of a preface, information about the structure of the book, a list of abbreviations and symbols, an introduction for each of 4 chapters, 8 appendices, a list of references and a subject index. A large number of examples and exercises are illustrated with MATLAB programs, and bibliographic tables clearly demonstrate the use of compression methods in industrial and research laboratories. The material in the book is discussed chapter by chapter. This modern monograph on data compression in spectroscopy will be useful as a teaching aid for students and teaching staff, as well as for specialists in analytical laboratories.

A New Method for Intelligent Multimedia Compression Based on Discrete Hartley Matrix

Multimedia data (video, audio, images) require storage space and transmission bandwidth when sent through social media networking. Despite rapid advances in the capabilities of digital communication systems, the high data size and data transfer bandwidth continue to exceed the capabilities of available technology, especially among social media users. The recent growth of multimedia-based web applications such as WhatsApp, Telegram, and Messenger has created a need for more efficient ways to compress media data. This is because the transmission speed of networks for multimedia data is relatively slow. In addition, there is a specific size for sending files via email or social networks, because much high-definition multimedia information can reach the Giga Byte size. Moreover, most smart cameras have high imaging resolution, which increases the bit rate of multimedia files of video, audio, and image. Therefore, the goal of data compression is to represent media (video, audio, images, etc.) as accurately as possible with the minimum number of bits (bit rate). Traditional data compression methods are complex for users. They require a high processing power for media data. This shows that most of the existing algorithms have loss in data during the process of compressing and decompressing data, with a high bitrate for media data (video, audio, and image). Therefore, this work describes a new method for media compression systems by discrete Hartley matrix (128) to get a high speed and low bit rate for compressing multimedia data. Finally, the results show that the proposed algorithm has a high-performance speed with a low bit rate for compression data, without losing any part of data (video, sound, and image). Furthermore, the majority of users of social media are satisfied with the data compression interactive system, with high performance and effectiveness in compressing multimedia data. This, in turn, will make it easier for users to easily send their files of video, audio, and images via social media networks.

Progressive Tree-Based Compression of Large-Scale Particle Data.

Scientific simulations and observations using particles have been creating large datasets that require effective and efficient data reduction to store, transfer, and analyze. However, current approaches either compress only small data well while being inefficient for large data, or handle large data but with insufficient compression. Toward effective and scalable compression/decompression of particle positions, we introduce new kinds of particle hierarchies and corresponding traversal orders that quickly reduce reconstruction error while being fast and low in memory footprint. Our solution to compression of large-scale particle data is a flexible block-based hierarchy that supports progressive, random-access, and error-driven decoding, where error estimation heuristics can be supplied by the user. For low-level node encoding, we introduce new schemes that effectively compress both uniform and densely structured particle distributions. Our proposed methods thus target all three phases of a tree-based particle compression pipeline, namely tree construction, tree traversal, and node encoding. The improved efficacy and flexibility of these methods over existing compressors are demonstrated through extensive experimentation, using a wide range of scientific particle datasets.