Fast Compression Research Articles

ROIBIN-SZ: Fast and Science-Preserving Compression for Serial Crystallography

ROIBIN-SZ: Fast and Science-Preserving Compression for Serial Crystallography

Distributed ℋ 2 -Matrices for Boundary Element Methods

Standard discretization techniques for boundary integral equations, e.g., the Galerkin boundary element method, lead to large densely populated matrices that require fast and efficient compression techniques like the fast multipole method or hierarchical matrices. If the underlying mesh is very large, running the corresponding algorithms on a distributed computer is attractive, e.g., since distributed computers frequently are cost-effective and offer a high accumulated memory bandwidth. Compared to the closely related particle methods, for which distributed algorithms are well-established, the Galerkin discretization poses a challenge, since the supports of the basis functions influence the block structure of the matrix and therefore the flow of data in the corresponding algorithms. This article introduces distributed ℋ 2 -matrices, a class of hierarchical matrices that is closely related to fast multipole methods and particularly well-suited for distributed computing. While earlier efforts required the global tree structure of the ℋ 2 -matrix to be stored in every node of the distributed system, the new approach needs only local multilevel information that can be obtained via a simple distributed algorithm, allowing us to scale to significantly larger systems. Experiments show that this approach can handle very large meshes with more than 130 million triangles efficiently.

A MHz X-ray diffraction set-up for dynamic compression experiments in the diamond anvil cell.

An experimental platform for dynamic diamond anvil cell (dDAC) research has been developed at the High Energy Density (HED) Instrument at the European X-ray Free Electron Laser (European XFEL). Advantage was taken of the high repetition rate of the European XFEL (up to 4.5 MHz) to collect pulse-resolved MHz X-ray diffraction data from samples as they are dynamically compressed at intermediate strain rates (≤103 s-1), where up to 352 diffraction images can be collected from a single pulse train. The set-up employs piezo-driven dDACs capable of compressing samples in ≥340 µs, compatible with the maximum length of the pulse train (550 µs). Results from rapid compression experiments on a wide range of sample systems with different X-ray scattering powers are presented. A maximum compression rate of 87 TPa s-1 was observed during the fast compression of Au, while a strain rate of ∼1100 s-1 was achieved during the rapid compression of N2 at 23 TPa s-1.

BtrBlocks: Efficient Columnar Compression for Data Lakes

Analytics is moving to the cloud and data is moving into data lakes. These reside on object storage services like S3 and enable seamless data sharing and system interoperability. To support this, many systems build on open storage formats like Apache Parquet. However, these formats are not optimized for remotely-accessed data lakes and today's high-throughput networks. Inefficient decompression makes scans CPU-bound and thus increases query time and cost. With this work we present BtrBlocks, an open columnar storage format designed for data lakes. BtrBlocks uses a set of lightweight encoding schemes, achieving fast and efficient decompression and high compression ratios.

Enhancing genomic mutation data storage optimization based on the compression of asymmetry of sparsity.

Background: With the rapid development of high-throughput sequencing technology and the explosive growth of genomic data, storing, transmitting and processing massive amounts of data has become a new challenge. How to achieve fast lossless compression and decompression according to the characteristics of the data to speed up data transmission and processing requires research on relevant compression algorithms. Methods: In this paper, a compression algorithm for sparse asymmetric gene mutations (CA_SAGM) based on the characteristics of sparse genomic mutation data was proposed. The data was first sorted on a row-first basis so that neighboring non-zero elements were as close as possible to each other. The data were then renumbered using the reverse Cuthill-Mckee sorting technique. Finally the data were compressed into sparse row format (CSR) and stored. We had analyzed and compared the results of the CA_SAGM, coordinate format (COO) and compressed sparse column format (CSC) algorithms for sparse asymmetric genomic data. Nine types of single-nucleotide variation (SNV) data and six types of copy number variation (CNV) data from the TCGA database were used as the subjects of this study. Compression and decompression time, compression and decompression rate, compression memory and compression ratio were used as evaluation metrics. The correlation between each metric and the basic characteristics of the original data was further investigated. Results: The experimental results showed that the COO method had the shortest compression time, the fastest compression rate and the largest compression ratio, and had the best compression performance. CSC compression performance was the worst, and CA_SAGM compression performance was between the two. When decompressing the data, CA_SAGM performed the best, with the shortest decompression time and the fastest decompression rate. COO decompression performance was the worst. With increasing sparsity, the COO, CSC and CA_SAGM algorithms all exhibited longer compression and decompression times, lower compression and decompression rates, larger compression memory and lower compression ratios. When the sparsity was large, the compression memory and compression ratio of the three algorithms showed no difference characteristics, but the rest of the indexes were still different. Conclusion: CA_SAGM was an efficient compression algorithm that combines compression and decompression performance for sparse genomic mutation data.

Robust and superelastic spider web-like polyimide fiber-based conductive composite aerogel for extreme temperature-tolerant linear pressure sensor

Pressure sensors represent the cornerstone of artificial tactile sensing. Extensive research has been conducted on high-performance pressure sensors; however, the realization of high sensitivity, wide linear response range, and wide working temperature range remains a tremendous challenge. Here, triethylamine was innovatively applied to achieve the homogeneous dispersion of hydrophobic polyimide fibres (PIFs) in a carbon nanotube (CNT) aqueous dispersion without deteriorating the structure of the fibres, and a robust and superelastic spider web-like PIF/CNT conductive composite aerogel was developed using the freeze-drying and thermal imidization technique for a pressure sensor with a wide linear sensing range (0.01–53.34 kPa), ultralow detection limit (10 Pa), high sensitivity (0.507 kPa−1), fast response/recovery time (85/80 ms), stable fast compression response (500 mm min−1), and excellent cyclic fatigue resistance (5000 times). Finite element analysis indicates that the hierarchical fibrous network induces a significant linear variation in the contact area between adjacent conductive fibres upon external pressure and contributes to the excellent linear sensing capacity. The pressure sensor is demonstrated to be applicable in human physiology and motion signal detection, electronic skin, and intelligence control. Notably, it also exhibits amazing sensing stability and thermal insulation under extreme temperature conditions, demonstrating much promise for emerging applications such as the sensing unit for space suits and inflatable structures for future lunar/Mars habitats. This work provides a simple but powerful strategy for developing next-generation linear pressure sensors.

The Compression Optimality of Asymmetric Numeral Systems.

Source coding has a rich and long history. However, a recent explosion of multimedia Internet applications (such as teleconferencing and video streaming, for instance) renews interest in fast compression that also squeezes out as much redundancy as possible. In 2009 Jarek Duda invented his asymmetric numeral system (ANS). Apart from having a beautiful mathematical structure, it is very efficient and offers compression with a very low coding redundancy. ANS works well for any symbol source statistics, and it has become a preferred compression algorithm in the IT industry. However, designing an ANS instance requires a random selection of its symbol spread function. Consequently, each ANS instance offers compression with a slightly different compression ratio. The paper investigates the compression optimality of ANS. It shows that ANS is optimal for any symbol sources whose probability distribution is described by natural powers of 1/2. We use Markov chains to calculate ANS state probabilities. This allows us to precisely determine the ANS compression rate. We present two algorithms for finding ANS instances with a high compression ratio. The first explores state probability approximations in order to choose ANS instances with better compression ratios. The second algorithm is a probabilistic one. It finds ANS instances whose compression ratios can be made as close to the best ratio as required. This is done at the expense of the number θ of internal random "coin" tosses. The algorithm complexity is O(θL3), where L is the number of ANS states. The complexity can be reduced to O(θLlog2L) if we use a fast matrix inversion. If the algorithm is implemented on a quantum computer, its complexity becomes O(θ(log2L)3).

FitDepth: fast and lite 16-bit depth image compression algorithm

This article presents a fast parallel lossless technique and a lossy image compression technique for 16-bit single-channel images. Nowadays, such techniques are “a must” in robotics and other areas where several depth cameras are used. Since many of these algorithms need to be run in low-profile hardware, as embedded systems, they should be very fast and customizable. The proposal is based on the consideration of depth images as surfaces, so the idea is to split the image into a set of polynomial functions that each describes a part of the surface. The developed algorithm herein proposed can achieve a similar—or better—compression rate and especially higher speed rates than the existing techniques. It also has the potential of being fully parallelizable and to run on several cores. This feature, compared to other approaches, makes it useful for handling and streaming multiple cameras simultaneously. The algorithm is assessed in different situations and hardware. Its implementation is rather simple and is carried out with LIDAR captured images. Therefore, this work is accompanied by an open implementation in C++.

A new fast image compression–encryption scheme based on compressive sensing and parallel blocks

A new fast image compression–encryption scheme based on compressive sensing and parallel blocks

Methods of transient em data compression

A transient electromagnetic signal is recorded in an earth formation while drilling at multiple times. Data for subsequent interpretation is transmitted via a hydrochannel with a narrow bandwidth, for that reason the amount of transmitted information is limited. Therefore, data compression is an indispensable element of processing. It is important to determine what parameters and how many of them are necessary to describe the measured EMF curve. The idea is to transfer to the surface only significant parameters allowing to restore the recorded curve with a given accuracy. The paper proposes simple and fast data compression methods based on the spline approximation of the signal. Also, an orthogonal basis in the measurement space is used to describe each signal by a linear combination of vectors from this basis. We analyze transient curves and consider various magnetic field components. Numerical experiments show how many parameters are required to describe and reconstruct the signal of a deep-reading electromagnetic tool.

Dynamic Compression of a SiC Foam

Silicon carbide foam is a material that can be used as reinforcement of interpenetrated composites. This paper presents an analysis of such a foam subjected to low and fast compression. The analysis is performed using the peridynamics (PD) method. This approach allows for an evaluation of failure modes and such effects of microcracks nucleation, their growth, and, finally, fragmentation. Furthermore, the material appears to behave qualitatively and quantitatively differently while subjected to low- and high-speed steel piston movement. Under slow compression case, damage appears in the entire specimen, but the shape of the structure is not changing significantly, whereas during the fast compression the sample is dynamically fragmented.

The density distribution and physical origins of intermittency in supersonic, highly magnetized turbulence with diverse modes of driving

ABSTRACT The probability density function (PDF) of the logarithmic density contrast, s = ln (ρ/ρ0), with gas density ρ and mean density ρ0, for hydrodynamical supersonic turbulence is well known to have significant non-Gaussian (intermittent) features that monotonically increase with the turbulent Mach number, $\mathcal {M}$. By studying the mass- and volume-weighted s-PDF for an ensemble of 36 sub-to-trans-Alfv́enic mean-field, supersonic, isothermal turbulence simulations with different modes of driving, relevant to molecular gas in the cool interstellar medium, we show that a more intricate picture emerges for the non-Gaussian nature of s. Using four independent measures of the non-Gaussian components, we find hydrodynamical-like structure in the highly magnetized plasma for $\mathcal {M} \lesssim 4$. However, for $\mathcal {M} \gtrsim 4$, the non-Gaussian signatures disappear, leaving approximately Gaussian s-statistics – exactly the opposite of hydrodynamical turbulence in the high-$\mathcal {M}$ limit. We also find that the non-Gaussian components of the PDF increase monotonically with more compressive driving modes. To understand the $\mathcal {M} \lesssim 4$ non-Gaussian features, we use one-dimensional pencil beams to explore the dynamics along and across the large-scale magnetic field, $\mathrm{{\boldsymbol {\mathit {B}}}}_0$. We discuss kinetic, density, and magnetic field fluctuations from the pencil beams, and identify physical sources of non-Gaussian components to the PDF as single, strong shocks coupled to fast magnetosonic compressions that form along $\mathrm{{\boldsymbol {\mathit {B}}}}_0$. We discuss the Gaussianization of the $\mathcal {M} \gtrsim 4$s-fields through the lens of two phenomenologies: the self-similarity of the s-field and homogenization of the dynamical time-scales between the over- and underdense regions in the compressible gas.

Dynamic compression of Ce and Pr with millisecond time-resolved X-ray diffraction

Both cerium (Ce) and praseodymium (Pr) undergo a volume collapse transition under compression that originate from similar electronic mechanisms. Yet the outcome could not be more different. In the case of Ce with one affected 4f electron the volume collapse leaves the crystal symmetry intact, whereas for Pr with two 4f electrons the crystal symmetry changes from a distorted face centered cubic structure to a lower symmetry orthorhombic structure. In this paper, we present a study of the effect of strain/compression rate spanning nearly 4 orders of magnitude on the volume collapse phase transitions in Ce and Pr. These dynamic compression experiments in a diamond anvil cell also reveal kinetic differences between the phase transformations observed in these two materials. The transition cannot be overdriven in pressure in Ce, which indicates a fast kinetic process, whereas fast compression rates in Pr lead to a shift of the phase boundary to higher pressures, pointing to slower kinetics possibly due to the realization of a new crystal structure.

Universal Real-Time Adaptive Signal Compression for High-Frame-Rate Optoacoustic Tomography.

Optoacoustic tomography (OAT) has recently been advanced toward ultrafast volumetric imaging frame rates in the kilohertz range. As a result, excessive data processing and storage capacity requirements are increasingly being imposed on the imaging systems. OAT data commonly exhibit significant sparsity across the spatial, temporal or spectral domains, which facilitated the development of compressed sensing algorithms exploiting various sparse acquisition and under-sampling schemes to reduce data rates. However, performance of compressed sensing critically depends on a priori knowledge on the type of acquired data and/or imaged object, commonly resulting in lack of general applicability and unpredictable image quality. In this work, we report on a fast adaptive OAT data compression framework operating on fully sampled tomographic data. It is based on a wavelet packet transform that maximizes the data compression ratio according to the desired signal energy loss. A dedicated reconstruction method was further developed that efficiently renders images directly from the compressed data. Up to 1000x compression ratios were achieved while providing efficient control over the resulting image quality from arbitrary datasets exhibiting diverse spatial, temporal and spectral characteristics. Our approach enables faster and longer acquisitions and facilitates long-term storage of large OAT datasets.

The dynamic axial compression of FRC with high-speed translated θ-pinch plasma

A novel field reversed configuration (FRC) axial compression experiment with high-speed translated θ-pinch plasma was conducted in the Keda Mirror with AXisymmetry device. The translated north FRC was decelerated and compressed by an oncoming θ-pinch plasma stream, and the dynamic process was revealed by a 2D magnetic probe array. The FRC separatrix length is compressed to one-third of the initial value while the radius expands by ∼57%, resulting in the ∼16% increase in the electron temperature and ∼26% in the density, which matches the calculation from an adiabatic compression model. The good agreement is explained by the fast compression and particle supplementation owing to the compression with plasma. The results reported in this work may contribute to the understanding of electron heating in collision-merged FRC and provide a new compression method for the magneto-inertial fusion concept.

Mesoscopic attenuation mechanism in VTI seismic anisotropy

In this work, the effect of mesoscopic heterogeneity in the attenuation and scattering of waves propagating in poroelastic media is studied analytically and numerically. In case of heterogeneity size lower than the dominant wavelength, the medium composed of interbedded poroelastic layers of different lithology or by layers saturated with different fluids behaves as a homogeneous VTI medium. We developed frequency dependent expressions to estimate attenuation and phase velocity in such media. The results indicate that the behavior of the slow and fast compression waves depend on the frequency and direction of propagation. When an wave propagates parallel to the symmetry axis, attenuation of the rapid P wave becomes highest while the attenuation of the slow P wave becomes lowest. The behavior of the acoustic waves confirms that the slow P wave responds to the fluid flow associated with heterogeneities that generate pressure gradients on the borders that separate poroelastic media. The wave propagating in all directions manifests the mesoscopic mechanism at low frequency and the Biot's one at high ones. The dominant mechanism depends on the path of propagation and the degree of heterogeneity of the poroelastic compound.

Ring Cross-over Based GA for DFMB Chip Design and Medical Image Compression

: Currently, the clinical data stored in the cloud is easily accessible, and the patient’s data can be shared among treatment centers. In such a case, to handle additional data, the cloud data must be of a lesser scale. A process of compression was introduced to minimize the data with no losing data in order to achieve this size reduction. This paper conducts the experiment in two approaches: fast routing operations and compression from the chip in the DMFB approach. To apply this process of compression, the collected data from the chip was transformed into an image, and then compression of the image was performed utilizing a genetic algorithm (GA) based on a ring crossover. Consequently, the biochip of the 8x8 array is integrated into the power and area with the ring cross-module for an effective energy consumption operation. The technique of the process is utilized by the Microfluidic (MF) feature to handle and maintain the droplets. Also, the optimization process is performed by combining related pin actuation segments in parallel and the control pin to prevent pin-actuation conflicts. Through the optimization process, it synchronizes the length. This proposed approach decreases the consumption of the power and area. The outcome of the simulation indicates an increase in dynamic power, static power, and delay. Image compression is performed with the aid of this algorithm. In addition, for better outcomes, this GA compression application was contrasted with wavelet compressions.

A Comparative Study Using DCT, Delta Modulation, and Double Shift Coding for Compressing Electroencephalogram Data

Storing, transferring, and processing high-dimensional electroencephalogram (EGG) signals is a critical challenge. The goal of EEG compression is to remove redundant data in EEG signals. Medical signals like EEG must be of high quality for medical diagnosis. This paper uses a compression system with near-zero Mean Squared Error (MSE) based on Discrete Cosine Transform (DCT) and double shift coding for fast and efficient EEG data compression. This paper investigates and compares the use or non-use of delta modulation, which is applied to the transformed and quantized input signal. Double shift coding is applied after mapping the output to positive as a final step. The system performance is tested using EEG data files from the CHB-MIT Scalp EEG Database. Compression Ratio (CR) is used to evaluate the compression system performance. The results are encouraging when compared with previous works on the same data samples.

CMIC: an efficient quality score compressor with random access functionality

BackgroundOver the past few decades, the emergence and maturation of new technologies have substantially reduced the cost of genome sequencing. As a result, the amount of genomic data that needs to be stored and transmitted has grown exponentially. For the standard sequencing data format, FASTQ, compression of the quality score is a key and difficult aspect of FASTQ file compression. Throughout the literature, we found that the majority of the current quality score compression methods do not support random access. Based on the above consideration, it is reasonable to investigate a lossless quality score compressor with a high compression rate, a fast compression and decompression speed, and support for random access.ResultsIn this paper, we propose CMIC, an adaptive and random access supported compressor for lossless compression of quality score sequences. CMIC is an acronym of the four steps (classification, mapping, indexing and compression) in the paper. Its framework consists of the following four parts: classification, mapping, indexing, and compression. The experimental results show that our compressor has good performance in terms of compression rates on all the tested datasets. The file sizes are reduced by up to 21.91% when compared with LCQS. In terms of compression speed, CMIC is better than all other compressors on most of the tested cases. In terms of random access speed, the CMIC is faster than the LCQS, which provides a random access function for compressed quality scores.ConclusionsCMIC is a compressor that is especially designed for quality score sequences, which has good performance in terms of compression rate, compression speed, decompression speed, and random access speed. The CMIC can be obtained in the following way: https://github.com/Humonex/Cmic.

Compression rate of dynamic diamond anvil cells from room temperature to 10 K.

There is an ever increasing interest in studying dynamic-pressure dependent phenomena utilizing dynamic Diamond Anvil Cells (dDACs), devices capable of a highly controlled rate of compression. Here, we characterize and compare the compression rate of dDACs in which the compression is actuated via three different methods: (1) stepper motor (S-dDAC), (2) gas membrane (M-dDAC), and (3) piezoactuator (P-dDAC). The compression rates of these different types of dDAC were determined solely on millisecond time-resolved R1-line fluorescence of a ruby sphere located within the sample chamber. Furthermore, these different dynamic compression-techniques have been described and characterized over a broad temperature and pressure range from 10 to 300K and 0-50GPa. At room temperature, piezoactuation (P-dDAC) has a clear advantage in controlled extremely fast compression, having recorded a compression rate of ∼7 TPa/s, which is also found to be primarily influenced by the charging time of the piezostack. At 40-250K, gas membranes (M-dDAC) have also been found to generate rapid compression of ∼0.5-3 TPa/s and are readily interfaced with moderate cryogenic and ultrahigh vacuum conditions. Approaching more extreme cryogenic conditions (<10K), a stepper motor driven lever arm (S-dDAC) offers a solution for high-precision moderate compression rates in a regime where P-dDACs and M-dDACs can become difficult to incorporate. The results of this paper demonstrate the applicability of different dynamic compression techniques, and when applied, they can offer us new insights into matter's response to strain, which is highly relevant to physics, geoscience, and chemistry.