Raw Data Compression Research Articles

Biostatistical Aspects of Whole Genome Sequencing Studies: Preprocessing and Quality Control.

Rapid advances in high-throughput DNA sequencing technologies have enabled large-scale whole genome sequencing (WGS) studies. Before performing association analysis between phenotypes and genotypes, preprocessing and quality control (QC) of the raw sequence data need to be performed. Because many biostatisticians have not been working with WGS data so far, we first sketch Illumina's short-read sequencing technology. Second, we explain the general preprocessing pipeline for WGS studies. Third, we provide an overview of important QC metrics, which are applied to WGS data: on the raw data, after mapping and alignment, after variant calling, and after multisample variant calling. Fourth, we illustrate the QC with the data from the GENEtic SequencIng Study Hamburg-Davos (GENESIS-HD), a study involving more than 9000 human whole genomes. All samples were sequenced on an Illumina NovaSeq 6000 with an average coverage of 35× using a PCR-free protocol. For QC, one genome in a bottle (GIAB) trio was sequenced in four replicates, and one GIAB sample was successfully sequenced 70 times in different runs. Fifth, we provide empirical data on the compression of raw data using the DRAGEN original read archive (ORA). The most important quality metrics in the application were genetic similarity, sample cross-contamination, deviations from the expected Het/Hom ratio, relatedness, and coverage. The compression ratio of the raw files using DRAGEN ORA was 5.6:1, and compression time was linear by genome coverage. In summary, the preprocessing, joint calling, and QC of large WGS studies are feasible within a reasonable time, and efficient QC procedures are readily available.

The O2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3

The O² software framework and GPU usage in ALICE online and offline reconstruction in Run 3

Performance-Optimized Quantization for SAR and InSAR Applications

For the design of present and next-generation spaceborne SAR missions, constantly increasing data rates are being demanded, which impose stringent requirements in terms of onboard memory and downlink capacity. In this scenario, the efficient quantization of SAR raw data is of primary importance, since the utilized compression rate is directly related to the volume of data to be stored and transmitted to the ground and, at the same time, it affects the resulting SAR imaging performance. In this paper, we introduce the performance-optimized block-adaptive quantization (PO-BAQ), a novel approach for SAR raw data compression which aims at optimizing the resource allocation and, at the same time, the quality of the resulting SAR and InSAR products. This goal is achieved by exploiting the a priori knowledge of the local SAR backscatter statistics, which allows for the generation of high-resolution bitrate maps that can be employed to fulfill a predefined performance requirement. Analyses on experimental TanDEM-X interferometric data are presented, which demonstrate the potentials of the proposed method as a helpful tool for performance budget definition and data rate optimization of present and future SAR missions.

A hybrid compression method for the NMR data based on window averaging and Discrete Cosine Transform

A large amount of raw echo data is collected by NMR logging, especially in multi-dimensional NMR logging, and a direct inversion for the raw echo data is time-consuming, so the echo data collected need to be compressed before inversion to improve inversion speed. This paper proposes a multi-dimensional NMR data compression method based on Discrete Cosine Transform (DCT). Utilizing the energy concentration characteristics of DCT, the low-frequency information of the echo data is retained to realize the compression of raw echo data. On this basis, a hybrid compression (WAC-DCT) method is proposed, which uses the window averaging method (WAC) to pre-compress the raw echo data, and then uses the DCT compression method to compress the pre-compressed echo data again to achieve a high compression rate. The compression effects of the WAC method, the FastSVD method, the WAC-SVD method, the DCT method and the WAC-DCT method are compared by numerical simulation. The results show that some fluid information is lost in the DCT compression process with a low compression rate. The FastSVD method, the WAC-SVD method and the WAC-DCT method all have the characteristic of a high compression rate. When the number of compressed echoes is very small, the WAC-DCT method has obvious advantage in T1 distribution. Compared with the case of without compression, the WAC-DCT method not only ensures a high compression rate, but also effectively reduces the inversion time, which has a good application effect in multi-dimensional NMR data compression.

A data reduction and compression description for high throughput time-resolved electron microscopy

Fast, direct electron detectors have significantly improved the spatio-temporal resolution of electron microscopy movies. Preserving both spatial and temporal resolution in extended observations, however, requires storing prohibitively large amounts of data. Here, we describe an efficient and flexible data reduction and compression scheme (ReCoDe) that retains both spatial and temporal resolution by preserving individual electron events. Running ReCoDe on a workstation we demonstrate on-the-fly reduction and compression of raw data streaming off a detector at 3 GB/s, for hours of uninterrupted data collection. The output was 100-fold smaller than the raw data and saved directly onto network-attached storage drives over a 10 GbE connection. We discuss calibration techniques that support electron detection and counting (e.g., estimate electron backscattering rates, false positive rates, and data compressibility), and novel data analysis methods enabled by ReCoDe (e.g., recalibration of data post acquisition, and accurate estimation of coincidence loss).

Impact of lossy compression of nanopore raw signal data on basecalling and consensus accuracy.

Nanopore sequencing provides a real-time and portable solution to genomic sequencing, enabling better assembly, structural variant discovery and modified base detection than second generation technologies. The sequencing process generates a huge amount of data in the form of raw signal contained in fast5 files, which must be compressed to enable efficient storage and transfer. Since the raw data is inherently noisy, lossy compression has potential to significantly reduce space requirements without adversely impacting performance of downstream applications. We explore the use of lossy compression for nanopore raw data using two state-of-the-art lossy time-series compressors, and evaluate the tradeoff between compressed size and basecalling/consensus accuracy. We test several basecallers and consensus tools on a variety of datasets at varying depths of coverage, and conclude that lossy compression can provide 35-50% further reduction in compressed size of raw data over the state-of-the-art lossless compressor with negligible impact on basecalling accuracy (≲0.2% reduction) and consensus accuracy (≲0.002% reduction). In addition, we evaluate the impact of lossy compression on methylation calling accuracy and observe that this impact is minimal for similar reductions in compressed size, although further evaluation with improved benchmark datasets is required for reaching a definite conclusion. The results suggest the possibility of using lossy compression, potentially on the nanopore sequencing device itself, to achieve significant reductions in storage and transmission costs while preserving the accuracy of downstream applications. The code is available at https://github.com/shubhamchandak94/lossy_compression_evaluation. Supplementary data are available at Bioinformatics online.

Quantization of dynamic speckle patterns with spatially varying statistics.

Raw data compression is mandatory for monitoring of processes by dynamic speckle analysis when two-dimensional activity maps are built by pointwise statistical processing of correlated speckle patterns formed on the surface of diffusely reflecting objects under laser illumination. Coarse quantization of speckle patterns enables storage and transfer of a huge amount of images, but it may be inefficient at spatially varying speckle statistics, such as for patterns recorded at non-uniform illumination or reflectivity. We prove efficacy of coarse quantization of the raw speckle data with varying statistics for a normalized algorithm by simulation and a polymer drop drying experiment. Both uniform and non-uniform quantization are proposed for treating such data. Decreasing the bit depth from 8 to 3 is possible without worsening the quality of the activity map.

Designing a GPU-parallel algorithm for raw SAR data compression: A focus on parallel performance estimation

When a Synthetic Aperture Radar (SAR) acquires raw data using a satellite or airborne platform, it must be transferred to the ground for further processing. For example, SAR raw data need a so-called ’focusing’ signal processing to render it into a visible image. Such processing is time and computing consuming, and it is commonly carried out in computing centres. Since the data transfer rate is a typical limitation when communicating with the ground station, compression is necessary to reduce transmission time. So far, this procedure has been implemented in application-specific hardware, but recent adoption of avionic computational GPUs opened to new high-performance onboard perspectives. Due to the limited availability of avionic GPUs, we focused on parallel performance estimation starting from measures relative to a similar off-the-shelf solution. In this paper, we present a GPU algorithm for raw SAR data compression, which uses 1-dimensional DCT transforms, followed by quantisation and entropy coding. We evaluate results using ENVISAT (Environmental Satellite) ASAR Image Mode level 0 data by measuring compression rates, statistical parameters, and distortion on decompressed and then focused images. Moreover, by evaluating the Algorithmic Overhead induced by the parallelisation strategy, we predict the best thread-block configuration for possible adoption of such a GPU algorithm on one of the most available avionic hardware.

Design and Implementation of Raw Data Compression System for Subsurface Detection SAR Based on FPGA

Synthetic aperture radar (SAR) has a wide range of applications in the field of remote sensing. Subject to the bandwidth and resource limitations of the onboard system, it is necessary to compress the raw data obtained from SAR. The traditional algorithms for the compression of SAR raw data result in loss of weak signals such as the essential subsurface signal in subsurface detection, leading to problems in practical applications. In order to solve the aforementioned problem, the present study proposes a novel SAR data compression algorithm (preserve weak signal based on block adaptive quantization) that aims to preserve the subsurface signals on the basis of the traditional block adaptive quantization algorithm and provides its corresponding field-programmable gate array implementation. The implementation of the system holds a certain application value with great advantages in the optimized resources and speed, the optional automatic working mode, and the strong scalability.

Improved BAQ algorithm for Tiangong‐2 interferometric imaging radar data compression

Tiangong-2 interferometric imaging radar altimeter (InIRA), the first space-borne radar altimeter in the world, applying a new mechanism of interferometry with short baseline and small incident angle, can achieve wide swath and high accuracy sea surface height measuring. Comparing to traditional altimeter, InIRA can obtain SAR image of observation area, which means InIRA needs higher AD sampling rate. Due to the limitation of satellite on-board storage and downlink speed, in order to extend the working time of InIRA, data compression algorithm for InIRA must be studied. Block adaptive quantisation (BAQ) is an efficient and widely adopted approach for space-borne SAR raw data compression. Based on the characters of InIRA echo waveform, this study discusses compressing InIRA raw data by changing quantisation bits and size of blocks of BAQ algorithm to achieve higher compression ratio and less sea surface height error.

Influence of the System MTF on the On-Board Lossless Compression of Hyperspectral Raw Data

A noticeable topic to be pursued in the field of on-board real-time data processing is the influence of the modulation transfer function (MTF) of the image acquisition system on the lossless compressibility of raw (that is, uncalibrated) hyperspectral data. Actually, notwithstanding the system device is constrained by several design and manufacturing requirements, the impact of the on-board MTF on the performance of data compressors is becoming remarkable. In particular, the aim of reducing both transmission bandwidth/power and mass storage can be efficiently pursued. Such an analysis is expected to be useful especially for systems employed in mini-satellites, whose payload must be compact and light. From this perspective, this paper investigates the performance of a typical imaging system that acquires low/medium-spatial-resolution images, by considering high-resolution reference data, which simulate the real scene to be imaged. To this end, standard Consultative Committee for Space Data Systems (CCSDS) Aviris 2006 data have been chosen, due to their spatial resolution of 17 m, which is adequate to be a reference for simulated data whose spatial resolution is foreseen between 50 and 150 m. MTF requirements are usually provided based on the cut-off value of the amplitude at the Nyquist frequency, which is defined as a half of the sampling frequency. Typically, a cut-off value between 0 . 2 and 0 . 3 ensures that a sufficient amount of information is delivered from the scene to the acquired image, by avoiding at the same time the degradation due to an excessive aliasing distortion. All the scores are achieved by running the standard lossless compression scheme CCSDS 1.2.3.0-B-1 for multispectral/hyperspectral data, as a function of the cut-off value and different noise acquisition levels. The final results, and related plots, show that this analysis can suggest a suitable choice for the cut-off value, to ensure both a sufficient quality and low bit rates for the transmitted data to the ground station.

Gpu-Based Online Track Reconstruction for the Alice Tpc in Run 3 With Continuous Read-Out

In LHC Run 3, ALICE will increase the data taking rate significantly to 50 kHz continuous read-out of minimum bias Pb—Pb collisions. The reconstruction strategy of the online-offline computing upgrade foresees a first synchronous online reconstruction stage during data taking enabling detector calibration and data compression, and a posterior calibrated asynchronous reconstruction stage. Many new challenges arise, among them continuous TPC read-out, more overlapping collisions, no a priori knowledge of the primary vertex and of location-dependent calibration in the synchronous phase, identification of low-momentum looping tracks, and sophisticated raw data compression. The tracking algorithm for the Time Projection Chamber (TPC) will be based on a Cellular Automaton and the Kalman filter. The reconstruction shall run online, processing 50 times more collisions per second than today, while yielding results comparable to current offline reconstruction. Our TPC track finding leverages the potential of hardware accelerators via the OpenCL and CUDA APIs in a shared source code for CPUs and GPUs for both reconstruction stages. We give an overview of the status of Run 3 tracking including performance on processors and GPUs and achieved compression ratios.

A framework for constraining image SNR loss due to MR raw data compression.

Computationally intensive image reconstruction algorithms can be used online during MRI exams by streaming data to remote high-performance computers. However, data acquisition rates often exceed the bandwidth of the available network resources creating a bottleneck. Data compression is, therefore, desired to ensure fast data transmission. The added noise variance due to compression was determined through statistical analysis for two compression libraries (one custom and one generic) that were implemented in this framework. Limiting the compression error variance relative to the measured thermal noise allowed for image signal-to-noise ratio loss to be explicitly constrained. Achievable compression ratios are dependent on image SNR, user-defined SNR loss tolerance, and acquisition type. However, a 1% reduction in SNR yields approximately four to ninefold compression ratios across MRI acquisition strategies. For free-breathing cine data reconstructed in the cloud, the streaming bandwidth was reduced from 37 to 6.1MB/s, alleviating the network transmission bottleneck. Our framework enabled data compression for online reconstructions and allowed SNR loss to be constrained based on a user-defined SNR tolerance. This practical tool will enable real-time data streaming and greater than fourfold faster cloud upload times.

A wavelet‐PCA method saves high mass resolution information in data treatment of SIMS molecular depth profiles

A detailed depth characterization of multilayered polymeric systems is a very attractive topic. Currently, the use of cluster primary ion beams in time‐of‐flight secondary ion mass spectrometry allows molecular depth profiling of organic and polymeric materials. Because typical raw data may contain thousands of peaks, the amount of information to manage grows rapidly and widely, so that data reduction techniques become indispensable in order to extract the most significant information from the given dataset. Here, we show how the wavelet‐based signal processing technique can be applied to the compression of the giant raw data acquired during time‐of‐flight secondary ion mass spectrometry molecular depth‐profiling experiments. We tested the approach on data acquired by analyzing a model sample consisting of polyelectrolyte‐based multilayers spin‐cast on silicon. Numerous wavelet mother functions and several compression levels were investigated. We propose some estimators of the filtering quality in order to find the highest ‘safe’ approximation value in terms of peaks area modification, signal to noise ratio, and mass resolution retention. The compression procedure allowed to obtain a dataset straightforwardly ‘manageable’ without any peak‐picking procedure or detailed peak integration. Moreover, we show that multivariate analysis, namely, principal component analysis, can be successfully combined to the results of the wavelet‐filtering, providing a simple and reliable method for extracting the relevant information from raw datasets. Copyright © 2016 John Wiley & Sons, Ltd.

Visually lossless compression for Bayer color filter array using optimized Vector Quantization

Devices using single sensors to capture colour images are cheaper due to high cost of Charge Couple Device (CCD) sensors or Complementary Metal-Oxide Semiconductor (CMOS) sensors. Single sensor devices use Colour Filter Array (CFA) to sample one colour band at every pixel location. Demosaicking process is applied to interpolate the two missing colours from the surrounding. Typically compression is done on the demosaicked images which may not be efficient due to the individual compression of the different colour space. This work investigated compression of raw data before demosaicking and performs demosaicking to reconstruct the R, G, B bands later. A novel Vector Quantization (VQ) technique for encoding the wavelet decomposed image using Modified Artificial Bee Colony (ABC) optimization algorithm is proposed. The proposed technique is compared with Genetic Algorithm based VQ and ABC based quantization and with standard LBG and Lloyd algorithm. Results show higher Peak Signal-to-Noise Ratio (PSNR) indicating better reconstruction.

Analysis-by-synthesis compression of range-focused SAR raw data

We propose a predictive compression scheme for synthetic aperture radar (SAR) raw data based on the analysis-by-synthesis encoding method. The algorithm is inspired by the code excited linear prediction (CELP) algorithm used in speech compression and exploits the correlation across azimuth of the range-focused SAR data. We also extend this approach to include multiple codebooks and obtain the performance of the proposed methods on recorded data. Numerical results show that these algorithms outperform the open-loop predictive quantization schemes.

Motion correction strategies for integrated PET/MR.

Integrated whole-body PET/MR facilitates the implementation of a broad variety of respiratory motion correction strategies, taking advantage of the strengths of both modalities. The goal of this study was the quantitative evaluation with clinical data of different MR- and PET-data-based motion correction strategies for integrated PET/MR. The PET and MR data of 20 patients were simultaneously acquired for 10 min on an integrated PET/MR system after administration of (18)F-FDG or (68)Ga-DOTANOC. Respiratory traces recorded with a bellows were compared against MR self-gating signals and signals extracted from PET raw data with the sensitivity method, by applying principal component analysis (PCA) or Laplacian eigenmaps and by using a novel variation combining the former and either of the latter two. Gated sinograms and MR images were generated accordingly, followed by image registration to derive MR motion models. Corrected PET images were reconstructed by incorporating this information into the reconstruction. An optical flow algorithm was applied for PET-based motion correction. Gating and motion correction were evaluated by quantitative analysis of apparent tracer uptake, lesion volume, displacement, contrast, and signal-to-noise ratio. The correlation between bellows- and MR-based signals was 0.63 ± 0.19, and that between MR and the sensitivity method was 0.52 ± 0.26. Depending on the PET raw-data compression, the average correlation between MR and PCA ranged from 0.25 ± 0.30 to 0.58 ± 0.33, and the range was 0.25 ± 0.30 to 0.42 ± 0.34 if Laplacian eigenmaps were applied. By combining the sensitivity method and PCA or Laplacian eigenmaps, the maximum average correlation to MR could be increased to 0.74 ± 0.21 and 0.70 ± 0.19, respectively. The selection of the best PET-based signal for each patient yielded an average correlation of 0.80 ± 0.13 with MR. Using the best PET-based respiratory signal for gating, mean tracer uptake increased by 17 ± 19% for gating, 13 ± 10% for MR-based motion correction, and 18 ± 15% for PET-based motion correction, compared with the static images. Lesion volumes were 76 ± 31%, 83 ± 18%, and 74 ± 22% of the sizes in the static images for gating, MR-based motion correction, and PET-based motion correction, respectively. Respiratory traces extracted from MR and PET data are comparable to those based on external sensors. The proposed PET-driven gating method improved respiratory signals and overall stability. Consistent results from MR- and PET-based correction methods enable more flexible PET/MR scan protocols while achieving higher PET image quality.

Study on an Algorithm for SAR Raw Data Compression

t is difficult to directly compress the raw data of synthetic aperture radar for its low relativity. In this paper, a new algorithm is put forward. Firstly range focusing is imposed to SAR raw data, which makes it have comparative high relativity, secondly a linear prediction is performed along the azimuth, lastly block adaptive quantization is used to the prediction difference series. The experiments manifest that with same bit rate, SQNR and SDNR of the algorithm proposed in this paper surpass that of BAQ algorithm. The calculation in this paper is far less than that of compression method after range focusing advised in corresponding reference. The algorithm proposed in this paper has a certain practical value.

Information-theoretic assessment of on-board near-lossless compression of hyperspectral data

A rate-distortion model to measure the impact of near-lossless compression of raw data, that is, compression with user-defined maximum absolute error, on the information available once the compressed data have been received and decompressed is proposed. Such a model requires the original uncompressed raw data and their measured noise variances. Advanced near-lossless methods are exploited only to measure the entropy of the datasets but are not required for on-board compression. In substance, the acquired raw data are regarded as a noisy realization of a noise-free spectral information source. The useful spectral information at the decoder is the mutual information between the unknown ideal source and the decoded source, which is affected by both instrument noise and compression-induced distortion. Experiments on simulated noisy images, in which the noise-free source and the noise realization are exactly known, show the trend of spectral information versus compression distortion, which in turn is related to the coded bit rate or equivalently to the compression ratio through the rate-distortion characteristic of the encoder used on satellite. Preliminary experiments on airborne visible infrared imaging spectrometer (AVIRIS) 2006 Yellowstone sequences match the trends of the simulations. The main conclusion that can be drawn is that the noisier the dataset, the lower the CR that can be tolerated, in order to save a prefixed amount of spectral information.

Development of a Global Backscatter Model in support to the Sentinel-1 mission design

The high data throughput of spaceborne Synthetic Aperture Radar (SAR) instruments introduces stringent requirements on the end-to-end system, from the on-board storage capacity and data downlink to processing power and archiving capacity in the ground segment. An efficient raw data compression scheme on-board the satellite is therefore required to maximise the return on investment for SAR missions. The next generation of European SAR satellites, the Sentinel-1 mission, will employ the Flexible Dynamic Block Adaptive Quantization (FDBAQ) compression scheme. In order to validate the implementation of the FDBAQ, expected radar reflectivity levels for the global land surface were needed.The paper presents the development of the Global Backscatter Model (GBM) which has been used to simulate realistic radar reflectivity acquisitions in support of performance validation of the Sentinel-1 FDBAQ implementation. The GBM consists of a parameter database and SAR image simulation software. The model parameter database characterises C-band radar reflectivity as a function of local incidence angle with a resolution of 1km for 85% of the global land surface. The parameter database was derived from a global multi-temporal coverage of ENVISAT Advanced Synthetic Aperture Radar (ASAR) data. The SAR image simulation software uses the parameter database together with orbit propagation and swath determination algorithms in order to account for the instrument acquisition geometry and the location specific backscattering characteristics. The model has the potential to simulate acquisitions for other SAR instruments. Furthermore, the parameter database could be a valuable source of information for global land cover studies.