High Data Compression Research Articles

Discrete Illumination-Based Compressed Ultrafast Photography for High-Fidelity Dynamic Imaging.

Compressed ultrafast photography (CUP) can capture irreversible or difficult-to-repeat dynamic scenes at the imaging speed of more than one billion frames per second, which is obtained by compressive sensing-based image reconstruction from a compressed 2D image through the discretization of detector pixels. However, an excessively high data compression ratio in CUP severely degrades the image reconstruction quality, thereby restricting its ability to observe ultrafast dynamic scenes with complex spatial structures. To address this issue, a discrete illumination-based CUP (DI-CUP) with high fidelity is reported. In DI-CUP, the dynamic scenes are loaded into an ultrashort laser pulse train with controllable sub-pulse number and time interval, thus the data compression ratio, as well as the overlap between adjacent frames, is greatly decreased and flexibly controlled through the discretization of dynamic scenes based on laser pulse train illumination, and high-fidelity image reconstruction can be realized within the same observation time window. Furthermore, the superior performance of DI-CUP is verified by observing femtosecond laser-induced ablation dynamics and plasma channel evolution, which are hardly resolved in the spatial structures using conventional CUP. It is anticipated that DI-CUP will be widely and dependably used in the real-time observations of various ultrafast dynamics.

An efficient deep learning-based workflow for real-time CO2 plume visualization in saline aquifer using distributed pressure and temperature measurements

Underground carbon dioxide (CO2) sequestration is widely accepted as a proven and established technology to respond to global warming from greenhouse gas emissions. It is crucial to monitor the CO2 plume effectively throughout the life cycle of a geologic CO2 sequestration project to ensure safety and storage efficacy. We propose a fast and efficient deep learning workflow for near real-time data assimilation, forecasting and visualization of CO2 plume evolution in saline aquifer and demonstrate its application at a field site.Our proposed workflow integrates field measurements, including distributed pressure and temperature at wells to visualize spatial and temporal migration of the CO2 plume in the subsurface. The deep learning framework has two key concepts that enhance the efficiency of the training and robustness of the CO2 plume prediction: first, instead of using multiple saturation images as output of the deep-learning model, a single Time of Flight (TOF) image are used to represent CO2 plume propagation; second, we use variational autoencoder-decoder (VAE) for high dimensional image data compression considering uncertainties in the predicted images. Time-of-Flight (TOF) is the travel time of a neural particle from the injection point, which is calculated along streamline based on the velocity field by considering reservoir heterogeneity and driving forces. The latent variables of VAE are estimated through a deep neural network model with inputs of distributed pressure and temperature measurements. Subsequently, the decoder part of the VAE expands the estimated latent variables to the original dimension of the TOF images. We demonstrate the efficacy of the proposed method by comparison with the more traditional pareto-based multi-objective Genetic Algorithm (MOGA) for the assimilation of the field measurements and forecasting of the CO2 plume migration.The power and utility of our proposed workflow are demonstrated by application to the Illinois Basin-Decatur Project (IBDP). The IBDP is a large scale 3-year CO2 storage test and is part of the Midwest Regional Carbon Sequestration Partnership (MRCSP). The field measurements include bottom-hole pressure and distributed temperature sensing (DTS) data at the injection well, and the distributed pressure measurements at several locations along the monitoring well. The dynamic model is a compositional model with 1.7 million cells. First, we identify the influential parameters using sensitivity analysis based on the pressure and temperature responses. Next, the sensitive parameters are sampled from a pre-specified range and a comprehensive training dataset is generated, including the pressure and temperature measurements and TOF images based on flow simulation results from a variety of geological model realizations. The trained neural network model can identify geological model realizations that capture the spatial and temporal migration of pressure and CO2 saturation based on Euclidean distance within the VAE latent space. For comparison with traditional history matching workflow, the selected sensitive parameters are also tuned using MOGA with the pressure and temperature measurements. A reasonable agreement is obtained for the predicted CO2 plume migration between two workflows. The proposed deep learning workflow shows significant speed-up compared to the traditional multi-objective optimization-based history matching workflow (5 h for training and seconds for prediction; compared to several days/weeks for the MOGA).The novelty of this work is the application of an efficient deep learning-based workflow to a large-scale CO2 storage test site for assimilating field measurements and predicting CO2 plume propagation in a near real-time and comparison with traditional history matching approach.

Robust multitask compressive sampling via deep generative models for crack detection in structural health monitoring

In structural health monitoring (SHM), there is an increasing demand for real-time image-based damage detection. Such a technology is essential for minimizing hazard loss caused by delayed emergency response after earthquakes or other natural disasters, or service interruption during structural inspection. Compressive sampling (CS) is a promising solution to achieve such a goal by greatly reducing the power consumption on high-resolution image transmission when using wireless devices. However, conventional CS failed to achieve high enough compression ratios, while existing generative-model-based CS requires laboriously training a high-quality generator with many large-scale images. To overcome such a bottleneck that hinders the practical use of CS in SHM, we propose a multitask CS algorithm that only relies on existing generators trained by low-pixel crack images. By exploiting the new discovery that similar crack images share a similar sparsity pattern in their latent vectors mapped by the generator, our algorithm achieves higher crack detection accuracy and robustness within a much shorter time when using a high data compression ratio. We verify the effectiveness of the proposed CS algorithm using synthetic and real image data. The results demonstrate that this work has moved a step closer toward successful implementation of operational CS-based crack detection systems in real-time SHM.

Construction of a reduced-order model based on tensor decomposition and its application to airbag deployment simulations

We present a construction method for reduced-order models (ROMs) to explore alternatives to numerical simulations. The proposed method can efficiently construct ROMs for non-linear problems with contact and impact behaviors by using tensor decomposition for factorizing multidimensional data and Akima-spline interpolation without tuning any parameters. First, we construct learning tensor data of nodal displacements or accelerations using finite element analysis with some representative parameter sets. Second, the data are decomposed into a set of mode matrices and one small core tensor using Tucker decomposition. Third, Akima-spline interpolation is applied to the mode matrices to predict values within the data range. Finally, the time history responses with new parameter sets are generated by multiplying the expanded mode matrices and small core tensor. The performance of the proposed method is studied by constructing ROMs for airbag impact simulations based on limited learning data. The proposed ROMs can accurately predict airbag deployment behavior even for new parameter sets using the Akima-spline interpolation scheme. Furthermore, an extremely high data compression ratio (more than 1000) and efficient predictions of the response surfaces and Pareto frontier (2000 times faster than that of full finite element analyses using all parameter sets) can be realized.

Empirical optimization of energy bin weights for compressing measurements with realistic photon counting x-ray detectors.

Photon counting detectors (PCDs) provide higher spatial resolution, improved contrast-to-noise ratio (CNR), and energy discriminating capabilities. However, the greatly increased amount of projection data in photon counting computed tomography (PCCT) systems becomes challenging to transmit through the slip ring, process, and store. This study proposes and evaluates an empirical optimization algorithm to obtain optimal energy weights for energy bin data compression. This algorithm is universally applicable to spectral imaging tasks including 2 and 3 material decomposition (MD) tasks and virtual monoenergetic images (VMIs). This method is simple to implement while preserving spectral information for the full range of object thicknesses and is applicable to different PCDs, for example, silicon detectors and CdTe detectors. We used realistic detector energy response models to simulate the spectral response of different PCDs and an empirical calibration method to fit a semi-empirical forward model for each PCD. We numerically optimized the optimal energy weights by minimizing the average relative Cramér-Rao lower bound (CRLB) due to the energy-weighted bin compression, for MD and VMI tasks over a range of material area density (0-40g/cm2 water, 0-2.16g/cm2 calcium). We used Monte Carlo simulation of a step wedge phantom and an anthropomorphic head phantom to evaluate the performance of this energy bin compression method in the projection domain and image domain, respectively. The results show that for 2 MD, the energy bin compression method can reduce PCCT data size by 75% and 60%, with an average variance penalty of less than 17% and 3% for silicon and CdTe detectors, respectively. For 3 MD tasks with a K-edge material (iodine), this method can reduce the data size by 62.5% and 40% with an average variance penalty of less than 12% and 13% for silicon and CdTe detectors, respectively. We proposed an energy bin compression method that is broadly applicable to different PCCT systems and object sizes, with high data compression ratio and little loss of spectral information.

Compression of Pulsed Infrared Thermography Data With Unsupervised Learning for Nondestructive Evaluation of Additively Manufactured Metals

Additive manufacturing (AM) of high-strength metals, which is typically based on laser powder bed fusion (LPBF), can introduce microscopic pores in the AM metal. Pulsed Infrared Thermography (PIT) offers several advantages for nondestructive imaging of subsurface defects in AM structures because the method is one-sided, non-contact and scalable to structures of arbitrary size. However, high-resolution PIT imaging results in the generation of a large volume of thermography data (~TB), which creates challenges for the storage and transmission of data. Compression of thermography data requires an approach that achieves high data compression ratio while preserving weak thermal features corresponding to microscopic material defects. We investigate thermography data compression using several unsupervised learning (UL) algorithms, which include Principal Component Analysis (PCA), Independent Component Analysis (ICA), Exploratory Factor Analysis (EFA), Sparse Dictionary Learning (SDL), and a novel lightweight Thermography Compressive Sparse Autoencoder (TCSA). Algorithms are benchmarked using PIT experimental data obtained from imaging of a stainless steel plate with calibrated porosity defects imprinted with AM process. For all algorithms, we obtain compression ratio >30 (highest compression of 46 is achieved with TCSA), and peak signal-to-noise ratio for reconstruction accuracy >73dB. Compared to existing methods, advantages of UL algorithms include achieving high compression ratio while preserving weak features to allow extraction of microscopic material defects from images. UL-based methods have general applicability because they are adaptable to compression of different data types, and allow for memory-efficient training and rapid on-line augmentation of the model.

Thermal-aware Test Data Compression for System-on-Chip Based on Modified Bitmask Based Methods

High temperature during test mode and the large volume of test data are the two prominent challenges in the testing of System-on-Chip (SoC). Temperature relies on the spatial power distribution between the blocks of the chip. An efficient don’t care filling technique is proposed to minimize the non-uniform spatial power distribution, which, in turn, reduces the peak temperature of the chip. However, high test data compression can be obtained by carefully mapping the don’t care bits to get more similar subvectors in the precomputed test patterns. The don’t care bits in the given test set can be utilized for test data compression and peak temperature reduction. As the same don’t care bits are to be used for both peak temperature reduction and test data compression, the two techniques conflict with each other. An integrated approach is presented to keep peak temperature under the safe limit with low test compression loss. Experimental results on ISCAS’89 benchmark circuits demonstrate the effectiveness of the proposed approach

The Energy-Aware Matrix Completion-Based Data Gathering Scheme for Wireless Sensor Networks

In the Wireless Sensor Networks (WSNs), ensuring long-term survival of the sensor devices is crucial, especially for non-energy harvesting networks where the sensors have to deal with the available limited power. Thus, there is a huge need to efficiently select, in each time-slot, a small set of source nodes to monitor the network area and deliver their data to the sink. Note that there is a trade-off between energy efficiency, achieved through data-compression, and the informative quality received by the sink. Moreover, although applying a high data compression ratio extremely reduces the overall network energy consumption, the network lifetime is not necessarily extended due to the uneven energy depletion of the nodes' batteries. To this end, in this paper, we propose the Energy-Aware Matrix Completion based data gathering approach (EAMC), which designates the active nodes according to their residual energy levels. To collect data readings, the proposed EAMC relies on a nodes clustering phase and a MC based data sampling. Then, the interpolation of all the missing data is performed by the sink thanks to a Three-stage MC based recovery framework. Since we are interested in high data loss scenarios, the limited amount of delivered data must be sufficient in terms of informative quality it holds in order to reach a satisfactory recovery accuracy for the entire data. Hence, the EAMC selects the nodes depending on their inter-correlation as well as the network energy efficiency, with the use of a combined energy-aware and correlation-based metric. This introduced active node cost function changes with the type of application one wants to perform with the intention to reach a longer lifespan for the network. Therewith, the numerical results show that the EAMC achieves an attractive and competitive trade-off between the data reconstruction quality and the network lifetime for all the investigated scenarios.

Data Uploading Strategy for Underwater Wireless Sensor Networks.

Underwater wireless sensor networks (UWSNs) have become a popular research topic due to the challenges of underwater communication. The existing mechanisms for collecting data from UWSNs focus on reducing the data redundancy and communication energy consumption, while ignoring the problem of energy-saving transmission after compression. In order to improve the efficiency of data collection, we propose a data uploading decision-making strategy based on the high similarity of the collected data and the energy consumption of the high similarity data compression. This decision-making strategy efficiently optimizes the energy consumption of the networks. By analyzing the data similarity, the quality of network communication, and uploading energy consumption, the decision-making strategy provides an energy-efficient data upload strategy for underwater nodes, which reduces the energy consumption in various network settings. The simulation results show that compared with several existing data compression and uploading methods, the proposed data upload methods has better energy saving effect in different network scenarios.

Innovative Methodology of On-Line Point Cloud Data Compression for Free-Form Surface Scanning Measurement

In order to obtain a highly accurate profile of a measured three-dimensional (3D) free-form surface, a scanning measuring device has to produce extremely dense point cloud data with a great sampling rate. Bottlenecks are created owing to inefficiencies in manipulating, storing and transferring these data, and parametric modelling from them is quite time-consuming work. In order to effectively compress the dense point cloud data obtained from a 3D free-form surface during the real-time scanning measuring process, this paper presents an innovative methodology of an on-line point cloud data compression algorithm for 3D free-form surface scanning measurement. It has the ability to identify and eliminate data redundancy caused by geometric feature similarity between adjacent scanning layers. At first, the new algorithm adopts the bi-Akima method to compress the initial point cloud data; next, the data redundancy existing in the compressed point cloud is further identified and eliminated; then, we can get the final compressed point cloud data. Finally, the experiment is conducted, and the results demonstrate that the proposed algorithm is capable of obtaining high-quality data compression results with higher data compression ratios than other existing on-line point cloud data compression/reduction methods.

Cross-layer access control in publish/subscribe middleware over software-defined networks

When technologies of software-defined networks (SDNs) provide a chance to improve the quality of service (QoS) of publish/subscribe middlewares, new chances are also arising for adversaries to attack the networks and the middlewares. We here propose a cross-layer access control solution to protect the publish/subscribe middleware over SDNs. Applications over a publish/subscribe middleware interact by an indirect, anonymous and multicast event communication paradigm, where we hope that the applications, the middleware, and the underlying network collaborate to realize the access control of reading/writing events. The key issue is how to use the flow matching capability of SDN switches to efficiently and securely enforce complex authorization policies that include multiple conjunction and disjunction structures. It is required to resist against the collusion attacks of SDN controllers and subscribers when the middleware/network is partially delegated to enforce the authorization policies of publishers. In our cross-layer solution, a policy representation method is presented to encode authorization policies into flow entries with high data compression and security, and a two-party computation method is presented to carry out secret sharing for defeating malicious SDN controllers and subscribers. Finally, our solution is evaluated to show its effectiveness.

Tensor-based predictive control for extremely large-scale single conjugate adaptive optics.

In this paper we propose a data-driven predictive control algorithm for large-scale single conjugate adaptive optics systems. At each time sample, the Shack-Hartmann wavefront sensor signal sampled on a spatial grid of size N×N is reshuffled into a d-dimensional tensor. Its spatial-temporal dynamics are modeled with a d-dimensional autoregressive model of temporal order p, where each tensor storing past data undergoes a multilinear transformation by factor matrices of small sizes. Equivalently, the vector form of this autoregressive model features coefficient matrices parametrized with a sum of Kronecker products between d-factor matrices. We propose an Alternating Least Squares algorithm for identifying the factor matrices from open-loop sensor data. When modeling each coefficient matrix with a sum of r terms, the computational complexity for updating the sensor prediction online reduces from O(pN4) in the unstructured matrix case to O(prd N2(d+1)d). Most importantly, this model structure breaks away from assuming any prior spatial-temporal coupling as it is discovered from the data. The algorithm is validated on a laboratory testbed that demonstrates the ability to accurately decompose the coefficient matrices of large-scale autoregressive models with a tensor-based representation, hence achieving high data compression rates and reducing the temporal error especially for a large Greenwood per sample frequency ratio.

Deep Surface Light Fields

A surface light field represents the radiance of rays originating from any points on the surface in any directions. Traditional approaches require ultra-dense sampling to ensure the rendering quality. In this paper, we present a novel neural network based technique called deep surface light field or DSLF to use only moderate sampling for high fidelity rendering. DSLF automatically fills in the missing data by leveraging different sampling patterns across the vertices and at the same time eliminates redundancies due to the network's prediction capability. For real data, we address the image registration problem as well as conduct texture-aware remeshing for aligning texture edges with vertices to avoid blurring. Comprehensive experiments show that DSLF can further achieve high data compression ratio while facilitating real-time rendering on the GPU.

ECG data compression using a neural network model based on multi-objective optimization.

Electrocardiogram (ECG) data analysis is of great significance to the diagnosis of cardiovascular disease. ECG compression should be processed in real time, and the data should be based on lossless compression and have high predictability. In terms of the real time aspect, short-time Fourier transformation is applied to the processing of signal wave for reducing computational time. For the lossless compression requirement, wavelet-transformation that is a coding algorithm can be used to avoid loss of data. In practice, compression is required to avoid storing redundant recording data that are not useful in the diagnosis platform. The obtained data can be preprocessed to remove noise by using wavelet transform, and then a multi-objective optimize neural network model is used to extract feature information. Compared with the existing traditional methods such as direct data processing method and transform method, our proposed compression model has self-learning ability to achieve high data compression ratio at 1:19 without losing important ECG information and compromising quality. Upon testing, we demonstrated that the proposed ECG data compression method based on multi-objective optimization neural network is effective and efficient in clinical practice.

Enhancing Test Compression With Dependency Analysis for Multiple Expansion Ratios

Scan test data compression is widely used in industry to reduce test data volume (TDV) and test application time (TAT). This paper shows how multiple scan chain expansion ratios can help to obtain high test data compression in system-on-chips. Scan chains are partitioned with a higher expansion ratio than normal in scan compression mode and then are gradually concatenated based on a cost function to detect any faults that could not be detected at the higher expansion ratios. It improves the overall test compression ratio since it potentially allows faults to be detected at the highest expansion ratio. This paper introduces a new cost function to choose scan chain concatenation candidates for concatenation for multiple expansion ratios. To avoid TDV and TAT increase by scan concatenation, the proposed method takes a logic structure and scan chain length into consideration. Experiment results show the proposed method reduces TAT and TDV by 53%–64% compared with a traditional scan compression method.

Neurological activity monitoring based on video inpainting.

Inpainting-based compression and reconstruction methodology can be applied to systems with limited resources to enable continuously monitor neurological activity. In this work, an approach based on sparse representations and K-SVD is augmented to a video processing in order to improve the recovery quality. That was mainly achieved by using another direction of spatial correlation and the extraction of cuboids across frames. The implementation of overlapping frames between the recorded data blocks avoids rising errors at the boundaries during the inpainting-based recovery. Controlling the electrode states per frame plays a key role for high data compression and precise recovery. The proposed 3D inpainting approach can compete with common methods like JPEG, JPEG2000 or MPEG-4 in terms of the degree of compression and reconstruction accuracy, which was applied on real measured local field potentials of a human patient.

Kronecker-ARX models in identifying (2D) spatial-temporal systems

In this paper we address the identification of (2D) spatial-temporal dynamical systems governed by the Vector Auto-Regressive (VAR) form. The coefficient-matrices of the VAR model are parametrized as sums of Kronecker products. When the number of terms in the sum is small compared to the size of the matrix, such a Kronecker representation leads to high data compression. Estimating in least-squares sense the coefficient-matrices gives rise to a bilinear estimation problem, which is tackled using a three-stage algorithm. A numerical example demonstrates the advantages of the new modeling paradigm. It leads to comparable performances with the unstructured least-squares estimation of VAR models. However, the number of parameters in the new modeling paradigm grows linearly w.r.t. the number of nodes in the 2D sensor network instead of quadratically in the full unstructured matrix case.

Robust reconstruction of a signal from its unthresholded recurrence plot subject to disturbances

To make valid inferences from recurrence plots for time-delay embedded signals, two underlying key questions are: (1) to what extent does an unthresholded recurrence (URP) plot carry the same information as the signal that generated it, and (2) how does the information change when the URP gets distorted. We studied the first question in our earlier work [1], where it was shown that the URP admits the reconstruction of the underlying signal (up to its mean and a choice of sign) if and only if an associated graph is connected. Here we refine this result and we give an explicit condition in terms of the embedding parameters and the discrete Fourier spectrum of the URP. We also develop a method for the reconstruction of the underlying signal which overcomes several drawbacks that earlier approaches had. To address the second question we investigate robustness of the proposed reconstruction method under disturbances. We carry out two simulation experiments which are characterized by a broad band and a narrow band spectrum respectively. For each experiment we choose a noise level and two different pairs of embedding parameters. The conventional binary recurrence plot (RP) is obtained from the URP by thresholding and zero-one conversion, which can be viewed as severe distortion acting on the URP. Typically the reconstruction of the underlying signal from an RP is expected to be rather inaccurate. However, by introducing the concept of a multi-level recurrence plot (MRP) we propose to bridge the information gap between the URP and the RP, while still achieving a high data compression rate. We demonstrate the working of the proposed reconstruction procedure on MRPs, indicating that MRPs with just a few discretization levels can usually capture signal properties and morphologies more accurately than conventional RPs.

Temperature and data size trade-off in dictionary based test data compression

This paper proposes a new thermal-aware test data compression technique using dictionary-based coding. Large test data volume and rise in chip temperature during a test, are the two major challenges for the test engineers. Efficient filling of the don't care bits of the test patterns minimises the transition count in the scan chains, which in turn reduces the temperature of the chip to a large extent. On the other hand, high test data compression can be achieved by filling the don't-cares in a manner to get more similar sub-vectors within the test vectors. Although, both the problems rely on don't-care bit filling, most of the existing works have considered them separately. Moreover, it has been observed that, in general, thermal-aware don't-care filling leads to poor test compression, while a compression-aware don't-care filling produces high temperature. However, a high test compression with low-temperature is the most desirable expectation. In our work, we have combined the temperature reduction and the test data compression into a single problem and solved it. We have presented an integrated approach that can perform a trade-off between temperature and test compression. Experimental results on ISCAS'89, ITC'99 and IWLS'05 benchmarks show the flexibility of the proposed method to achieve a balance between temperature and test compression. The work has further been extended to reduce temperature without sacrificing compression.

Lazy Management for Frequency Table on Hardware-Based Stream Lossless Data Compression

The demand for communicating large amounts of data in real-time has raised new challenges with implementing high-speed communication paths for high definition video and sensory data. It requires the implementation of high speed data paths based on hardware. Implementation difficulties have to be addressed by applying new techniques based on data-oriented algorithms. This paper focuses on a solution for this problem by applying a lossless data compression mechanism on the communication data path. The new lossless data compression mechanism, called LCA-DLT, provides dynamic histogram management for symbol lookup tables used in the compression and the decompression operations. When the histogram memory is fully used, the management algorithm needs to find the least used entries and invalidate these entries. The invalidation operations cause the blocking of the compression and the decompression data stream. This paper proposes novel techniques to eliminate blocking by introducing a dynamic invalidation mechanism, which allows achievement of a high throughput data compression.