Effective Data Compression Research Articles

Data Compression in Adaptive and 4D Radiotherapy

Adaptive and 4D radiotherapy manipulates information from multiple fractions/phases. Such information includes 4DCT and daily CT, daily doses and deformation maps. Typically, Adaptive/4D radiotherapy involved hundreds time of data than conventional radiotherapy. Such huge data pose significant disk/database storage and network burden. An efficient and effective data compression method is essential to make the adaptive/4D radiotherapy clinically applicable. We developed a wavelet-based compression technique to handle the huge data from adaptive/4D radiotherapy. We use this technique to compress various dose images and deformation maps. The dose images are tomotherapy daily doses of the prostate, head & neck and lung patients. The deformation maps are calculated between the different phase of 4DCT image or between the daily image and the planning CT image using an intensity based deformable registration technique. For a dimension of 256 × 256 × 100, the dose image is about 30MB, while deformation maps are three times larger (90 MB). To evaluate the information lost through the lossy compression, we compare the compressed image with original image. The difference is the compression error. Two metrics are used, standard error and maximum error. Figure 1 plots the errors vs. the compression ratio for a typical dose image and a deformation map. For dose image, the errors are represented as the percentage of the maximum dose (Dmax). At 100:1 compression, the standard error is ∼0.05% Dmax, the maximum error is ∼0.5% Dmax. For deformation map, the errors are represented in the unit of voxels. At 100:1 compression, the standard error is ∼0.01 voxels, and the maximum error is ∼0.18 voxels. Both dose and deformation map can be easily compressed with 100:1 ratio without any clinically significant effect. Under this compression level, as for the dose image, the maximum error at ∼1% Dmax and the standard deviation of errors at ∼0.1% Dmax is well within the uncertainty of dose calculation and dose measurement. As for the deformation map, the maximum error at ∼0.2 voxel and standard deviation of errors at ∼0.01 voxel is well-acceptable since the deformation maps are calculated from voxel-based CT image and only have voxel precision. With 100:1 compression ratio, for an adaptive therapy of 40 fractions, all 40 deformation maps and daily doses would only take extra storage that is less than 3–4 times of the planning CT image, instead of 300–400 times if not compressed. In conclusion, the wavelet compression can achieve great compression ratio while maintain the image quality relevant to clinical applications. It will make those advanced treatment techniques like adaptive or 4D to be practical.

Wavelet Compression of ECG Signals Using SPIHT Algorithm

Abstract — In this paper we present a novel approach for waveletcompression of electrocardiogram (ECG) signals based on the set partitioning in hierarchical trees (SPIHT) coding algorithm. SPIHT algorithm has achieved prominent success in image compression. Here we use a modified version of SPIHT for one dimensional signals. We applied wavelet transform with SPIHT coding algorithmon different records of MIT-BIH database. The results show the high efficiency of this method in ECG compression. Keywords — ECG compression, wavelet, SPIHT. I. I NTRODUCTION LECTROCARDIOGRAM (ECG) signal is a very usefulsource of information for physicians in diagnosing heartabnormalities. With the increasing use of ECG in heart diagnosis, such as 24 hour monitoring or in ambulatorymonitoring systems, the volume of ECG data that should be stored or transmitted, has greatly increased. For example, a 3channel, 24 hour ambulatory ECG, typically has storagerequirement of over 50 MB. Therefore we need to reduce thedata volume to decrease storage cost or make ECG signalsuitable and ready for transmission through commoncommunication channels such as phone line or mobilechannel. So, we need an effective data compression method.The main goal of any compression technique is to achievemaximum data reduction while preserving the significantsignal morphology features upon reconstruction. Datacompression methods have been mainly divided into twomajor categories: 1) direct methods, in which actual signalsamples are analyzed (time domain), 2) transformationalmethods, in which first apply a transform to the signal and dospectral and energy distribution analysis of signals.Examples of direct methods are: differential pulse codemodulation (DPCM), amplitude zone time epoch coding (A TEC), turning point, coordinate reduction time encoding system (CORTES), Fan algorithm, ASEC. Reference [1] is agood review of some direct compression methods used inECG compression.

Efficient TPC data compression by track and cluster modeling

Current and future heavy ion experiments pose an increasing challenge with respect to the handling of the generated data rates. By introducing effective data compression techniques into the detector readout chain, the amount of events written to mass storage can be greatly enhanced thus improving the experimental conditions in terms of statistics. Based on a data modeling scheme highly adapted to Time Projection Chamber (TPC) data, we show that compression factors close to 10 are achievable.

Multivariate calibration of near infrared spectra by orthogonal WAVElet correction using a genetic algorithm

Orthogonal WAVElet correction (OWAVEC) is a pre-processing method aimed at simultaneously accomplishing two essential needs in multivariate calibration, signal correction and data compression, by combining the application of an orthogonal signal correction algorithm to remove information unrelated to a certain response with the great potential that wavelet analysis has shown for signal processing. In the previous version of the OWAVEC method, once the wavelet coefficients matrix had been computed from NIR spectra and deflated from irrelevant information in the orthogonalization step, effective data compression was achieved by selecting those largest correlation/variance wavelet coefficients serving as the basis for the development of a reliable regression model. This paper presents an evolution of the OWAVEC method, maintaining the first two stages in its application procedure (wavelet signal decomposition and direct orthogonalization) intact but incorporating genetic algorithms as a wavelet coefficients selection method to perform data compression and to improve the quality of the regression models developed later. Several specific applications dealing with diverse NIR regression problems are analyzed to evaluate the actual performance of the new OWAVEC method. Results provided by OWAVEC are also compared with those obtained with original data and with other orthogonal signal correction methods.

Generalization of OWAVEC method for simultaneous noise suppression, data compression and orthogonal signal correction

This paper presents modifications to our recently introduced pre-processing method, orthogonal WAVElet correction (OWAVEC), based on the combination of wavelet analysis and an orthogonal correction algorithm, described in detail in a former paper [I. Esteban-Díez, J.M. González-Sáiz, C. Pizarro, OWAVEC: a combination of wavelet analysis and an orthogonalization algorithm as a pre-processing step in multivariate calibration, Anal. Chim. Acta 515 (2004) 31–41], aimed at extending its applicability and at improving its performance; thanks to an additional use of OWAVEC as an effective data compression tool. The OWAVEC method uses the discrete wavelet transform (DWT) to decompose each individual signal into the wavelet domain, and then an orthogonalization algorithm is applied to the obtained wavelet coefficients matrix to remove the information not related to a considered response variable. Later, the corrected wavelet coefficients are ranked by their variance or by their correlation coefficient with the response variable, and the subset providing the most stable and reliable calibration model is finally selected (data compression). The new version of OWAVEC has been applied to two NIR data sets to test its performance. For both regression problems studied, high quality calibration models with very high compression ratios were obtained, providing improved predictive results and a considerably lower overfitting than other orthogonal signal correction methods. The generalized OWAVEC method presented here may be used as a global tool for simultaneous noise suppression, data compression and orthogonal correction of signals.

Constructing Three-Dimensional Multiple-Radar Reflectivity Mosaics: Examples of Convective Storms and Stratiform Rain Echoes

Abstract The advent of Internet-2 and effective data compression techniques facilitates the economic transmission of base-level radar data from the Weather Surveillance Radar-1988 Doppler (WSR-88D) network to users in real time. The native radar spherical coordinate system and large volume of data make the radar data processing a nontrivial task, especially when data from several radars are required to produce composite radar products. This paper investigates several approaches to remapping and combining multiple-radar reflectivity fields onto a unified 3D Cartesian grid with high spatial (≤1 km) and temporal (≤5 min) resolutions. The purpose of the study is to find an analysis approach that retains physical characteristics of the raw reflectivity data with minimum smoothing or introduction of analysis artifacts. Moreover, the approach needs to be highly efficient computationally for potential operational applications. The appropriate analysis can provide users with high-resolution reflectivity data that preserve the important features of the raw data, but in a manageable size with the advantage of a Cartesian coordinate system. Various interpolation schemes were evaluated and the results are presented here. It was found that a scheme combining a nearest-neighbor mapping on the range and azimuth plane and a linear interpolation in the elevation direction provides an efficient analysis scheme that retains high-resolution structure comparable to the raw data. A vertical interpolation is suited for analyses of convective-type echoes, while vertical and horizontal interpolations are needed for analyses of stratiform echoes, especially when large vertical reflectivity gradients exist. An automated brightband identification scheme is used to recognize stratiform echoes. When mosaicking multiple radars onto a common grid, a distance-weighted mean scheme can smooth possible discontinuities among radars due to calibration differences and can provide spatially consistent reflectivity mosaics. These schemes are computationally efficient due to their mathematical simplicity. Therefore, the 3D multiradar mosaic scheme can serve as a good candidate for providing high-spatial- and high-temporal-resolution base-level radar data in a Cartesian framework in real time.

TriMeDeC tool for preparing visual teaching materials based on triangular networks

AbstractTriangular meshes are frequently used to present different natural phenomena and free‐form shapes. They are generated by different mathematical, simulation, and design tools. Usually they contain a huge number of triangles, which represents a serious obstacle for storing, manipulation, and transmission via internet. However, triangular meshes can be simplified (decimated) yet still preserving the main features of the original mesh. In this paper, TriMeDeC tool for decimation, undecimation, and compression of triangular meshes is presented. The tool helps teachers and students to visualize triangular meshes and simplify them in an interactive way. Its salient feature is an effective data compression algorithm, which produces 2 to 3.5 times smaller output files than the popular WinZIP algorithm. TriMeDeC is very simple to use and can be easily employed by teachers and students. © 2002 Wiley Periodicals, Inc. Comput Appl Eng Educ 10: 144–154, 2002; Published online in Wiley InterScience (www.interscience.wiley.com); DOI 10.1002/cae.10031

PPM performance with BWT complexity: a fast and effective data compression algorithm

This paper introduces a new data compression algorithm. The goal underlying this new code design is to achieve a single lossless compression algorithm with the excellent compression ratios of the prediction by partial mapping (PPM) algorithms and the low complexity of codes based on the Burrows Wheeler Transform (BWT). Like the BWT-based codes, the proposed algorithm requires worst case O(n) computational complexity and memory; in contrast, the unbounded-context PPM algorithm, called PPM*, requires worst case O(n/sup 2/) computational complexity. Like PPM*, the proposed algorithm allows the use of unbounded contexts. Using standard data sets for comparison, the proposed algorithm achieves compression performance better than that of the BWT-based codes and comparable to that of PPM*. In particular, the proposed algorithm yields an average rate of 2.29 bits per character (bpc) on the Calgary corpus; this result compares favorably with the 2.33 and 2.34 bpc of PPM5 and PPM* (PPM algorithms), the 2.43 bpc of BW94 (the original BWT-based code), and the 3.64 and 2.69 bpc of compress and gzip (popular Unix compression algorithms based on Lempel-Ziv (LZ) coding techniques) on the same data set. The given code does not, however, match the best reported compression performance-2.12 bpc with PPMZ9-listed on the Calgary corpus results web page at the time of this publication. Results on the Canterbury corpus give a similar relative standing. The proposed algorithm gives an average rate of 2.15 bpc on the Canterbury corpus, while the Canterbury corpus web page gives average rates of 1.99 bpc for PPMZ9, 2.11 bpc for PPM5, 2.15 bpc for PPM7, 2.23 bpc for BZIP2 (a popular BWT-based code), and 3.31 and 2.53 bpc for compress and gzip, respectively.

A B-spline based method for data compression, process monitoring and diagnosis

Modern computerized monitoring and control systems in chemical plants result in vast amount of data being collected daily. This data contains valuable information about the process and can aid in effective decision making. Hence it has to be stored without distortion of the underlying trends, using effective data compression techniques, such that it can be readily accessible upon request. In this work, a dyadic B-Splines based data compression algorithm has been developed, which achieves data compression by denoising (removing noise) the data belonging to each sensor. This denoised and compressed data, in addition to being archived in a historical data base, is used in trend based process monitoring and diagnosis. The process monitoring algorithm can detect abnormal frequencies, identify changes in correlation among sensor variables and perform root cause analysis, thus reducing the number of alarms produced during abnormal situations. The application of the data compression algorithm and process monitoring algorithm is demonstrated on the data from a furnace in a crude distillation unit.

Using spatio-temporal chaos and intermediate-scale determinism to quantify spatially extended ecosystems

We discuss the question of how to quantify and analyse dynamics and patterns in spatially extended ecologies and introduce several new tools and ideas which use space-time dynamical structure. To illustrate our ideas, we introduce an artificial ecology model of a resource-predator-prey community which is interesting in its own right both ecologically and mathematically. This is a generalized probabilistic cellular automata. The model is stochastic and spatially non-homogeneous. We show how to identify a spatial scale intermediate between the noise dominated microscale and the infinite size limit at which non-trivial determinism is maximized. This is the scale at which to measure the system's dynamics. At this scale the population dynamics are essentially deterministic, low-dimensional and chaotic. This allows us to characterize the complex spatial patterns by a low-dimensional vector. This mapping from spatial patterns to low dimensional vectors provides effective and faithful data compression and is a powerful technique for synthesizing ecological information. It facilitates new analytical techniques. As an application we consider how to distinguish structural change within an ecosystem from natural dynamics. Such change is detected by using our parameterization to construct recurrence plots. Other applications such as the reconstruction of the dynamics of invisible species are discussed elsewhere.

Data compression using the deconvolution algorithm CLEAN

AbstractWe describe an application of the nonlinear deconvolution algorithm CLEAN in which a priori knowledge of the point‐spread function allows transmission of nonredundant information. We refer to this as CLEAN compression. The point‐spread function is viewed as a redundancy function. The data set may be regarded as a convolution of the nonredundant information with the redundancy function. Since the nonredundant data is a small subset of the overall data set, images or telecommunication messages may be transmitted over narrowband channels using CLEAN. Effective analog data compression is maximized; the analog signal may be significantly compressed with CLEAN even before any additional compression or digital encoding algorithms are applied.©1994 John Wiley & Sons Inc

An estimation of a model explaining the mechanism causing neurotic disorders using a theory of fuzzy sets.

Constructing and estimating a model to explain the mechanism of neurotic disorders is important and significant. The model helps the rearrangement or representation of knowledge obtained from professional physicians. However, it is a very difficult problem, because the objects requiring analysis are mental activities of human beings, and they originally include a comparatively large variance between individuals. The object data were obtained from patients with neurotic disorders who were diagnosed by several doctors for 10 years in a subagricultural areas in Japan. We analyzed the data and calculated the weights attached for personal information depending upon the similarity between the information and the kinds of neurotic disorders using the theory of fuzzy sets. From the results of our analysis, we constructed and estimated a model explaining the mechanism causing neurotic disorders as several linear equations. From data processing points of view, the estimation we attempted is placed in a kind of effective data compression with respect to discrete statistical data.

An interactive multivariate analysis of FCM data.

The procedure and results of the interactive multivariate analysis of FCM data are described. Using principal-components analysis, cluster analysis, and interactive maneuvers, this procedure facilitates an effective data compression from a four-dimensional space into two-dimensional space, then allows cluster separation. The procedure is especially effective for separating clusters, which are degenerated in the usual scattergrams. Programs were mostly written in C language on MS-DOS and were tested on four-dimensional analysis of the blood cells, which resulted in a successful separation of the degenerated clusters.

ディジタルマップのポイント・リダクションに関する研究-II

Of primary importance in the use of digital maps is to find an effective data compression method that permits the data to be stored comactly and reproduced easily because digital maps normaly consist of a large number of points. In this paper, a new method is proposed for data compression of coastline data in digital maps based on the best approximation to cubic spline functions via a dynamic programming approach and the influence on the shape of the reproduced coastlines are compared when different degree of continuous conditions, continuity only and continuity up to the first derivative, are imposed on the spline functions.

Digital pre-processing techniques of landsat data in high relief area—an Himalayan example

The analysis of the Landsat data for a Himalayan Test area recorded with a low sun elevation has shown that most of the information is due to illumination and shadow effects, which can be satisfactorily separated from the effects of the ground-cover on the radiance levels, by using the PC analysis and the Euclidean Distance and Directional Cosine techniques. At the same time effective data compression is achieved without loss of information. For a visual interpretation, both the images of “relief” and “groundcover” are necessary. Proper location, using drainage divides and channel network may be done on the “relief” image. Vegetation boundaries appear on the other image. Comparison of groundcover images of September 1972 and May 1974 indicates the utility of these these techniques for sequential studies, particularly the separation of deciduous and non-deciduous vegetation in a broad manner.

Simple data compression by redundancy replacement : E. E. Wallingford and I. Alleslev. Royal Military Coll. of Canada, Electrical Engng. Dept., (1101-7.Canada.203) T72-00716 REPRINT-3441 (1971) 4 pp

Abstract : A redundacny replacement system is described. Whenever m bits of an n-bit word are identical with corresponding bits in the preceding word, these bits are replaced by m bits from a second source. An extra bit is added to each word to signal the substitution. If words with m redundant bits occur with probability P(m), one gains an extra channel of average capacity mP(m) - 1. The system is analyzed as a data compressor that enhances the source entropy by replacing redundancies. An effective data compression of 2.11 is obtained for typical voice sources. (Author)

Simple data compression by redundancy replacement (Corresp.)

A redundancy replacement system is described. Whenever <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</tex> bits of an <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> -bit word are identical with corresponding bits in the preceding word, these bits are replaced by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</tex> bits from a second source. An extra bit is added to each word to signal the substitution. If words with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</tex> redundant bits occur with probability <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">P(m)</tex> , one gains an extra channel of average capacity <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mP(m) -- 1</tex> . The system is analyzed as a data compressor that enhances the source entropy by replacing redundancies. An effective data compression of 2.11 is obtained for typical voice sources.