Efficient Data Reduction Method Research Articles

Incremental conic functions algorithm for large scale classification problems

In order to cope with classification problems involving large datasets, we propose a new mathematical programming algorithm by extending the clustering based polyhedral conic functions approach. Despite the high classification efficiency of polyhedral conic functions, the realization previously required a nested implementation of k-means and conic function generation, which has a computational load related to the number of data points. In the proposed algorithm, an efficient data reduction method is employed to the k-means phase prior to the conic function generation step. The new method not only improves the computational efficiency of the successful conic function classifier, but also helps avoiding model over-fitting by giving fewer (but more representative) conic functions.

An efficient data reduction method and its application to cluster analysis

Data reduction plays a very important role in the data mining field, but the existing methods have not been able to efficiently identify all major features which are hidden in the large datasets. On some occasions, they even cause the loss of the original key features. In this paper, a new efficient measure was developed to reduce a given dataset and to uncover the major features by multiplying the defined absolute density with the defined local density of any data. These two kinds of densities were estimated with a fast grid-based bisecting method. To test its performance on feature reduction and sample reduction, a group of feature-different synthetic datasets and 24 benchmark datasets were used as examples and the clustering accuracy, runtime and separability among clusters were used as measurements. The results strongly proved the proposed method could fast reduce a dataset and identify the most important key features. Additionally, it also can effectively determine the optimal number of clusters by suppressing the noisy data and enhancing the separation among clusters.

Principal component analysis-based data reduction model for wireless sensor networks

Wireless sensor networks (WSNs) are widely used in monitoring environmental and physical conditions, such as temperature, vibration, humidity, light and voltage. However, the high dimension of sensed data, especially in multivariate sensor applications, increases the power consumption in transmitting this data to the base station and hence shortens the lifetime of sensors. Therefore, efficient data reduction methods are needed to minimise the power consumption in data transmission. In this paper, an efficient model for multivariate data reduction is proposed based on the principal component analysis (PCA). The performance of the model was evaluated using Intel Berkeley Research Lab (IBRL) dataset. The experimental results show the advantages of the proposed model as it allows 50% reduction rate and 96% approximation accuracy after reduction. A comparison with an existing model shows the superiority of the proposed model in terms of approximation accuracy as the reconstruction error is always smaller for different datasets.

A Principal Components-Based Clustering Method to Identify Variants Associated with Complex Traits

Background: Multivariate methods ranging from joint SNP to principal components analysis (PCA) have been developed for testing multiple markers in a region for association with disease and disease-related traits. However, these methods suffer from low power and/or the inability to identify the subset of markers contributing to evidence for association under various scenarios. Methods: We introduce orthoblique principal components-based clustering (OPCC) as an alternative approach to identify specific subsets of markers showing association with a quantitative outcome of interest. We demonstrate the utility of OPCC using simulation studies and an example from the literature on type 2 diabetes. Results: Compared to traditional methods, OPCC has similar or improved power under various scenarios of linkage disequilibrium structure and genotype availability. Most importantly, our simulations show how OPCC accurately parses large numbers of markers to a subset containing the causal variant or its proxy. Conclusion: OPCC is a powerful and efficient data reduction method for detecting associations between gene variants and disease-related traits. Unlike alternative methodologies, OPCC has the ability to isolate the effect of causal SNP(s) from among large sets of markers in a candidate region. Therefore, OPCC is an improvement over PCA for testing multiple SNP associations with phenotypes of interest.

A Versatile Program for the Analysis of Electromyographic Data

Ergonomists and researchers often utilize electromyographic (EMG) recordings to produce estimates of muscular work load in occupational exposure assessment. Prolonged measurements, possible with recent technological advances in portable data acquisition and compact memory storage technologies, require efficient data reduction methods not always available in commercial software packages. The Iowa EMG Analysis Program (IEAP) was created to provide researchers the means to incorporate multiple processing techniques suited for the analysis of prolonged EMG measurements. IEAP currently includes subroutines to calculate the amplitude probability distribution function, exposure variation analysis, clustered exposure variation analysis, and gaps analysis profiles for up to four channels of root-mean-square processed EMG data. Data management functions include the creation of customized hypertext markup language (HTML) documentation and text files able to seamlessly incorporate analysis results into existing statistical software packages. IEAP is a powerful EMG analysis tool ideally suited for ergonomists and researchers involved in occupational ergonomics studies.

Methodology and empirical results of program behaviour measurements

Program behaviour characteristics were examined using data gathered from real program executions. Experiments were performed in a segmented virtual memory with a working set policy; the analyzing costs were kept low using an efficient data reduction method. Empirical results were obtained concerning the influence of the window size on program behaviour characteristics, the accuracy of some average working set size approximations and the sensitivity of program behaviour to the program's input data. These results show that some commonly used assumptions concerning program behaviour are inaccurate. Also there seem to exist “ill-behaving” programs, the behaviour of which does not correspond well with results obtained earlier. The effects of real-time delays during program execution were considered using a new simple method. As an additional experiment, segmenting and paging were compared using various performance statistics; the results seem to favour segmenting.