Data Pre-processing Algorithm Research Articles

Data-driven multi-time-step ahead daily rainfall forecasting using singular spectrum analysis-based data pre-processing

Abstract Accurate forecasting of rainfall, especially daily time-step rainfall, remains a challenging task for hydrologists' invariance with the existence of several deterministic, stochastic and data-driven models. Several researchers have fine-tuned the hydrological models by using pre-processed input data but improvement rate in prediction of daily time-step rainfall data is not up to the expected level. There are still chances to improve the accuracy of rainfall predictions with an efficient data pre-processing algorithm. Singular spectrum analysis (SSA) is one such technique found to be a very successful data pre-processing algorithm. In the past, the artificial neural network (ANN) model emerged as one of the most successful data-driven techniques in hydrology because of its ability to capture non-linearity and a wide variety of algorithms. This study aims at assessing the advantage of using SSA as a pre-processing algorithm in ANN models. It also compares the performance of a simple ANN model with SSA-ANN model in forecasting single time-step as well as multi-time-step (3-day and 7-day) ahead daily rainfall time series pertaining to Koyna watershed, India. The model performance measures show that data pre-processing using SSA has enhanced the performance of ANN models both in single as well as multi-time-step ahead daily rainfall prediction.

Product quality improvement method in manufacturing process based on kernel optimisation algorithm

Quality data in manufacture process has the features of mixed type, uneven distribution, dimension curse and data coupling. To apply the massive manufacturing quality data effectively to the quality analysis of the manufacture enterprise, the data pre-processing algorithm based on equivalence relation is employed to select the characteristic of hybrid data and preprocess data. KML-SVM (Optimised kernel-based hybrid manifold learning and support vector machines algorithm) is proposed. KML is adopted to solve the problems of manufacturing process quality data dimension curse. SVM is adopted to classify and predict low-dimensional embedded data, as well as to optimise support vector machine kernel function so that the classification accuracy can be maximised. The actual manufacturing process data of AVIC Shenyang Liming Aero-Engine Group Corporation Ltd is demonstrated to simulate and verify the proposed algorithm.

Antenna cross-polarization correction for spaceborne polarimetric microwave correlation radiometer (I): Derivation of the antenna temperature equation

Spaceborne polarimetric microwave correlation radiometer is a new type of instrument for the passive microwave remote sensing from space, which can provide an important way for remote sensing of sea surface wind vector and other ocean atmosphere environmental parameters. Antenna cross-polarization correction is an important part of the data pre-processing algorithm. In this paper, for the demand of antenna cross-polarization correction in spaceborne polarimetric microwave correlation radiometer, we have independently derived the full polarization antenna temperature equation based on the polarization coherent detection theory and the definition of Stokes parameters. The equation for the four parameters of Stokes antenna temperature introduced the cross polarization amplitude and phase between the Stokes parameters. Besides, the influence of the polarization rotation angle on the antenna pattern computation was considered. Finally, we established a geometric correspondence between the antenna scanning beam and the earth scene. The principle for determining the parameters of the antenna temperature equation was also discussed. Stokes antenna temperature equation laid the foundation for antenna cross-polarization correction for spaceborne polarimetric microwave correlation radiometer in the future.

Unsupervised Analysis of Classical Biomedical Markers: Robustness and Medical Relevance of Patient Clustering Using Bioinformatics Tools

MotivationIt has been proposed that clustering clinical markers, such as blood test results, can be used to stratify patients. However, the robustness of clusters formed with this approach to data pre-processing and clustering algorithm choices has not been evaluated, nor has clustering reproducibility. Here, we made use of the NHANES survey to compare clusters generated with various combinations of pre-processing and clustering algorithms, and tested their reproducibility in two separate samples.MethodValues of 44 biomarkers and 19 health/life style traits were extracted from the National Health and Nutrition Examination Survey (NHANES). The 1999–2002 survey was used for training, while data from the 2003–2006 survey was tested as a validation set. Twelve combinations of pre-processing and clustering algorithms were applied to the training set. The quality of the resulting clusters was evaluated both by considering their properties and by comparative enrichment analysis. Cluster assignments were projected to the validation set (using an artificial neural network) and enrichment in health/life style traits in the resulting clusters was compared to the clusters generated from the original training set.ResultsThe clusters obtained with different pre-processing and clustering combinations differed both in terms of cluster quality measures and in terms of reproducibility of enrichment with health/life style properties. Z-score normalization, for example, dramatically improved cluster quality and enrichments, as compared to unprocessed data, regardless of the clustering algorithm used. Clustering diabetes patients revealed a group of patients enriched with retinopathies. This could indicate that routine laboratory tests can be used to detect patients suffering from complications of diabetes, although other explanations for this observation should also be considered.ConclusionsClustering according to classical clinical biomarkers is a robust process, which may help in patient stratification. However, optimization of the pre-processing and clustering process may be still required.

Reproducible Cancer Biomarker Discovery in SELDI-TOF MS Using Different Pre-Processing Algorithms

BackgroundThere has been much interest in differentiating diseased and normal samples using biomarkers derived from mass spectrometry (MS) studies. However, biomarker identification for specific diseases has been hindered by irreproducibility. Specifically, a peak profile extracted from a dataset for biomarker identification depends on a data pre-processing algorithm. Until now, no widely accepted agreement has been reached.ResultsIn this paper, we investigated the consistency of biomarker identification using differentially expressed (DE) peaks from peak profiles produced by three widely used average spectrum-dependent pre-processing algorithms based on SELDI-TOF MS data for prostate and breast cancers. Our results revealed two important factors that affect the consistency of DE peak identification using different algorithms. One factor is that some DE peaks selected from one peak profile were not detected as peaks in other profiles, and the second factor is that the statistical power of identifying DE peaks in large peak profiles with many peaks may be low due to the large scale of the tests and small number of samples. Furthermore, we demonstrated that the DE peak detection power in large profiles could be improved by the stratified false discovery rate (FDR) control approach and that the reproducibility of DE peak detection could thereby be increased.ConclusionsComparing and evaluating pre-processing algorithms in terms of reproducibility can elucidate the relationship among different algorithms and also help in selecting a pre-processing algorithm. The DE peaks selected from small peak profiles with few peaks for a dataset tend to be reproducibly detected in large peak profiles, which suggests that a suitable pre-processing algorithm should be able to produce peaks sufficient for identifying useful and reproducible biomarkers.

Modelling steel heat treatment data using granular data compression and multiple granularity modelling

In this paper, a systematic modelling approach is presented, involving two algorithmic procedures: a data pre-processing and data compression algorithm using granular computing and statistics; and a granular neural-fuzzy ensemble network consisting of multiple granularity models. Both algorithmic procedures aim to reduce the data and modelling scatter often found in real industrial complex data. The study focuses on the prediction of the mechanical property of heat treated steel, in particular Charpy Toughness. This mechanical property yields high data scatter caused by unknown underlying fractural dynamics. The proposed methodology is shown to successfully model the process under investigation using a real industrial dataset.

A statistical correction method for minimization of systemic artefact in a continuous-rotate X-ray based industrial CT system

The use of a linear detector array (LDA) in X-ray computed tomography (CT) imaging is well established. Generally the CT system using an LDA operates in a fan-beam configuration. In a non-medical tomography set-up, the X-ray source and detectors are stationary and the object is rotated for scanning. Equi-spaced angular projections over a complete 360° object rotation are required for CT image reconstruction using standard Convolution Back Projection (CBP) algorithm as applied to the fan-beam scanning geometry. If there is a lack of timing synchronization between the LDA data acquisition system and the rotary motion in a continuous-rotate system, the incremental angle for the acquired projections may not be known exactly. The error may cause artefacts and blurring in the CT image. The present paper describes a possible way of numerically tuning the acquired transmission data matrix for minimization of such artefacts. It is based on finding out statistically similar projections, which represent 0° and 360° angular views. The correction method has been developed to avoid modification in detector hardware and interface devices. The experimental CT system for industrial applications has been developed by making use of an independent scintillator-based linear detector array, commonly used for on-line radiography of low-density specimens. The proposed data pre-processing algorithm with some experimental results as well as limitations of the correction method is discussed.

A new method for classification of wines based on proton and carbon-13 NMR spectroscopy in combination with pattern recognition techniques

Proton and carbon-13 nuclear magnetic resonance (NMR) spectroscopic methods were applied in the differentiation of 53 German white wines from the regions Rheinhessen, Rheingau and Mosel—Saar—Ruwer. The robustness of these differentiations was tested by leaving out one sample at a time and replacing it into the set as a test sample. Small variations in the position of the lines in the spectra, which lowered the classification ability of the tested systems, were successfully eliminated using a novel data pre-processing algorithm called partial linear fit.