Statistical Preprocessing Research Articles

Statistical data pre-processing and time series incorporation for high-efficacy calibration of low-cost NO2 sensor using machine learning

Air pollution stands as a significant modern-day challenge impacting life quality, the environment, and the economy. It comprises various pollutants like gases, particulate matter, biological molecules, and more, stemming from sources such as vehicle emissions, industrial operations, agriculture, and natural events. Nitrogen dioxide (NO2), among these harmful gases, is notably prevalent in densely populated urban regions. Given its adverse effects on health and the environment, accurate monitoring of NO2 levels becomes imperative for devising effective risk mitigation strategies. However, the precise measurement of NO2 poses challenges as it traditionally relies on costly and bulky equipment. This has prompted the development of more affordable alternatives, although their reliability is often questionable. The aim of this article is to introduce a groundbreaking method for precisely calibrating cost-effective NO2 sensors. This technique involves statistical preprocessing of low-cost sensor readings, aligning their distribution with reference data. Central to this calibration is an artificial neural network (ANN) surrogate designed to predict sensor correction coefficients. It utilizes environmental variables (temperature, humidity, atmospheric pressure), cross-references auxiliary NO2 sensors, and incorporates short time series of previous readings from the primary sensor. These methods are complemented by global data scaling. Demonstrated using a custom-designed cost-effective monitoring platform and high-precision public reference station data collected over 5 months, every component of our calibration framework proves crucial, contributing to its exceptional accuracy (with a correlation coefficient near 0.95 concerning the reference data and an RMSE below 2.4 µg/m3). This level of performance positions the calibrated sensor as a viable, cost-effective alternative to traditional monitoring approaches.

Computational design synthesis for Fabrication-Aware assembly problems using building objects with dimensional variations

State-of-the-art computational design approaches for designing the assembly of a structure often rely on high-precision building objects, which can be difficult and costly to fabricate. To consider a wider range of construction materials and fabrication processes, e.g. low-cost 3D printing and brick making, this research introduces a novel computational design method for fabrication-aware assembly problems using existing building components with dimensional variations. Given a finite set of building objects, this study develops a statistical preprocessing approach and a modified pattern search with an efficient integer linear programming method to generate the assembly of a target structure, such that it is statically stable and optimized for geometric smoothness and space utilization. In two case studies, the proposed algorithm exhibits significantly better boundary fitting and space utilization compared with a genetic algorithm and a standard pattern search method and lower computational cost compared with a genetic algorithm.

The Application of Statistical Preprocessing on Spectral Data Does Not Always Guarantee the Improvement of the Predictive Quality of Multivariate Models: Case of Soil Spectroscopy Applied to Moroccan Soils (Rhamna Region)

The Application of Statistical Preprocessing on Spectral Data Does Not Always Guarantee the Improvement of the Predictive Quality of Multivariate Models: Case of Soil Spectroscopy Applied to Moroccan Soils (Rhamna Region)

Determination of radiological background fields designated for inverse modelling during atypical low wind speed meteorological episode

This article focuses on the formation of complex trajectories of radiological background fields for atypical accidental discharges of radioactivity into the atmosphere during very low wind speed episodes (calms). Within several hours of a calm meteorological situation, a relatively significant level of radioactivity can be accumulated around the source. In the next stage, the calm situation is assumed to terminate and convective movement of the air immediately starts. A packet of accumulated radioactivity, which has the form of multiple Gaussian puffs, is drifted by wind. Consequently, the pollution is disseminated over the terrain. Significant transport of radioactivity even behind the protective zone of a nuclear facility (up to between 15 and 20 km) has been observed. Original optional statistical pre-processing of calm's results is inserted between the calm and convective stages of the trajectory generation which may improve performance of following computationally expensive methods of Bayesian filtering. Determination of complex trajectory passing through both calm and convective stages of the release scenario represents inevitable prior information for prospective assimilation techniques. Draft twin experiment is outlined for a simple assimilation scenario for re-estimation of the main model parameters based on a notional monitoring network in the outer convective region.

Relative effects of statistical preprocessing and postprocessing on a regional hydrological ensemble prediction system

Abstract. The relative roles of statistical weather preprocessing and streamflow postprocessing in hydrological ensemble forecasting at short- to medium-range forecast lead times (day 1–7) are investigated. For this purpose, a regional hydrologic ensemble prediction system (RHEPS) is developed and implemented. The RHEPS is comprised of the following components: (i) hydrometeorological observations (multisensor precipitation estimates, gridded surface temperature, and gauged streamflow); (ii) weather ensemble forecasts (precipitation and near-surface temperature) from the National Centers for Environmental Prediction 11-member Global Ensemble Forecast System Reforecast version 2 (GEFSRv2); (iii) NOAA's Hydrology Laboratory-Research Distributed Hydrologic Model (HL-RDHM); (iv) heteroscedastic censored logistic regression (HCLR) as the statistical preprocessor; (v) two statistical postprocessors, an autoregressive model with a single exogenous variable (ARX(1,1)) and quantile regression (QR); and (vi) a comprehensive verification strategy. To implement the RHEPS, 1 to 7 days weather forecasts from the GEFSRv2 are used to force HL-RDHM and generate raw ensemble streamflow forecasts. Forecasting experiments are conducted in four nested basins in the US Middle Atlantic region, ranging in size from 381 to 12 362 km2. Results show that the HCLR preprocessed ensemble precipitation forecasts have greater skill than the raw forecasts. These improvements are more noticeable in the warm season at the longer lead times (> 3 days). Both postprocessors, ARX(1,1) and QR, show gains in skill relative to the raw ensemble streamflow forecasts, particularly in the cool season, but QR outperforms ARX(1,1). The scenarios that implement preprocessing and postprocessing separately tend to perform similarly, although the postprocessing-alone scenario is often more effective. The scenario involving both preprocessing and postprocessing consistently outperforms the other scenarios. In some cases, however, the differences between this scenario and the scenario with postprocessing alone are not as significant. We conclude that implementing both preprocessing and postprocessing ensures the most skill improvements, but postprocessing alone can often be a competitive alternative.

Model-Based Outlier Detection System with Statistical Preprocessing

Reliability, lack of error, and security are important improvements to quality of service. Outlier detection is a process of detecting the erroneous parts or abnormal objects in defined populations, and can contribute to secured and error-free services. Outlier detection approaches can be categorized into four types: statistic-based, unsupervised, supervised, and semi-supervised. A model-based outlier detection system with statistical preprocessing is proposed, taking advantage of the statistical approach to preprocess training data and using unsupervised learning to construct the model. The robustness of the proposed system is evaluated using the performance evaluation metrics sum of squared error (SSE) and time to build model (TBM). The proposed system performs better for detecting outliers regardless of the application domain.

Upgrading the Performance of Speech Emotion Recognition at the Segmental Level

This paper presents an efficient approach for maximizing the accuracy of automatic speech emotion recognition in English, using minimal inputs, minimal features, lesser algorithmic complexity and reduced processing time. Whereas the findings reported here are based on the exclusive use of vowel formants, most of the related previous works used tens or even hundreds of other features. In spite of using a greater level of signal processing, the recognition accuracy reported earlier was often lesser than that obtained by our approach. This method is based on vowel utterances and the first step comprises statistical pre-processing of the vowel formants. This is followed by the identification of the best formants using the KMeans, K-nearest neighbor and Naive Bayes classifiers. The Artificial neural network that was used for the final classification gave an accuracy of 95.6% on elicited emotional speech. Nearly 1500 speech files from ten female speakers in the neutral and six basic emotions were used to prove the efficiency of the proposed approach. Such a result has not been reported earlier for English and is of significance to researchers, sociologists and others interested in speech.

Self-organizing maps for texture classification

A further investigation of our intelligent machine vision system for pattern recognition and texture image classification is discussed in this paper. A data set of 335 texture images is to be classified into several classes, based on their texture similarities, while no a priori human vision expert knowledge about the classes is available. Hence, unsupervised learning and self-organizing maps (SOM) neural networks are used for solving the classification problem. Nevertheless, in some of the experiments, a supervised texture analysis method is also considered for comparison purposes. Four major experiments are conducted: in the first one, classifiers are trained using all the extracted features without any statistical preprocessing; in the second simulation, the available features are normalized before being fed to a classifier; in the third experiment, the trained classifiers use linear transformations of the original features, received after preprocessing with principal component analysis; and in the last one, transforms of the features obtained after applying linear discriminant analysis are used. During the simulation, each test is performed 50 times implementing the proposed algorithm. Results from the employed unsupervised learning, after training, testing, and validation of the SOMs, are analyzed and critically compared with results from other authors.

Preprocessing and Quality Control Strategies for Illumina DASL Assay-Based Brain Gene Expression Studies with Semi-Degraded Samples.

Available statistical preprocessing or quality control analysis tools for gene expression microarray datasets are known to greatly affect downstream data analysis, especially when degraded samples, unique tissue samples, or novel expression assays are used. It is therefore important to assess the validity and impact of the assumptions built in to preprocessing schemes for a dataset. We developed and assessed a data preprocessing strategy for use with the Illumina DASL-based gene expression assay with partially degraded postmortem prefrontal cortex samples. The samples were obtained from individuals with autism as part of an investigation of the pathogenic factors contributing to autism. Using statistical analysis methods and metrics such as those associated with multivariate distance matrix regression and mean inter-array correlation, we developed a DASL-based assay gene expression preprocessing pipeline to accommodate and detect problems with microarray-based gene expression values obtained with degraded brain samples. Key steps in the pipeline included outlier exclusion, data transformation and normalization, and batch effect and covariate corrections. Our goal was to produce a clean dataset for subsequent downstream differential expression analysis. We ultimately settled on available transformation and normalization algorithms in the R/Bioconductor package lumi based on an assessment of their use in various combinations. A log2-transformed, quantile-normalized, and batch and seizure-corrected procedure was likely the most appropriate for our data. We empirically tested different components of our proposed preprocessing strategy and believe that our results suggest that a preprocessing strategy that effectively identifies outliers, normalizes the data, and corrects for batch effects can be applied to all studies, even those pursued with degraded samples.

Prediction and Analysis of Deposition Efficiency of Plasma Spray Coating Using Artificial Intelligence Method

Modern industrial technologies call for the development of novel materials with improved surface properties, lower costs and environmentally suitable processes. Plasma spray coating process has become a subject of intense research which attempts to create functional layers on the surface is obviously the most economical way to provide high per- formance to machinery and industrial equipments. The present work aims at developing and studying the industrial wastes (Flay-ash, Quartz and illmenite composite mixture) as the coating material, which is to be deposited on Mild Steel and Copper substrates. To study and evaluate Coating deposition efficiency, artificial neural network analysis (ANN) technique is used. By this quality control technique, it is sufficient to describe approximation complex of in- ter-relationships of operating parameters in atmospheric plasma spray process. ANN technique helps in saving time and resources for experimental trials. The aim of this work is to outline a procedure for selecting an appropriate input vec- tors in ANN coating efficiency models, based on statistical pre-processing of the experimental data set. This methodology can provide deep understanding of various co-relationships across multiple scales of length and time, which could be essential for improvement of product and process performance. The deposition efficiency of coatings has a strong dependence on input power level, particle size of the feed material, powder feed rate and torch to substrate distance. ANN experimental results indicate that the projection network has good generalization capability to optimize the deposition efficiency, when an appropriate size of training set and network is utilized.

PreP+07: improvements of a user friendly tool to preprocess and analyse microarray data

BackgroundNowadays, microarray gene expression analysis is a widely used technology that scientists handle but whose final interpretation usually requires the participation of a specialist. The need for this participation is due to the requirement of some background in statistics that most users lack or have a very vague notion of. Moreover, programming skills could also be essential to analyse these data. An interactive, easy to use application seems therefore necessary to help researchers to extract full information from data and analyse them in a simple, powerful and confident way.ResultsPreP+07 is a standalone Windows XP application that presents a friendly interface for spot filtration, inter- and intra-slide normalization, duplicate resolution, dye-swapping, error removal and statistical analyses. Additionally, it contains two unique implementation of the procedures – double scan and Supervised Lowess-, a complete set of graphical representations – MA plot, RG plot, QQ plot, PP plot, PN plot – and can deal with many data formats, such as tabulated text, GenePix GPR and ArrayPRO. PreP+07 performance has been compared with the equivalent functions in Bioconductor using a tomato chip with 13056 spots. The number of differentially expressed genes considering p-values coming from the PreP+07 and Bioconductor Limma packages were statistically identical when the data set was only normalized; however, a slight variability was appreciated when the data was both normalized and scaled.ConclusionPreP+07 implementation provides a high degree of freedom in selecting and organizing a small set of widely used data processing protocols, and can handle many data formats. Its reliability has been proven so that a laboratory researcher can afford a statistical pre-processing of his/her microarray results and obtain a list of differentially expressed genes using PreP+07 without any programming skills. All of this gives support to scientists that have been using previous PreP releases since its first version in 2003.

Automated identification of tumor microscopic morphology based on macroscopically measured scatter signatures

An automated algorithm and methodology is presented to identify tumor-tissue morphologies based on broadband scatter data measured by raster scan imaging of the samples. A quasi-confocal reflectance imaging system was used to directly measure the tissue scatter reflectance in situ, and the spectrum was used to identify the scattering power, amplitude, and total wavelength-integrated intensity. Pancreatic tumor and normal samples were characterized using the instrument, and subtle changes in the scatter signal were encountered within regions of each sample. Discrimination between normal versus tumor tissue was readily performed using a K-nearest neighbor classifier algorithm. A similar approach worked for regions of tumor morphology when statistical preprocessing of the scattering parameters was included to create additional data features. This type of automated interpretation methodology can provide a tool for guiding surgical resection in areas where microscopy imaging cannot be realized efficiently by the surgeon. In addition, the results indicate important design changes for future systems.

Constructing neural network sediment estimation models using a data-driven algorithm

Artificial neural network (ANN) models are designed for suspended sediment estimation using statistical pre-processing of the data. Statistical properties such as cross-, auto- and partial auto-correlation of the data series are used for identifying a unique input vector to the ANN that best represents the sediment estimation process for a basin. The methodology is evaluated using the flow and sediment data from the stations Quebrada Blanca and Rio Valenciano in USA. The result of the study indicates that the statistical pre-processing of the data could significantly reduce the effort and computational time required in developing an ANN model. Three ANN training algorithms are also compared with each other for the selected input vector.

Discriminating benign from malignant thyroid lesions using artificial intelligence and statistical selection of morphometric features

The objective of this study was to perform a comparative investigation of the capability of various classifiers in discriminating benign from malignant thyroid lesions. Using May Grunvald-Giemsa-stained smears taken by fine needle aspiration (FNA) and a custom image analysis system, 25 nuclear features describing the size, shape and texture of the nuclei were measured in each case. A statistical pre-processing of features revealed that only 4 of the 25 features are important when discriminating benign from malignant thyroid lesions, which were transformed and fed to four classifiers for subsequent analysis. The cases were divided into one set used for the training of classifiers, a second set used as the test set, and the remaining cases with no clear classification formed an ambiguous test set. Classification was performed at the nuclear and patient level. The technique described in this study produced encouraging results and promises to be a helpful tool in the daily cytological laboratory routine.

A statistical downscaling method for monthly total precipitation over Turkey

AbstractResearchers are aware of certain types of problems that arise when modelling interconnections between general circulation and regional processes, such as prediction of regional, local‐scale climate variables from large‐scale processes, e.g. by means of general circulation model (GCM) outputs. The problem solution is called downscaling. In this paper, a statistical downscaling approach to monthly total precipitation over Turkey, which is an integral part of system identification for analysis of local‐scale climate variables, is investigated. Based on perfect prognosis, a new computationally effective working method is introduced by the proper predictors selected from the National Centers for Environmental Prediction–National Center for Atmospheric Research reanalysis data sets, which are simulated as perfectly as possible by GCMs during the period of 1961–98. The Sampson correlation ratio is used to determine the relationships between the monthly total precipitation series and the set of large‐scale processes (namely 500 hPa geopotential heights, 700 hPa geopotential heights, sea‐level pressures, 500 hPa vertical pressure velocities and 500–1000 hPa geopotential thicknesses). In the study, statistical preprocessing is implemented by independent component analysis rather than principal component analysis or principal factor analysis. The proposed downscaling method originates from a recurrent neural network model of Jordan that uses not only large‐scale predictors, but also the previous states of the relevant local‐scale variables. Finally, some possible improvements and suggestions for further study are mentioned. Copyright © 2004 Royal Meteorological Society

Two new hybrid diagnostic methodologies for real time diagnosis and supervision: A novel approach

This paper presents a new hybrid diagnostic methodology. This new hybrid system involves statistical pre-processing, Artificial Neural Networks (ANN) and Fuzzy Logic (FL). The statistical pre-processing of data allows outliers to be eliminated (filtering) and dimensional reduction while keeping the most important data. Then, each process state is learned by one of the original constructive algorithms discussed: Multidimensional Fuzzy Sets (MFS) which fuzzyfies binary ANN or Generalized Radial Basis Functions (GRBF) learned by the Estimation Maximization (EM) algorithm. The final step is the final decision by Fuzzy Logic algorithms, in which an automatic defuzzification method is proposed. These methodologies produce some parameters which allows real-time to be performed.

The Gaussian hare and the Laplacian tortoise: computability of squared-error versus absolute-error estimators

Since the time of Gauss, it has been generally accepted that $\ell_2$-methods of combining observations by minimizing sums of squared errors have significant computational advantages over earlier $\ell_1$-methods based on minimization of absolute errors advocated by Boscovich, Laplace and others. However, $\ell_1$-methods are known to have significant robustness advantages over $\ell_2$-methods in many applications, and related quantile regression methods provide a useful, complementary approach to classical least-squares estimation of statistical models. Combining recent advances in interior point methods for solving linear programs with a new statistical preprocessing approach for $\ell_1$-type problems, we obtain a 10- to 100-fold improvement in computational speeds over current (simplex-based) $\ell_1$-algorithms in large problems, demonstrating that $\ell_1$-methods can be made competitive with $\ell_2$-methods in terms of computational speed throughout the entire range of problem sizes. Formal complexity results suggest that $\ell_1$-regression can be made faster than least-squares regression for n sufficiently large and p modest.

SELF-ORGANIZED LEARNING IN MULTI-LAYER NETWORKS

We present a framework for the self-organized formation of high level learning by a statistical preprocessing of features. The paper focuses first on the formation of the features in the context of layers of feature processing units as a kind of resource-restricted associative multiresolution learning We clame that such an architecture must reach maturity by basic statistical proportions, optimizing the information processing capabilities of each layer. The final symbolic output is learned by pure association of features of different levels and kind of sensorial input. Finally, we also show that common error-correction learning for motor skills can be accomplished also by non-specific associative learning.

Bioreactor Modeling and Control by Principal Component Based Neural Networks

In the work are developed structures for training and prediction by neural networks based on principal components analysis of input/output patterns. The structures are composed of functional modules dedicated for specific tasks in view of process engineering applications. The structures are serial connections of the following modules: ARMA for approximation by autoregression moving averages of process dynamics and process or transport delays; PS for statistical preprocessing, detection of gross errors, and rejection of redundant patterns; PCA for decomposition of patterns into principal components aimed for data compression and random noise elimination; and SISO-NN subsystem with a parallel connection of separated single input smgle output neural networks. Each NN submodule is structured with a single hidden layer with static neurons and feedforward progression. Parameters of NN modules are determined by Pollak-Ribiére-Powell conjugate gradient optimization. Number of principal components and elements in a hidden layer are optimized for minimal predicted sum of squares (PRESS) with untrained set of patterns The models are applied on data from industrial production of baker s yeast in a deep jet bioreactor.