Outlier Replacement Research Articles

Outliers in diffusion-weighted MRI: Exploring detection models and mitigation strategies

Diffusion-weighted MRI (dMRI) is a medical imaging method that can be used to investigate the brain microstructure and structural connections between different brain regions. The method, however, requires relatively complex data processing frameworks and analysis pipelines. Many of these approaches are vulnerable to signal dropout artefacts that can originate from subjects moving their head during the scan.To combat these artefacts and eliminate such outliers, researchers have proposed two approaches: to replace outliers or to downweight outliers during modelling and analysis. With the rising interest in dMRI for clinical research, these types of corrections are increasingly important. Therefore, we set out to investigate the differences between outlier replacement and weighting approaches to help the dMRI community to select the best tool for their data processing pipelines. We evaluated dMRI motion correction registration and single tensor model fit pipelines using Gaussian Process and Spherical Harmonic based replacement approaches and outlier downweighting using highly realistic whole-brain simulations. As a proof of concept, we applied these approaches to dMRI infant data sets that contained varying numbers of dropout artefacts.Based on our results, we concluded that the Gaussian Process based outlier replacement provided similar tensor fit results to Gaussian Process based outlier detection and downweighting. Therefore, if only the least-squares estimate of the single tensor model is of interest, our recommendation is to use outlier replacement. However, outlier downweighting can potentially provide a more accurate estimate of the model precision which could be relevant for applications such as probabilistic tractoraphy.

Handling outliers in bankruptcy prediction models based on logistic regression

The primary tool for managing bank default risk is the credit rating of potential customers. The focus of the present study is on the logistic regression method used to construct 95% of the lender scorecards. The aim of the research is to determine how much the treatment of outliers improves the classification accuracy of the models when using a method that is highly sensitive to outliers, and which method of treating outliers results in the highest classification accuracy. Furthermore, what criteria should be used to determine the cut-off value of the models for a sample that does not contain solvent and insolvent businesses in equal proportions. The analysis was carried out on a sample of 1677 construction companies. The results show that the treatment of outliers significantly improves the predictive ability of the models, while the replacement of outliers with the closest non-outlier proved to be the most effective for treating outliers. When determining the cut-off, it is inappropriate to use the value that results in the highest classification accuracy, as this may lead to an increase in the proportion of first-order errors. The optimisation of this value may depend on the degree of credit risk taken by a given financial institution in its portfolio of loans.

Application of k-sigma Algorithm Based on Monte Carlo in Outlier Detection

Under the background of intelligent times based on information technology, the analysis and replacement of outliers become particularly critical for databases in the face of massive sequential data streams. In order to improve the effectiveness and practicability of the detection method, this paper determines the abnormal scores of data points in a specific data set by using the K-Sigma algorithm of the Monte Carlo method, adaptively adjusts the k value according to the abnormal scores, and marks the abnormal points.The advantages of the traditional k-sigma algorithm are fast operation speed and theoretical basis, while the disadvantages are that each index dimension is determined independently. However, in industrial production, due to the multi-index dimension of time series data, multiple size indicators of a data set are related to each other. Therefore, it is necessary to comprehensively consider whether an anomaly is abnormal by combining it with the data of other indicators. In addition, when there are no outliers in the data set, the traditional k-sigma algorithm is used to take the mean and variance of the data set as parameters. Due to the limitations of the data set itself, some data points will be mislabeled as outliers. Through the k-sigma algorithm based on the Monte Carlo method, we can effectively solve the above problems.The k-sigma algorithm based on the Monte Carlo method can generate a normal distribution according to the distribution of original data points, and extract a large quantity of data from the distribution to generate Monte Carlo data set. The Mahalanobis distance li from each sample point to the mean in the Monte Carlo data set is calculated and compared with the Mahalanobis distance lj from the samples to be detected to the mean in the original data set. According to the number of sample points satisfying li < lj in the Monte Carlo data set, the value of the parameter kj is determined adaptively, and thus the outliers are determined. We implemented the k-sigma algorithm based on the Monte Carlo method through python, evaluated the effectiveness of the algorithm by accuracy, recall rate, and F1 score, and compared it with some machine learning algorithms. The results verified the feasibility and effectiveness of the algorithm, which can be used for real-time anomaly detection in the energy management database.

Test-retest reliability of diffusion tensor imaging scalars in 5-year-olds.

Diffusion tensor imaging (DTI) has provided great insights into the microstructural features of the developing brain. However, DTI images are prone to several artifacts and the reliability of DTI scalars is of paramount importance for interpreting and generalizing the findings of DTI studies, especially in the younger population. In this study, we investigated the intrascan test–retest repeatability of four DTI scalars: fractional anisotropy (FA), mean diffusivity (MD), axial diffusivity (AD), and radial diffusivity (RD) in 5‐year‐old children (N = 67) with two different data preprocessing approaches: a volume censoring pipeline and an outlier replacement pipeline. We applied a region of interest (ROI) and a voxelwise analysis after careful quality control, tensor fitting and tract‐based spatial statistics. The data had three subsets and each subset included 31, 32, or 33 directions thus a total of 96 unique uniformly distributed diffusion encoding directions per subject. The repeatability of DTI scalars was evaluated with intraclass correlation coefficient (ICC(3,1)) and the variability between test and retest subsets. The results of both pipelines yielded good to excellent (ICC(3,1) > 0.75) reliability for most of the ROIs and an overall low variability (<10%). In the voxelwise analysis, FA and RD had higher ICC(3,1) values compared to AD and MD and the variability remained low (<12%) across all scalars. Our results suggest high intrascan repeatability in pediatric DTI and lend confidence to the use of the data in future cross‐sectional and longitudinal studies.

Data Analytics and Reporting API – A Reliable Tool for Data Visualization and Predictive Analysis

Data analytics, the science of analyzing raw data is widely being used in a broad range of applications to makeeffective, efficient and timely decisions in the real-time prediction problems. Analytics requires the user to havean in-depth knowledge about the various steps involved in the process for arriving at the conclusions. Hence, tomake it easier for the naïve users, Data Analytics and Reporting API (DARAPI) provides Analytics as a Service inthe form of an Application Programming Interface (API) has been proposed. DARAPI is aimed to ease the processof analyzing the data by the users without requiring their expertise in the field. It accepts the input file from theuser and performs pre-processing techniques at various stages including imputation of missing data, detectionand replacement of outliers, encoding of categorical variables, and etc. Furthermore, feature engineering is performed, based on which DARAPI will execute the different classification/regression models and finally deliversthe model which provides the best accuracy for future predictions. This entire system is rendered to the user inthe form of an API that can be called from any device that is Internet enabled. DARAPI stands unique with its embedded feedback mechanism which generates constructive input for future predictions. This feature enhancesthe performance of the system in comparison with the existing tools. DARAPI proves beneficial not only to thenaïve users but also to the experts by saving their time and efforts needed in understanding the data.

Multilevel RTN Removal Tools for Dynamic FBG Strain Measurements Corrupted by Peak-Splitting Artefacts.

Strain measurements using fibre Bragg grating (FBG) optical sensors are becoming ever more commonplace. However, in some cases, these measurements can become corrupted by sudden jumps in the signal, which manifest as spikes or step-like offsets in the data. These jumps are caused by a defect in the FBG itself, which is referred to as peak-splitting. The effects of peak splitting artefacts on FBG strain measurements show similarities with an additive multi-level telegraph noise process, in which the amplitudes and occurrences of the jumps are related to fibre deformation states. Whenever it is not possible to re-assess the raw spectral data with advanced peak tracking software, other means for removing the jumps from the data have to be found. The two methods presented in this article are aimed at removing additive multi-level random telegraph noise (RTN) from the raw data. Both methods are based on denoising the sample wise difference signal using a combination of an outlier detection scheme followed by an outlier replacement step. Once the difference signal has been denoised, the cumulative sum is used to arrive back at a strain time series. Two methods will be demonstrated for reconstructing severely corrupted strain time series; the data for this verification has been collected from sub-soil strain measurements obtained from an operational offshore wind-turbine. The results show that the proposed methods can be used effectively to reconstruct the dynamic content of the corrupted strain time series. It has been illustrated that errors in the outlier replacements accumulate and can cause a quasi-static drift. A representative mean value and drift correction are proposed in terms of an optimization problem, which maximizes the overlap between the reconstruction and a subset of the raw data; whereas a high-pass filter is suggested to remove the quasi static drift if only the dynamic band of the signal is of interest.

Anomaly Detection in Photovoltaic Production Factories via Monte Carlo Pre-Processed Principal Component Analysis

This paper investigates a use case of robust anomaly detection applied to the scenario of a photovoltaic production factory—namely, Enel Green Power’s 3SUN solar cell production plant in Catania, Italy—by considering a Monte Carlo based pre-processing technique as a valid alternative to other typically used methods. In particular, the proposed method exhibits the following advantages: (i) Outlier replacement, by contrast with traditional methods which are limited to outlier detection only, and (ii) the preservation of temporal locality with respect to the training dataset. After pre-processing, the authors trained an anomaly detection model based on principal component analysis and defined a suitable key performance indicator for each sensor in the production line based on the model errors. In this way, by running the algorithm on unseen data streams, it is possible to isolate anomalous conditions by monitoring the above-mentioned indicators and virtually trigger an alarm when exceeding a reference threshold. The proposed approach was tested on both standard operating conditions and an anomalous scenario. With respect to the considered use case, it successfully anticipated a fault in the equipment with an advance of almost two weeks, but also demonstrated its robustness to false alarms during normal conditions.

Investigation of the Effects of Simple Outlier Replacement Protocols in Forecasting Analyses: Are they Robust-in-Utility?

Investigation of the Effects of Simple Outlier Replacement Protocols in Forecasting Analyses: Are they Robust-in-Utility?

Short-Term Forecasting for the Electricity Spot Prices With Extreme Values Treatment

Nowadays, modeling and forecasting electricity spot prices are challenging due to their specific features, including multiple seasonalities, calendar effects, and extreme values (also known as jumps, spikes, or outliers). This study aims to provide a comprehensive analysis of electricity price forecasting by comparing several outlier filtering techniques followed by various modeling frameworks. To this end, extreme values are first treated with five different filtering techniques and are then replaced by four different outlier replacement approaches. Next, the spikes-free series is divided into deterministic and stochastic components. The deterministic component includes long-term trend, yearly and weekly seasonalities, and bank holidays and is estimated through parametric and nonparametric approaches. On the other hand, the stochastic component accounts for the short-run dynamics of the price time series and is modeled using different univariate and multivariate models. The one-day-ahead out-of-sample forecast results for the Italian Power Exchange (IPEX), obtained for a whole year, suggest that the outliers pre-filtering give a high accuracy gain. In addition, multivariate modeling for the stochastic component outperforms univariate models.

Forecasting of salmonellosis epidemic proces in Ukraine using autoregressive integrated moving average model.

The article highlights the problem of salmonellosis among the population of the Kharkov region, Ukraine. Three time series were used for calculations: a series of incidence rates for men, a series of incidence rates for women and a series of incidence rates for the general population, each of the series was an ordered set of monthly values from December 2015 to December 2018. It was revealed that the most effective tool for analyzing these statistical data is the use of the autoregressive moving average model (ARIMA). The following steps were used: identification and replacement of outliers, the use of smoothing and decomposition of the series. The developed model allows you to explicitly indicate the order of the model using the arima () function or automatically generate a set of optimal values (p, d, q) using the auto.arima () function. The validated model allows to calculate the predicted values of the incidence of salmonellosis for 50 days. In certain cases, models of exponential smoothing are able to give forecasts that are not inferior in accuracy to forecasts obtained using more complex models.

Deep learning-based method for reducing residual motion effects in diffusion parameter estimation.

Conventional motion-correction techniques for diffusion MRI can introduce motion-level-dependent bias in derived metrics. To address this challenge, a deep learning-based technique was developed to minimize such residual motion effects. The data-rejection approach was adopted in which motion-corrupted data are discarded before model-fitting. A deep learning-based parameter estimation algorithm, using a hierarchical convolutional neural network (H-CNN), was combined with motion assessment and corrupted volume rejection. The method was designed to overcome the limitations of existing methods of this kind that produce parameter estimations whose quality depends strongly on a proportion of the data discarded. Evaluation experiments were conducted for the estimation of diffusion kurtosis and diffusion-tensor-derived measures at both the individual and group levels. The performance was compared with the robust approach of iteratively reweighted linear least squares (IRLLS) after motion correction with and without outlier replacement. Compared with IRLLS, the H-CNN-based technique is minimally sensitive to motion effects. It was tested at severe motion levels when 70% to 90% of the data are rejected and when random motion is present. The technique had a stable performance independent of the numbers and schemes of data rejection. A further test on a data set from children with attention-deficit hyperactivity disorder shows the technique can potentially ameliorate spurious group-level difference caused by head motion. This method shows great potential for reducing residual motion effects in motion-corrupted diffusion-weighted-imaging data, bringing benefits that include reduced bias in derived metrics in individual scans and reduced motion-level-dependent bias in population studies employing diffusion MRI.

The efficacy of different preprocessing steps in reducing motion-related confounds in diffusion MRI connectomics

Head motion is a major confounding factor in neuroimaging studies. While numerous studies have investigated how motion impacts estimates of functional connectivity, the effects of motion on structural connectivity measured using diffusion MRI have not received the same level of attention, despite the fact that, like functional MRI, diffusion MRI relies on elaborate preprocessing pipelines that require multiple choices at each step. Here, we report a comprehensive analysis of how these choices influence motion-related contamination of structural connectivity estimates. Using a healthy adult sample (N = 294), we evaluated 240 different preprocessing pipelines, devised using plausible combinations of different choices related to explicit head motion correction, tractography propagation algorithms, track seeding methods, track termination constraints, quantitative metrics derived for each connectome edge, and parcellations. We found that an approach to motion correction that includes outlier replacement and within-slice volume correction led to a dramatic reduction in cross-subject correlations between head motion and structural connectivity strength, and that motion contamination is more severe when quantifying connectivity strength using mean tract fractional anisotropy rather than streamline count. We also show that the choice of preprocessing strategy can significantly influence subsequent inferences about network organization, with the location of network hubs varying considerably depending on the specific preprocessing steps applied. Our findings indicate that the impact of motion on structural connectivity can be successfully mitigated using recent motion-correction algorithms that include outlier replacement and within-slice motion correction.

PIV-DCNN: cascaded deep convolutional neural networks for particle image velocimetry

Velocity estimation (extracting the displacement vector information) from the particle image pairs is of critical importance for particle image velocimetry. This problem is mostly transformed into finding the sub-pixel peak in a correlation map. To address the original displacement extraction problem, we propose a different evaluation scheme (PIV-DCNN) with four-level regression deep convolutional neural networks. At each level, the networks are trained to predict a vector from two input image patches. The low-level network is skilled at large displacement estimation and the high- level networks are devoted to improving the accuracy. Outlier replacement and symmetric window offset operation glue the well- functioning networks in a cascaded manner. Through comparison with the standard PIV methods (one-pass cross-correlation method, three-pass window deformation), the practicability of the proposed PIV-DCNN is verified by the application to a diversity of synthetic and experimental PIV images.

Temporal interpolation alters motion in fMRI scans: Magnitudes and consequences for artifact detection.

Head motion can be estimated at any point of fMRI image processing. Processing steps involving temporal interpolation (e.g., slice time correction or outlier replacement) often precede motion estimation in the literature. From first principles it can be anticipated that temporal interpolation will alter head motion in a scan. Here we demonstrate this effect and its consequences in five large fMRI datasets. Estimated head motion was reduced by 10–50% or more following temporal interpolation, and reductions were often visible to the naked eye. Such reductions make the data seem to be of improved quality. Such reductions also degrade the sensitivity of analyses aimed at detecting motion-related artifact and can cause a dataset with artifact to falsely appear artifact-free. These reduced motion estimates will be particularly problematic for studies needing estimates of motion in time, such as studies of dynamics. Based on these findings, it is sensible to obtain motion estimates prior to any image processing (regardless of subsequent processing steps and the actual timing of motion correction procedures, which need not be changed). We also find that outlier replacement procedures change signals almost entirely during times of motion and therefore have notable similarities to motion-targeting censoring strategies (which withhold or replace signals entirely during times of motion).

Divergence-free smoothing for volumetric PIV data

This paper proposes a divergence-free smoothing (DFS) method for the post-process of volumetric particle image velocimetry (PIV) data, which can smooth out noise and divergence error at the same time. The method is a combination of the penalized least squares regression and the divergence corrective scheme (DCS), employing the generalized cross-validation method to automatically determine the best smoothing parameter. By introducing a weight-changing algorithm similar to the all-in-one method, a robust version of DFS can simultaneously deal with vector validation, replacement of outliers and missing vectors, smoothing, and zero-divergence correction of the velocity field. Direct numerical simulation data of turbulent channel flow (Johns Hopkins Turbulence Databases) added with artificial noise, outliers and missing vectors are used to test the accuracy of DFS. The results show that DFS can smooth the velocity field to divergence-free and performs better than the all-in-one method, DCS and some other available conventional processing methods for post-process of velocity field, especially in dealing with clustered outliers and missing vectors. A block DFS is suggested to process large velocity field to save both time and memory. Tests on tomographic PIV data validate the effectiveness of DFS on improving both flow statistics and flow visualization.

China's only children and psychopathology: A quantitative synthesis.

The goal of this study is to synthesize quantitatively the results of studies of psychopathology among Chinese only children. Since 1979, China's 1-child policy has generated large numbers of only children, especially in large urban centers, where the 1-child family has become a social norm. Motivated by concern for mental health, 22 studies, based on the SCL-90, have been published that compare the scores of only children to their peers with siblings. The raw effect sizes generated by each study underwent adjustments to enhance the reliability of the findings, including the identification and replacement of outliers, and weighting by inverse-sample size. In addition, analyses were conducted to evaluate the degree of publication bias exhibited by this collection of studies and the results from the SCL-90 studies were compared to studies using alternative measures of anxiety and depression. Overall, the synthesis found small, but significant advantages for only children compared to their peers with siblings, regardless of subscale. However, moderators of this only-child effect were also found: Only children as college students reported significantly fewer symptoms, regardless of subscale, while only children as military recruits reported more symptoms, although the findings about military recruits received less support from the analyses. Furthermore, the size of the only-child advantage was found to be greater for only children born after the policy. Conclusions based on this synthesis are limited by the fact that this body of studies is based on convenience samples of relatively successful youth.

Nonlinear temporal filtering of time-resolved digital particle image velocimetry data

Nonlinear filtering methods have been developed to identify and replace outlying data points in velocity time series obtained with time-resolved digital particle image velocimetry (PIV) of the flow around a surface-mounted cube at a Reynolds number of 20,000. Nuances associated with the spectral computation of the cross-correlation are highlighted, including the requirement of zero-padding an image interrogation area to eliminate the circular components of the cross-correlation. Three nonlinear filtering methods for the replacement of outliers are applied to the velocity time series sampled at 1,000 Hz: a median filter, a decision-based Hampel filter, and a PIV-specific Hampel filter. The particular benefit of the PIV-specific Hampel filter is that it allows the retention of actual measured data, sometimes derived from alternate peaks in the cross-correlation function, while still providing for the removal of outliers when a consistent, nonoutlying measurement is not available.

A Robust PCA Modeling Method for Process Monitoring

A robust method for dealing with the gross errors in the data collected for PCA model is proposed. This method, using M-estimator based on the generalized t distribution, adaptively transforms the data in the score space in order to eliminate the effects of the outliers in the original data. The robust estimation of the covariance or correlation matrix is obtained by the proposed approach so that the accurate PCA model can be obtained for the process monitoring purpose. Comparisons with the conventional PCA modeling and other robust outlier's replacement approaches are illustrated through a chemical engineering example.

A method of robust multivariate outlier replacement

Robust multivariate methods for dealing with problems caused by outliers in the data are essential especially when process data are used to validate mechanistic models, develop regression models, and in applications such as controller design and process monitoring. Gross outliers are detected easily by simple methods such as range checking, however, a multivariate outlier is very difficult to discern and techniques that rely on data to generate empirical models may produce erroneous results. In this work, a methodology to perform multivariate outlier replacement in the score space generated by principal component analysis (PCA) is proposed. The objective was to find an accurate estimate of the covariance matrix of the data so that a PCA model might be developed that could then be used for monitoring and fault detection and identification. The methodology uses the concept of winsorization to provide robust estimates of the mean (location) and S.D. (scale) iteratively, yielding a robust set of data. The paper develops the approach, discusses the concept of robust statistics and winsorization, and presents the procedures for robust multivariate outlier filtering. One simulated and two industrial examples are provided to demonstrate the approach.

Modelling and analysing outliers in spatial lattice data

We describe a procedure for the detection and replacement of outliers in spatial lattice data. The procedure is designed to produce bias-robust estimates of autocorrelation functions and spectral density functions. The application of the method is illustrated using ecological and environmental data.