Identification Of Outliers Research Articles

Rate Distortion Theory for Descriptive Statistics.

Rate distortion theory was developed for optimizing lossy compression of data, but it also has applications in statistics. In this paper, we illustrate how rate distortion theory can be used to analyze various datasets. The analysis involves testing, identification of outliers, choice of compression rate, calculation of optimal reconstruction points, and assigning "descriptive confidence regions" to the reconstruction points. We study four models or datasets of increasing complexity: clustering, Gaussian models, linear regression, and a dataset describing orientations of early Islamic mosques. These examples illustrate how rate distortion analysis may serve as a common framework for handling different statistical problems.

Evaluation of uncertainty in the measurement of the stress-optic coefficient

In this study, a method for evaluating the uncertainty of stress-optic coefficient measurements of photoelastic materials on a uniaxial tension/compression specimen is presented. The same problem is also approached in other previously used methods, in which multiple data fitted with ordinary least squares are considered. However, only the repeatability contribution to uncertainty is considered in those other methods, which are therefore not consistent with the ISO–Guide to the expression of Uncertainty in Measurement. All possible contributions to uncertainty can be taken into account using the General Least Squares-Lagrange Multipliers (GLS-LM) method presented here. The application of the method is illustrated with an example, from which it can be seen that uncertainty has been underestimated in the other methods used to date. As well as the estimate of the stress-optic coefficient and its corresponding standard uncertainty, the method also provides a data consistency test and an outlier identification tool.

Effective peer review audit and identification of the surgeon outlier.

Surgical audit aims to identify ways to maintain and improve the quality of care for patients, in part by assessment of a surgeon's activities and outcomes. However effective data systems to facilitate audit are uncommon. We aimed to assess the effectiveness of a tool for Peer Review Audit. All General Surgeons in Darwin and the Top End were encouraged to self-record their surgical activity, including procedures and adverse events related to procedures, using the College's Morbidity Audit and Logbook Tool (MALT). A total of 6 surgeons and 3518 operative events were recorded in MALT between 2018 and 2019. De-identified reports of each surgeon's activities, compared directly to the audit group, were created by each surgeon, with correction for complexity of procedures and ASA status. Nine complications Grade 3 and greater were recorded, plus 6 deaths, 25 unplanned returns to theatre (8% failure to rescue rate), 7 unplanned admissions to ICU and 8 unplanned readmissions. One surgeon outlier was identified (>3 standard deviation over group mean) for unplanned returns to theatre. This surgeon's specific cases were reviewed at our morbidity and mortality meeting using the MALT Self Audit Report and changes were implemented as a result, with future progress monitored. The College's MALT system effectively enabled Peer Group Audit. All participating surgeons were readily able to present and validate their own results. A surgeon outlier was reliably identified. This led to effective practice change. The proportion of surgeons who participated was low. Adverse events were likely under-reported.

Loading on a wind turbine high-speed shaft gearbox bearing: Ultrasonic field measurements and predictions from a multi-body simulation

Wind turbine gearbox failures, many of which result from load-driven rolling element bearing failures, account for long downtimes and high maintenance costs. Non-invasive, in-situ measurement of bearing roller loads using ultrasound facilitates better understanding of the load history and failure mechanisms these gearbox bearings experience. In this study, ultrasonic load measurements from an instrumented high-speed shaft bearing were compared against results from a static multibody simulation. Measurements taken under steady operating conditions were found to agree well with the model. After quantification of the error in ultrasonic measurements the model enabled identification of load outliers which were attributed to transient events. Over- and under-loading instances may act to promote a variety of failure mechanisms, leading to premature bearing failure.

Student-Engagement Detection in Classroom Using Machine Learning Algorithm

Student engagement is a flexible, complicated concept that includes behavioural, emotional, and cognitive involvement. In order for the instructor to understand how the student interacts with the various activities in the classroom, it is essential to predict their participation. The current work aims to identify the best algorithm for predicting student engagement in the classroom. In this paper, we gathered data from VLE and prepared them using a variety of data preprocessing techniques, including the elimination of missing values, normalization, encoding, and identification of outliers. On our data, we ran a number of machine learning (ML) classification algorithms, and we assessed each one using cross-validation methods and many helpful indicators. The performance of the model is evaluated with metrics like accuracy, precision, recall, and AUC scores. The results show that the CATBoost model is having higher accuracy than the rest. This proposed model outperformed in all the aspects compared to previous research. The results part of this paper indicates that the CATBoost model had an accuracy of approximately 92.23%, a precision of 94.40%, a recall of 100%, and an AUC score of 0.9624. The XGBoost predictive model, the random forest model, and the multilayer perceptron model all demonstrated approximately the same performance overall. We compared the AISAR model with Our model achieved an accuracy of 94.64% compared with AISAR 91% model and it concludes that our results are better. The AISAR model had only around 50% recall compared to our models, which had around 92%. This shows that our models return more relevant results, i.e., if our models predict that a student has high engagement, they are correct 94.64% of the time.

Preconditioning of clinical data for intraocular lens formula constant optimisation using Random Forest Quantile Regression Trees

PurposeTo implement a fully data driven strategy for identifying outliers in clinical datasets used for formula constant optimisation, in order to achieve proper formula predicted refraction after cataract surgery, and to assess the capabilities of this outlier detection method. Methods2 clinical datasets (DS1/DS2: N = 888/403) of eyes treated with a monofocal aspherical intraocular lens (Hoya XY1/Johnson&Johnson Vision Z9003) containing preoperative biometric data, power of the lens implant and postoperative spherical equivalent (SEQ) were transferred to us for formula constant optimisation. Original datasets were used to generate baseline formula constants. A random forest quantile regression algorithm was set up using bootstrap resampling with replacement. Quantile regression trees were grown and the 25% and 75% quantile, and the interquartile range were extracted from SEQ and formula predicted refraction REF for the SRKT, Haigis and Castrop formulae. Fences were defined from the quantiles and data points outside the fences were marked and removed as outliers before recalculating the formula constants. ResultsNB = 1000 bootstrap samples were derived from both datasets, and random forest quantile regression trees were grown to model SEQ versus REF and to estimate the median and 25% and 75% quantiles. The fence boundaries were defined as being from 25% quantile - 1.5·IQR to 75% quantile + 1.5·IQR, with data points outside the fence being marked as outliers. In total, for DS1 and DS2, 25/27/32 and 4/5/4 data points were identified as outliers for the SRKT/Haigis/Castrop formulae respectively. The respective root mean squared formula prediction errors for the three formulae were slightly reduced from: 0.4370 dpt;0.4449 dpt/0.3625 dpt;0.4056 dpt/and 0.3376 dpt;0.3532 dpt to: 0.4271 dpt;0.4348 dpt/0.3528 dpt;0.3952 dpt/0.3277 dpt;0.3432 dpt for DS1;DS2. ConclusionWe were able to prove that with random forest quantile regression trees a fully data driven outlier identification strategy acting in the response space is achievable. In a real life scenario this strategy has to be complemented by an outlier identification method acting in the parameter space for a proper qualification of datasets prior to formula constant optimisation.

Defining the Phenotype and Prognosis of People With Idiopathic Intracranial Hypertension After Cerebrospinal Fluid Diversion Surgery

To characterize the phenotype of patients with idiopathic intracranial hypertension (IIH) who received cerebrospinal (CSF) diversion surgery and to detail the trajectory of recovery. Prospective cohort registry study. Patients with IIH with sight-threatening papilledema presenting to a single United Kingdom neuroscience center between 2019 and 2021 were included. Outcomes consisted of perimetric mean deviation (PMD) and optical coherence tomography measures of papilledema (retinal nerve fiber layer [RNFL]) and macular ganglion cell layer (GCL) in both eyes. Headache outcomes included monthly headache days (MHD). Logistic regression methods were used to model long-term outcomes. Fifty-one patients without previous surgical interventions were included (92% female, mean age 28.1 years [SD 8.4], body mass index 37.4 kg/m2 [SD 9.7], mean days of follow-up 330 [SD 209]). Measurements before surgery showed mean PMD -11.4 dB (SD 9.7), RNFL 364 µm (SD 128), Frisén grade papilledema 4.3 (SD 0.9). and MHD 23 (SD 10.6). At 1 month postoperatively, RNFL and PMD had improved by 38% and 4%, respectively. At 4 months postoperatively, papilledema had resolved. GCL declined by 13% over 12 months. MHD reduced by 75% 3 months postoperatively before returning to baseline levels by 12 months. Five patients (9.8%) required revision surgeries. Detailed characteristics of patients with sight-threatening IIH who received CSF diversion surgery and their typical postoperative recovery are presented. These parameters should guide physicians as to when patients with IIH may require surgery and enable the early identification of outliers who fail to respond. Papilledema and PMD recovered but GCL atrophy continued for 12 months. The implication of this delayed atrophy is unknown.

Outlier identification and monitoring of institutional or clinician performance: an overview of statistical methods and application to national audit data

BackgroundInstitutions or clinicians (units) are often compared according to a performance indicator such as in-hospital mortality. Several approaches have been proposed for the detection of outlying units, whose performance deviates from the overall performance.MethodsWe provide an overview of three approaches commonly used to monitor institutional performances for outlier detection. These are the common-mean model, the ‘Normal-Poisson’ random effects model and the ‘Logistic’ random effects model. For the latter we also propose a visualisation technique. The common-mean model assumes that the underlying true performance of all units is equal and that any observed variation between units is due to chance. Even after applying case-mix adjustment, this assumption is often violated due to overdispersion and a post-hoc correction may need to be applied. The random effects models relax this assumption and explicitly allow the true performance to differ between units, thus offering a more flexible approach. We discuss the strengths and weaknesses of each approach and illustrate their application using audit data from England and Wales on Adult Cardiac Surgery (ACS) and Percutaneous Coronary Intervention (PCI).ResultsIn general, the overdispersion-corrected common-mean model and the random effects approaches produced similar p-values for the detection of outliers. For the ACS dataset (41 hospitals) three outliers were identified in total but only one was identified by all methods above. For the PCI dataset (88 hospitals), seven outliers were identified in total but only two were identified by all methods. The common-mean model uncorrected for overdispersion produced several more outliers. The reason for observing similar p-values for all three approaches could be attributed to the fact that the between-hospital variance was relatively small in both datasets, resulting only in a mild violation of the common-mean assumption; in this situation, the overdispersion correction worked well.ConclusionIf the common-mean assumption is likely to hold, all three methods are appropriate to use for outlier detection and their results should be similar. Random effect methods may be the preferred approach when the common-mean assumption is likely to be violated.

Development of LSTM-MLR hybrid model for radar detector missing and outlier traffic volume correction

ABSTRACT Traffic volume data collected by radar detector contain inherent types of errors that are hard to observe. Therefore, the identification of missing data and outliers based on reliable verification volume data is essential, and the development of a methodology for correcting detector traffic data while considering various specificities at the detector installation sites is required. This study proposed a deep-learning-based long short-term memory–multiple linear regression (LSTM–MLR) hybrid model. First, the corrected detector traffic volume was calculated via the MLR model after the preprocessing of the missing and outlier detector traffic volumes. Second, the corrected radar detector traffic volume was learned via the LSTM model to predict the detector traffic volume for the target time periods. The results confirmed a correction effect in terms of radar detector traffic volume data at most of the 30 target sites. The results of this study provide three contributions. First, the missing and outlier traffic volume correction algorithm is easy to apply and can be applied to traffic volume data collected from various detectors. Second, the MLR model developed in this study derived a causal relationship between the traffic volume of the detector and the complex factors that could not reveal the obvious cause, such as the undetected small car driving next to a large vehicle and the radio transmission/reception problem. Third, complex neural networks and dropout techniques to avoid overfitting do not necessarily improve prediction accuracy.

Bootstrap Outlier Identification in Clinical Datasets for Lens Power Formula Constant Optimization

PurposeBootstrapping is a modern technique widely used in statistics to evaluate the performance of model parameters. The purpose of this study was to develop a strategy to identify and eliminate outliers in a dataset used for optimizing formula constants for lens power calculation.MethodsIn a dataset with N = 888 clinical cases treated with a monofocal aspherical intraocular lens (XC1/XY1, Hoya) constants for the SRKT, Haigis and Castrop formula were optimized and the prediction error PE calculated. The PE was bootstrapped NB = 1000 times, and the mean and trimmed mean of the bootstrapped PE were derived to generate the Bootlier plot showing the probability density function of the mean minus trimmed mean. With outliers this Bootlier plot shows some multimodality, and a Bootlier Index was extracted as a measure for multimodality. Outliers were removed from the tails of the PE distribution in a stepwise fashion until the Bootlier Index fell below a threshold of 0.001.ResultsWith the entire dataset the mean/SD/median/mean absolute/root mean squared PE using the optimized formula constants were -0.0045/0.44415/0.0134/0.3406/0.4412 dpt with SRKT, 0.0065/0.3711/-0.0056/0.2830/0.3710 dpt with Haigis, and 0.0034/0.3452/0.0023/0.2683/0.3451 dpt with the Castrop formula. After identifying and removing outliers the respective metrics for the PE were -0.0036/0.4028/0.0134/0.3205/0.4026 dpt for the SRKT (13 cases removed), 0.0050/0.3375/-0.0056/0.2656/0.3373 dpt with Haigis (11 cases removed), and 0.0035/0.3168/0.0023/0.2531/0.3166 dpt with Castrop (11 cases removed). The multimodality in the Bootlier plots was reduced from 0/0.1567/0.0587/0.0258/0.0007/0 with SRKT, 0/0.0981/0.0261/0.0202/0.0003/0 with Haigis, and 0.0006/0.0006/0.0161/0.0191/0.0005/0 with Castrop for the entire dataset to values below 1e-3 for trimming both tails of the PE distribution by ⅛, ¼, ½, 1, 2.5, and 5% respectively.ConclusionWe were able to prove that bootstrapping with outlier identification based on Bootlier plots and the Bootlier Index is a powerful tool to clean a dataset of outliers for formula constant optimization.

Identification of Medical Outliers with the use of Machine Learning

Machine learning has the potential to not only improve outcomes in healthcare data, but also to lessen the burden on healthcare workers. This algorithm has the potential to aid in the resolution of problems and the identification of new information crucial to the development of medical data. As part of our study, we provide a novel method for identifying outliers across many data sets. if one considers the possibility that information gleaned from medical records might shed light on the dynamics at play in an illness or person’s way of life. The presented approach uses a combination of supervised and unsupervised learning to achieve its goals. It’s probable that this programme might detect any discrepancies in the patient’s medical file. How well regional and international data sources cooperate to detect anomalies in medical records in real time. The model is used here regardless of whether or not it was trained and verified using actual patient data. The cleaning method takes full use of all the advantages of analogous processes. The utilisation of medical data sets in research is common. The arithmetical results demonstration that the machine learning-based approach to outlier identification is more accurate.

Outlier Identification Analysis of the General Administration Management Component of the Capital Expenditure of the Bangalore Municipal Corporation

Outlier Identification Analysis of the General Administration Management Component of the Capital Expenditure of the Bangalore Municipal Corporation

Abnormal Data Classification Based on SSA-AELSTM

Maintenance of production machinery and monitoring product quality through data analysis are the key problems with smart factories. There are far more normal values in the data collected for analysis during the production process than there are aberrant values. Data preparation uses a lot of energy before data analysis in order to make use of the data. In this research, we suggest a new hybrid technique that can perform anomaly detection more efficiently by merging two techniques, effective SSA techniques for reducing time series data and LSTM Autoencoder, which demonstrated great performance in time series outlier identification.

Human-assisted robotic detection of foreign object debris inside confined spaces of marine vessels using probabilistic mapping

Many complex vehicular systems, such as large marine vessels, contain confined spaces like water tanks, which are critical for the safe functioning of the vehicles. It is particularly hazardous for humans to inspect such spaces due to limited accessibility, poor visibility, and unstructured configuration. While robots provide a viable alternative, they encounter the same set of challenges in realizing robust autonomy. In this work, we specifically address the problem of detecting foreign object debris (FODs) left inside the confined spaces using a visual mapping-based system that relies on Mahalanobis distance-driven comparisons between the nominal and online maps for local outlier identification. The identified outliers, corresponding to candidate FODs, are used to generate waypoints that are fed to a mobile ground robot to take camera photos. The photos are subsequently labeled by humans for final identification of the presence and types of FODs, leading to high detection accuracy while mitigating the effect of recall–precision tradeoff. Preliminary simulation studies, followed by extensive physical trials on a prototype tank, demonstrate the capability and potential of our FOD detection system.

Efficient machine learning model to predict fineness, in a vertical raw meal of Morocco cement plant

Soft sensor enables computing parameters that can be physically impossible to measure. This work aims to develop a soft sensor for raw meal fineness in a vertical roller mill of a cement plant. In previous research, some key indicators of the process were ignored. It is well known that vertical raw meal is feeded with heterogenous material that impacts raw meal fineness, and therefore, can deteriorates prediction accuracy of the soft sensor. To deal with this constraint, one-year history of industrial data was exploited and processed by a robust multivariate outlier identification method, followed by process expert outlier verification. Then a particular attention was given to process and material quality parameters using a statistical method to test parameters significance on fineness based on data variance calculation. Eight process and four quality parameters were retained to develop robust nonlinear model using artificial neural network: Bayesian regularization and Levenberg Marquart, then compared with Support vector regression with different acquisition function and kernels to tune and select optimum model hyperparameters. The best model performance was obtained with the Bayesian regularization algorithm.

OUTLIER IDENTIFICATION ON PENALIZED SPLINE REGRESSION MODELING FOR POVERTY GAP INDEX IN JAVA

Java is one of the islands in Indonesia which has good establishment acceleration. Even though economic growth was good, poverty is still a serious problem. Three of six provinces, including DI Yogyakarta, Central Java, and East Java still have poverty rates above national rates in March 2020. This problem indicates that an imbalance in poverty happens between those regions. Several regions have extreme conditions or known as outliers. Besides that, poverty gap data have a complex pattern so modeling using a non-parametric approach is suitable. This study aims to build an appropriate model to support the success of poverty alleviation in Java and the identification of outliers was carried out using an adjusted boxplot. The best-penalized regression spline model for Poverty Gap Index in Java Island was obtained by Generalized minimum Cross-Validation (GCV) using optimum smoothing parameter (λ) 0,12 and knot combination (1, 2, 4, 1, 5, 3, and 1) for seven predictor variables. The result shows that penalized spline regression model has a higher R2 than OLS regression. The R2 is obtained 69,10%, so the model is feasible to explain the variability of the poverty gap in Java. Moreover, based on the outliers’ identification shows a dependency between outlier in data and residual because some districts/cities are identified as outliers in both.

A Self-Calibrating Localization Solution for Sport Applications with UWB Technology

This study addressed the problem of localization in an ultrawide-band (UWB) network, where the positions of both the access points and the tags needed to be estimated. We considered a fully wireless UWB localization system, comprising both software and hardware, featuring easy plug-and-play usability for the consumer, primarily targeting sport and leisure applications. Anchor self-localization was addressed by two-way ranging, also embedding a Gauss-Newton algorithm for the estimation and compensation of antenna delays, and a modified isolation forest algorithm working with low-dimensional set of measurements for outlier identification and removal. This approach avoids time-consuming calibration procedures, and it enables accurate tag localization by the multilateration of time difference of arrival measurements. For the assessment of performance and the comparison of different algorithms, we considered an experimental campaign with data gathered by a proprietary UWB localization system.

SpheroidAnalyseR-an online platform for analyzing data from 3D spheroids or organoids grown in 96-well plates.

Spheroids and organoids are increasingly popular three-dimensional (3D) cell culture models. Spheroid models are more physiologically relevant to a tumor compared to two-dimensional (2D) cultures and organoids are a simplified version of an organ with similar composition. Spheroids are often only formed from a single cell type which does not represent the situation in vivo. However, despite this, both spheroids and organoids can be used in cell migration studies, disease modelling and drug discovery. A drawback of these models is, however, the lack of appropriate analytical tools for high throughput imaging and analysis over a time course. To address this, we have developed an R Shiny app called SpheroidAnalyseR: a simple, fast, effective open-source app that allows the analysis of spheroid or organoid size data generated in a 96-well format. SpheroidAnalyseR processes and analyzes datasets of image measurements that can be obtained via a bespoke software, described herein, that automates spheroid imaging and quantification using the Nikon A1R Confocal Laser Scanning Microscope. However, templates are provided to enable users to input spheroid image measurements obtained by user-preferred methods. SpheroidAnalyseR facilitates outlier identification and removal followed by graphical visualization of spheroid measurements across multiple predefined parameters such as time, cell-type and treatment(s). Spheroid imaging and analysis can, thus, be reduced from hours to minutes, removing the requirement for substantial manual data manipulation in a spreadsheet application. The combination of spheroid generation in 96-well ultra-low attachment microplates, imaging using our bespoke software, and analysis using SpheroidAnalyseR toolkit allows high throughput, longitudinal quantification of 3D spheroid growth whilst minimizing user input and significantly improving the efficiency and reproducibility of data analysis. Our bespoke imaging software is available from https://github.com/GliomaGenomics. SpheroidAnalyseR is available at https://spheroidanalyser.leeds.ac.uk, and the source code found at https://github.com/GliomaGenomics.

Practical guide on chemometrics/informatics in x-ray photoelectron spectroscopy (XPS). II. Example applications of multiple methods to the degradation of cellulose and tartaric acid

Chemometrics/informatics, and data analysis in general, are increasingly important in x-ray photoelectron spectroscopy (XPS) because of the large amount of information (spectra/data) that is often collected in degradation, depth profiling, operando, and imaging studies. In this guide, we present chemometrics/informatics analyses of XPS data using a summary statistic (pattern recognition entropy), principal component analysis, multivariate curve resolution (MCR), and cluster analysis. These analyses were performed on C 1s, O 1s, and concatenated (combined) C 1s and O 1s narrow scans obtained by repeatedly analyzing samples of cellulose and tartaric acid, which led to their degradation. We discuss the following steps, principles, and methods in these analyses: gathering/using all of the information about samples, performing an initial evaluation of the raw data, including plotting it, knowing which chemometrics/informatics analyses to choose, data preprocessing, knowing where to start the chemometrics/informatics analysis, including the initial identification of outliers and unexpected features in data sets, returning to the original data after an informatics analysis to confirm findings, determining the number of abstract factors to keep in a model, MCR, including peak fitting MCR factors, more complicated MCR factors, and the presence of intermediates revealed through MCR, and cluster analysis. Some of the findings of this work are as follows. The various chemometrics/informatics methods showed a break/abrupt change in the cellulose data set (and in some cases an outlier). For the first time, MCR components were peak fit. Peak fitting of MCR components revealed the presence of intermediates in the decomposition of tartaric acid. Cluster analysis grouped the data in the order in which they were collected, leading to a series of average spectra that represent the changes in the spectra. This paper is a companion to a guide that focuses on the more theoretical aspects of the themes touched on here.

Hospital profiling using Bayesian decision theory.

For evaluating the quality of care provided by hospitals, special interest lies in the identification of performance outliers. The classification of healthcare providers as outliers or non-outliers is a decision under uncertainty, because the true quality is unknown and can only be inferred from an observed result of a quality indicator. We propose to embed the classification of healthcare providers into a Bayesian decision theoretical framework that enables the derivation of optimal decision rules with respect to the expected decision consequences. We propose paradigmatic utility functions for two typical purposes of hospital profiling: the external reporting of healthcare quality and the initiation of change in care delivery. We make use of funnel plots to illustrate and compare the resulting optimal decision rules and argue that sensitivity and specificity of the resulting decision rules should be analyzed. We then apply the proposed methodology to the area of hip replacement surgeries by analyzing data from 1,277 hospitals in Germany which performed over 180,000 such procedures in 2017. Our setting illustrates that the classification of outliers can be highly dependent upon the underlying utilities. We conclude that analyzing the classification of hospitals as a decision theoretic problem helps to derive transparent and justifiable decision rules. The methodology for classifying quality indicator results is implemented in an R package (iqtigbdt) and is available onGitHub.