Treatment Of Outliers Research Articles

A rigorous workflow and comparative analysis for accurate determination of vitrinite reflectance using data-driven approaches in the Persian Gulf region

Vitrinite reflectance (VR) is a critical measure of source rock maturity in geochemistry. Although VR is a widely accepted measure of maturity, its accurate measurement often proves challenging and costly. Rock–Eval pyrolysis offers the advantages of being cost-effective, fast, and providing accurate data. Previous studies have employed empirical equations and traditional machine learning methods using T-max data for VR prediction, but these approaches often yielded subpar results. Therefore, the quest to develop a precise method for predicting vitrinite reflectance based on Rock–Eval data becomes particularly valuable. This study presents a novel approach to predicting VR using advanced machine learning models, namely ExtraTree and XGBoost, along with new ways to prepare the data, such as winsorization for outlier treatment and principal component analysis (PCA) for dimensionality reduction. The depth and three Rock–Eval parameters (T-max, S1/TOC, and HI) were used as input variables. Three model sets were examined: Set 1, which involved both Winsorization and PCA; Set 2, which only included Winsorization; and Set 3, which did not include either. The results indicate that the ExtraTree model in Set 1 demonstrated the highest level of predictive accuracy, whereas Set 3 exhibited the lowest level of accuracy, confirming the methodology's effectiveness. The ExtraTree model obtained an overall R2 score of 0.997, surpassing traditional methods by a significant margin. This approach improves the accuracy and dependability of virtual reality predictions, showing significant advancements compared to conventional empirical equations and traditional machine learning methods.

A Comprehensive Study on the Role of Momentum in the Sports Games

This study aims to explore the effects of momentum and swing on tennis match results. The paper believes that momentum may be a key factor affecting player performance and match results. The research methods include data preprocessing, PCA dimensionality reduction, Pearson correlation coefficient analysis, logistic regression model and random forest model. Data preprocessing includes outlier treatment, one-hot encoding and normalization. PCA is used to reduce the dimension of match indicators and extract the main information. Pearson correlation coefficient analysis is used to test the correlation between match results and momentum. Logistic regression model is used to predict the probability of a player winning a point, and random forest model is used to predict match fluctuations. The study uses data from the 2023 US Open to validate the model. Sensitivity analysis is performed by changing model parameters to ensure model accuracy. The conclusion shows that momentum does affect match results, and our model can predict match performance and fluctuations with high accuracy. This is of great significance for coaches and athletes to formulate strategies and improve match performance. This study is of great value in understanding the momentum effect in tennis and how to use this knowledge to improve athlete performance.

ANN‐based analysis of the effect of SCM on recycled aggregate concrete

AbstractRising environmental awareness has prompted in‐depth studies on how the concrete industry affects the environment. Using recycled concrete aggregates (RCAs) and supplementary cementitious materials (SCMs) in concrete manufacturing provides advantages for sustainability. However, the broader chemical composition of SCMs and the inferior qualities of RCAs compared with natural aggregates (NAs) often lead to a decrease in concrete mechanical strength. The difficulty lies in foreseeing how the inclusion of SCMs and RCAs will affect the concrete compressive strength. The artificial neural network (ANN) approach presented herein can precisely forecast the recycled aggregate concrete (RAC) compressive strength, even when incorporates SCMs. The analysis employing the connection weight approach (CWA) determines how input variables influence compressive strength. Results indicate silica fume contributes most to compressive strength, followed by cement content, silica modulus, fine natural aggregate dosage, and coarse natural aggregate. Additionally, the amount of water utilized, the water/cement ratio, and the presence of RCA are all detrimental to compressive strength. The adverse effect of the cementitious materials' alumina modulus can be attributed to increased water demand during their reaction. Performance metrics of the final ANN model on the testing data subset include R2 = 0.94, and RMSE = 3.11, utilizing 834 data observations after outlier treatment for training and validation purposes. In summary, the ANN‐based approach demonstrates its efficacy in predicting concrete compressive strength when incorporating SCMs and RCAs, shedding light on the influential factors in concrete performance.

Tolerated outlier prediction method of excavation damaged zone thickness of drift based on interpretable SOA-QRF ensemble learning

ABSTRACT Drift excavation induces excavation damaged zones (EDZ) due to stress redistribution, impacting drift stability and rock deformation support. Predicting EDZ thickness is crucial, but traditional machine learning models are susceptible to potential outliers in dataset. Directly eliminating outliers, however, impacts training effectiveness. This study introduces an EDZ thickness prediction model utilising quantile loss and random forest (RF) optimised by the seagull optimisation algorithm (SOA), enabling median regression with tolerated outlier performance. 209 sets of data sets containing 34 mine borehole data were used to establish the prediction model. Evaluation using R 2, explained variance score (EVS), mean absolute error (MAE), and mean square error (MSE) demonstrates the superior accuracy of the proposed SOA-QRF model compared to traditional models. Based on the discussion on the treatment of outliers, the outcomes indicate that the SOA-QRF model is more suitable for the dataset with outliers as well as being able to effectuate tolerated outlier prediction. Additionally, three interpretation methods were utilised to explain the SOA-QRF model and enhance the transparency of the model’s prediction process and facilitating the analysis of dispatcher regulation.

Check your outliers! An introduction to identifying statistical outliers in R with easystats.

Beyond the challenge of keeping up to date with current best practices regarding the diagnosis and treatment of outliers, an additional difficulty arises concerning the mathematical implementation of the recommended methods. Here, we provide an overview of current recommendations and best practices and demonstrate how they can easily and conveniently be implemented in the R statistical computing software, using the {performance} package of the easystats ecosystem. We cover univariate, multivariate, and model-based statistical outlier detection methods, their recommended threshold, standard output, and plotting methods. We conclude by reviewing the different theoretical types of outliers, whether to exclude or winsorize them, and the importance of transparency. A preprint of this paper is available at: 10.31234/osf.io/bu6nt.

Data attribution and quantitative analysis for heart disease screening

Heart disease is a globally common fatal diseases, and its appearance has always been of great concern. The goal of this paper was to analyse factors of variation in the appearance of heart disease in Cleveland area with a view to provide a scientific basis for the prevention and comprehension of heart disease. This paper utilized data which includes the prevalence of heart disease in Cleveland region, which contains several characteristic variables such as age, gender, type of chest pain, etc., as well as the target variable, the prevalence of heart disease. First, the data were subjected to data pre-processing, including missing value treatment, outlier treatment, and feature engineering. Then, Granger causality test could be utilized to find out the correlation among features and target in order to distinguish the features that have a significant impact on the prevalence of heart disease. The outcome of the experiment showed that there was a major causal relationship between features such as ST depression led by oldpeak, slope of the highest exercise ST segment, a blood disorder called thalassemia (thal), and heart disease incidence. And different degrees of blood diseases may increase the risk of the disease. These results have significant implications for the progression of targeted heart disease prevention strategies and health management policies. Further studies can incorporate more features and data to enhance models precision and accuracy and interpretability, providing more valuable information for better understanding the pathogenesis of heart disease and predicting risk.

Measuring the Willingness-to-pay for standardizing medication information: a two-pronged stated preference approach

ABSTRACT When patients lack information about their medications, they are more likely to experience adverse events that may have costly health outcomes. One possible approach for improving the information that patients receive about their medications is to standardize the written information that pharmacies provide so that it is uniform and easier to read. While patients have expressed a preference for this standardization, no previous study has provided nationally representative estimates of its monetary benefits. In this paper, we estimate willingness to pay (WTP) to measure the potential benefits of standardizing the medication information leaflets that pharmacies provide to patients. We use data collected from an online, nationally representative, stated preference survey administered to non-institutionalized adults age 18 and over in the United States. Survey respondents were randomized to receive either open-ended valuation questions or referendum-style valuation questions. Responses from the open-ended questions suggest a mean WTP between approximately $3 and $13 per year depending on the treatment of outliers. Regression results suggest that those who received the referendum-style questions would be willing to pay, on average, just under $7 per year for the standardization.

Handling outliers in bankruptcy prediction models based on logistic regression

The primary tool for managing bank default risk is the credit rating of potential customers. The focus of the present study is on the logistic regression method used to construct 95% of the lender scorecards. The aim of the research is to determine how much the treatment of outliers improves the classification accuracy of the models when using a method that is highly sensitive to outliers, and which method of treating outliers results in the highest classification accuracy. Furthermore, what criteria should be used to determine the cut-off value of the models for a sample that does not contain solvent and insolvent businesses in equal proportions. The analysis was carried out on a sample of 1677 construction companies. The results show that the treatment of outliers significantly improves the predictive ability of the models, while the replacement of outliers with the closest non-outlier proved to be the most effective for treating outliers. When determining the cut-off, it is inappropriate to use the value that results in the highest classification accuracy, as this may lead to an increase in the proportion of first-order errors. The optimisation of this value may depend on the degree of credit risk taken by a given financial institution in its portfolio of loans.

Modelling biochemical oxygen demand in a large inland aquaculture zone of India: Implications and insights

Water quality surveillance is tough, and a specific timely management is necessary for the inland aquaculture ponds and ecology as well. Real time quality monitoring involves the study of numerous parameters includes physical (turbidity, temperature, and specific conductivity), chemical (pH, calcium, manganese, chlorides, iron, biochemical oxygen demand), and biological (bacteria and algae). It is also crucial to recognize the inter-dependence among the parameters. Alternatively, these relationships can be predicted with statistical and numerical modelling. Organic strength parameter 5-day biochemical oxygen demand (BOD) is a significant parameter to evaluate since its impact is very high on the quality of water, aquatic life, and other biological concerns. This study focuses on the prediction of BOD using six traditional and four boosting algorithms considering ten input physicochemical attributes. The attributes were fine-tuned for highly precise predictions by removing extreme values from the data set using data outlier treatment. The prediction results are compared using performance metrics such as coefficient of determination (R2), root mean square error (RMSE), mean square error (MSE), and mean absolute error (MAE). The findings revealed that boosting algorithms outperform the results of traditional models with the highest prediction accuracy. Among the boosting algorithms, eXtreme Gradient Boosting algorithm (XGBM) is found highly appropriate for the inland aquaculture waters with R2 = 0.95, RMSE = 0.31, MSE = 0.09, MAE = 0.1. Finally, this study provides a systematic evaluation of the BOD in the aquaculture waters and has a significant contribution to water management and eco-balance.

Smart Grid Outlier Detection Based on the Minorization-Maximization Algorithm.

Outliers can be generated in the power system due to aging system equipment, faulty sensors, incorrect line connections, etc. The existence of these outliers will pose a threat to the safe operation of the power system, reduce the quality of the data, affect the completeness and accuracy of the data, and thus affect the monitoring analysis and control of the power system. Therefore, timely identification and treatment of outliers are essential to ensure stable and reliable operation of the power system. In this paper, we consider the problem of detecting and localizing outliers in power systems. The paper proposes a Minorization-Maximization (MM) algorithm for outlier detection and localization and an estimation of unknown parameters of the Gaussian mixture model (GMM). To verify the performance of the method, we conduct simulation experiments by simulating different test scenarios in the IEEE 14-bus system. Numerical examples show that in the presence of outliers, the MM algorithm can detect outliers better than the traditional algorithm and can accurately locate outliers with a probability of more than 95%. Therefore, the algorithm provides an effective method for the handling of outliers in the power system, which helps to improve the monitoring analyzing and controlling ability of the power system and to ensure the stable and reliable operation of the power system.

Detection and treatment of outliers for multivariate robust loss reserving

AbstractTraditional techniques for calculating outstanding claim liabilities such as the chain-ladder are notoriously at risk of being distorted by outliers in past claims data. Unfortunately, the literature in robust methods of reserving is scant, with notable exceptions such as Verdonck & Debruyne (2011, Insurance: Mathematics and Economics, 48, 85–98) and Verdonck & Van Wouwe (2011, Insurance: Mathematics and Economics,49, 188–193). In this paper, we put forward two alternative robust bivariate chain-ladder techniques to extend the approach of Verdonck & Van Wouwe (2011, Insurance: Mathematics and Economics,49, 188–193). The first technique is based on Adjusted Outlyingness (Hubert & Van der Veeken, 2008. Journal of Chemometrics,22, 235–246) and explicitly incorporates skewness into the analysis while providing a unique measure of outlyingness for each observation. The second technique is based on bagdistance (Hubert et al., 2016. Statistics: Methodology, 1–23) which is derived from the bagplot; however; it is able to provide a unique measure of outlyingness and a means to adjust outlying observations based on this measure.Furthermore, we extend our robust bivariate chain-ladder approach to an N-dimensional framework. The implementation of the methods, especially beyond bivariate, is not trivial. This is illustrated on a trivariate data set from Australian general insurers and results under the different outlier detection and treatment mechanisms are compared.

Eye blinking, musical processing, and subjective states-A methods account.

Affective sciences often make use of self-reports to assess subjective states. Seeking a more implicit measure for states and emotions, our study explored spontaneous eye blinking during music listening. However, blinking is understudied in the context of research on subjective states. Therefore, a second goal was to explore different ways of analyzing blink activity recorded from infra-red eye trackers, using two additional data sets from earlier studies differing in blinking and viewing instructions. We first replicate the effect of increased blink rates during music listening in comparison with silence and show that the effect is not related to changes in self-reported valence, arousal, or to specific musical features. Interestingly, but in contrast, felt absorption reduced participants' blinking. The instruction to inhibit blinking did not change results. From a methodological perspective, we make suggestions about how to define blinks from data loss periods recorded by eye trackers and report a data-driven outlier rejection procedure and its efficiency for subject-mean analyses, as well as trial-based analyses. We ran a variety of mixed effects models that differed in how trials without blinking were treated. The main results largely converged across accounts. The broad consistency of results across different experiments, outlier treatments, and statistical models demonstrates the reliability of the reported effects. As recordings of data loss periods come for free when interested in eye movements or pupillometry, we encourage researchers to pay attention to blink activity and contribute to the further understanding of the relation between blinking, subjective states, and cognitive processing.

Exploring the multiverse: the impact of researchers' analytic decisions on relationships between depression and inflammatory markers.

In recent years, a replication crisis in psychiatry has led to a growing focus on the impact of researchers' analytic decisions on the results from studies. Multiverse analyses involve examining results across a wide array of possible analytic decisions (e.g., log-transforming variables, number of covariates, or treatment of outliers) and identifying if study results are robust to researchers' analytic decisions. Studies have begun to use multiverse analysis for well-studied relationships that have some heterogeneity in results/conclusions across studies.We examine the well-studied relationship between peripheral inflammatory markers (PIMs; e.g., white blood cell count (WBC) and C-reactive protein (CRP)) and depression severity in the large NHANES dataset (n = 25,962). Specification curve analyses tested the impact of 9 common analytic decisions (comprising of 58,000+ possible combinations) on the association of PIMs and depression severity. Relationships of PIMs and total depression severity are robust to analytic decisions (based on tests of inference jointly examining effect sizes and p-values). However, moderate/large differences are noted in effect sizes based on analytic decisions and the majority of analyses do not result in significant findings, with the percentage of analyses with statistically significant results being 46.1% for WBC and 43.8% for CRP. For associations of PIMs with specific symptoms of depression, some associations (e.g., sleep, appetite) in males (but not females) were robust to analytic decisions. We discuss how multiverse analyses can be used to guide research and also the need for authors, reviewers, and editors to incorporate multiverse analyses to enhance replicability of research findings.

Estimating reliability for response-time difference measures: Toward a standardized, model-based approach

AbstractA slowdown or a speedup in response times across experimental conditions can be taken as evidence of online deployment of knowledge. However, response-time difference measures are rarely evaluated on their reliability, and there is no standard practice to estimate it. In this article, we used three open data sets to explore an approach to reliability that is based on mixed-effects modeling and to examine model criticism as an outlier treatment strategy. The results suggest that the model-based approach can be superior but show no clear advantage of model criticism. We followed up these results with a simulation study to identify the specific conditions in which the model-based approach has the most benefits. Researchers who cannot include a large number of items and have a moderate level of noise in their data may find this approach particularly useful. We concluded by calling for more awareness and research on the psychometric properties of measures in the field.

Implementation of a multiverse approach in developmental research: The longitudinal relation between self‐esteem and peer victimization

AbstractResearch in social sciences has been questioned in the past decade due to the failure to replicate earlier work. To improve the trust in results, best practices that reduce data‐analytical flexibility have been proposed, such as preregistration. Another more recent such development regards the multiverse approach. The current study sought to demonstrate the benefits of applying the multiverse approach within developmental psychology. Using the relation between self‐esteem and peer victimization in adolescence as an example, we examined how both the concurrent as well as longitudinal association was impacted by data‐analytical decisions, such as the operationalization of variables, treatment of outliers, inclusion of covariates, and paths estimated in the model. Indeed, through the multiverse approach, we showed that and how these decisions matter for the model fit and estimates of the associations. We were thus able to illustrate that the multiverse approach increases transparency, makes it possible to examine the robustness of effects and can help advance theory building. Limitations of this approach as well as recommendations for other researchers are discussed.

Power Curve Modeling of Wind Turbines through Clustering-Based Outlier Elimination

The estimation of power curve is the central task for efficient operation and prediction of wind power generation. It is often the case, however, that the actual data exhibit a great deal of variations in power output with respect to wind speed, and thus the power curve estimation necessitates the detection and proper treatment of outliers. This study proposes a novel procedure for outlier detection and elimination for estimating power curves of wind farms by employing clustering algorithms of vector quantization and density-based spatial clustering of applications with noise. Testing different parametric models of power output curve, the proposed methodology is demonstrated for obtaining power curves of individual wind turbines in a Korean wind farm. It is asserted that the outlier elimination procedure for power curve modeling outlined in this study can be highly efficient at the presence of noises.

Outliers and momentum in the corporate bond market

How we filter outliers matters in empirical research. As a demonstration, we analyze how momentum returns respond to different outlier treatments in the corporate bond market TRACE database. We find that momentum profitability depends crucially on return outliers. Specifically, outlier trimming vanishes momentum returns, whereas winsorization yields a robust but conservative assessment of the momentum effect. Positive outliers are better than negative ones at identifying bonds that display return continuation and, thus, yield momentum gains. Price and volume-based sample treatments show that momentum gains stem from low-priced bonds and bonds solely traded by retail investors. Lastly, finer partitions of the bond cross-section deliver superior momentum gains without sacrificing portfolio diversification over bonds and issuers.

How people deal with …............................ outliers

AbstractPeople regularly make sense of distributions that are complicated by noise. How do individuals determine whether an outlying observation should be incorporated into one's understanding of the true distribution of the population or considered a fluke that ought to be disregarded? In a simple prediction task, we examine how individuals incorporate outliers and compare their behavior to various prescriptive models (e.g., averaging and tests of discordancy). We find that, on average, individuals do discount outlying values and that their outlier detection strategies approximate approaches that statisticians have recommended for Gaussian distributions, even when the observed distributions are not Gaussian. However, there are notable differences in treatment of outliers across individuals.

Archard model guided feature engineering improved support vector regression for rail wear analysis

This paper applied Archard wear law in feature engineering for the improvement of Support Vector Regression (SVR) performance and realized the rail steel wear behavior prediction. The actual complex rail wear multidimensional degradation information was obtained from field maintenance records over a decade and the hidden data outliers raised the modelling challenges. We applied pre-process technologies including feature importance analysis, physical model guided feature generation and outlier detection to build up the SVR based robust nonlinear regression analysis framework. Individual railway parameters’ effects on the wear process were investigated and revealed through model interpretation-based post analysis. This work provides a practical approach to deploying machine learning algorithms for rail service maintenance data analysis and treatment of data outliers.

Risk-Based Robust Statistical Learning by Stochastic Difference-of-Convex Value-Function Optimization

For the treatment of outliers, the paper “Risk-Based Robust Statistical Learning by Stochastic Difference-of-Convex Value-Function Optimization” by Junyi Liu and Jong-Shi Pang proposes a risk-based robust statistical learning model. Employing a variant of the conditional value-at-risk risk measure, called the interval conditional value-at-risk (In-CVaR), the model aims to exclude the risks associated with the left and right tails of the loss. The resulting nonsmooth and nonconvex model considers the population In-CVaR risk and distinguishes the upside and downside losses with asymmetric weights. For the solution of the model in both regression and classification, the authors show that the objective function is the difference of two convex functions each being the optimal objective value of a univariate convex stochastic program. A sampling and convex programming-based algorithm is developed with the appropriate control of incremental sample sizes, and its subsequential almost-sure convergence to a critical point is established. Numerical results illustrate the practical performance of the model and methodology.