Extreme Data Points Research Articles

Extreme-value analysis in nano-biological systems: applications and implications

AbstractExtreme value analysis (EVA) is a statistical method that studies the properties of extreme values of datasets, crucial for fields like engineering, meteorology, finance, insurance, and environmental science. EVA models extreme events using distributions such as Fréchet, Weibull, or Gumbel, aiding in risk prediction and management. This review explores EVA’s application to nanoscale biological systems. Traditionally, biological research focuses on average values from repeated experiments. However, EVA offers insights into molecular mechanisms by examining extreme data points. We introduce EVA’s concepts with simulations and review its use in studying motor protein movements within cells, highlighting the importance of in vivo analysis due to the complex intracellular environment. We suggest EVA as a tool for extracting motor proteins’ physical properties in vivo and discuss its potential in other biological systems. While there have been only a few applications of EVA to biological systems, it holds promise for uncovering hidden properties in extreme data, promoting its broader application in life sciences.

Application of the outlier detection method for web-based blood glucose level monitoring system

Recent advancements in biosensors have empowered individuals with diabetes to autonomously monitor their blood glucose levels through continuous glucose monitoring (CGM) sensors. Nevertheless, the data collected from these sensors may occasionally include outliers due to the inherent imperfections of the sensor devices. Consequently, the identification of these outliers is critical to determine whether blood glucose levels deviate significantly from the norm, necessitating further action. This study employs an outlier detection approach based on the 3-sigma method and the interquartile range (IQR), along with the application of the Winsorizing technique to correct the identified outliers. Additionally, a web-based system for visualizing blood glucose levels is developed, utilizing both outlier detection methods. In order to assess the system's performance, two types of testing are conducted: black box testing and load testing. The results of black box testing indicate that all test scenarios operate as anticipated. As for the load testing response times, it is observed that the 3-sigma visualization page loads an average of 606.75 milliseconds faster compared to the IQR visualization page. This study's outcomes are expected to enhance data quality, enhance the precision of analyses, and facilitate more informed decision-making by identifying and addressing extreme data points.

Addressing limitations of the K-means clustering algorithm: outliers, non-spherical data, and optimal cluster selection

<p>Clustering is essential in data analysis, with K-means clustering being widely used for its simplicity and efficiency. However, several challenges can affect its performance, including the handling of outliers, the transformation of non-spherical data into a spherical form, and the selection of the optimal number of clusters. This paper addressed these challenges by developing and enhancing specific models. The primary objective was to improve the robustness and accuracy of K-means clustering in the presence of these issues. To handle outliers, this research employed the winsorization method, which uses threshold values to minimize the influence of extreme data points. For the transformation of non-spherical data into a spherical form, the KROMD method was introduced, which combines Manhattan distance with a Gaussian kernel. This approach ensured a more accurate representation of the data, facilitating better clustering performance. The third objective focused on enhancing the gap statistic for selecting the optimal number of clusters. This was achieved by standardizing the expected value of reference data using an exponential distribution, providing a more reliable criterion for determining the appropriate number of clusters. Experimental results demonstrated that the winsorization method effectively handles outliers, leading to improved clustering stability. The KROMD method significantly enhanced the accuracy of converting non-spherical data into spherical form, achieving an accuracy level of 0.83 percent and an execution time of 0.14 per second. Furthermore, the enhanced gap statistic method outperformed other techniques in selecting the optimal number of clusters, achieving an accuracy of 93.35 percent and an execution time of 0.1433 per second. These advancements collectively enhance the performance of K-means clustering, making it more robust and effective for complex data analysis tasks.</p>

Feeding, mating and animal wellbeing: new insights from phylogenetic comparative methods.

Feeding, mating and animal wellbeing: new insights from phylogenetic comparative methods.

Comparison of two validated evidence-based medicine assessments: Do they correlate?

It was hypothesized that these instruments do not correlate between one another, based on inherent differences between them, including assessment format, grading method, and scoring range. The authors sought to examine whether a correlation between the results of these two instruments exists in a population of U.S. medical students. A retrospective cohort study of 158 fourth-year U.S. medical students in academic year 2018-2019 was conducted. All students were exposed to a focused EBM curriculum, consisting of three guided discussions of separate journal articles clinically relevant to the practice of emergency medicine. Outcomes measured included scores on both the ACE tool and Fresno test using descriptive statistics. Spearman's rho was used to determine the correlation between the ACE and Fresno scores for each student among the entire group. A subgroup analysis was performed to assess for correlations at more extreme data points. The median scores on the ACE tool and Fresno test were 66.7% and 62.7%. There was no statistically significant correlation between the results of these two assessments (Spearman's rho 0.023, p=0.774) in our population. The scores from the subgroup of advanced performers on the Fresno test showed a weak statistically significant positive correlation (p=0.045) to advanced scores on the ACE tool. No other subgroups showed statistically significant correlation. In our population of U.S. medical students, the results of two known EBM assessment instruments do not correlate with one another. The assessments may differ in what categories of learning they measure or in generalizability or perhaps in what depth of understanding they test overall. Further study is needed to determine what each instrument is measuring and whether there is demonstrable variation across populations.

Sinusoidal obstruction syndrome after liver transplantation: A multicenter observational study.

Sinusoidal obstruction syndrome after liver transplantation: A multicenter observational study.

The new normal? Redaction bias in biomedical science.

A concerning amount of biomedical research is not reproducible. Unreliable results impede empirical progress in medical science, ultimately putting patients at risk. Many proximal causes of this irreproducibility have been identified, a major one being inappropriate statistical methods and analytical choices by investigators. Within this, we formally quantify the impact of inappropriate redaction beyond a threshold value in biomedical science. This is effectively truncation of a dataset by removing extreme data points, and we elucidate its potential to accidentally or deliberately engineer a spurious result in significance testing. We demonstrate that the removal of a surprisingly small number of data points can be used to dramatically alter a result. It is unknown how often redaction bias occurs in the broader literature, but given the risk of distortion to the literature involved, we suggest that it must be studiously avoided, and mitigated with approaches to counteract any potential malign effects to the research quality of medical science.

How influential are COVID–19 data points? A fresh look at an estimated small scale DSGE model for the Philippines

Shocks emanating from the global pandemic continue to reshape the macroeconomic landscape—dimming national growth prospects, prolonging widespread financial distress among households, firms, and governments and heightening uncertainty. Using a small-scale New Keynesian Dynamic Stochastic General Equilibrium (DSGE) model for the Philippines, we examine the model’s sensitivity to COVID-19 datapoints or extreme observations. Relative to estimates during the base period (2002Q1 to 2019Q4), the inclusion of extreme datapoints worsens the model’s log data density progressively, from the consideration of the first quarter of 2020 to the full sample – an indication that shock propagation mechanisms associated with COVID–19 and other natural disasters should be integrated into the model. Even with the inclusion of said extreme observations, however, the model’s parameters are identified, provided identification schemes are evaluated at posterior median estimates. Judging from the sets of parameter estimates relative to the base sample, the effects of extreme observations are found to be non–uniform, especially the size of the shocks. But there are other parameters, notably those that are embedded in the Taylor rule, which are relatively as stable as some household related parameters. These results imply that the size of standard errors for demand, supply, and monetary policy shocks adjust to partially capture the impact of extreme datapoints.

Correlations between home sleep apnea tests and polysomnography outcomes do not fully reflect the diagnostic accuracy of these tests.

The clinical performance of home sleep apnea tests (HSATs) can be described by their (diagnostic) accuracy, defined as the percentage agreement with the obstructive sleep apnea severity category (normal, mild, moderate, and severe) based on polysomnography. Rather than reporting on accuracy, there has been a strong reliance in the literature to report correlation coefficients between the apnea-hypopnea index of HSATs and polysomnography to support claims of diagnostic performance. This is surprising, as it has been well described that correlation coefficients are inadequate to evaluate equivalence between 2 parameters. The aim of this study was to systematically investigate the magnitude of the discrepancies between correlation coefficients and diagnostic accuracy reported in or retrievable from HSAT validation studies. We compared the discrepancy between accuracy and apnea-hypopnea index correlation coefficients of all validation papers that met the inclusion criteria. A total of 20 papers were retained, representing a participant pool of 1,652. The weighted average apnea-hypopnea index correlation across all 20 papers was 0.82 and the weighted average accuracy was 0.61, highlighting a discrepancy of 0.21 and an overall misdiagnosis rate of 39%. The results of our study confirm the need for increased scientific rigor in selecting primary performance endpoints to support clinical performance claims of HSATs. Massie F, Van Pee B, Bergmann J. Correlations between home sleep apnea tests and polysomnography outcomes do not fully reflect the diagnostic accuracy of these tests. J Clin Sleep Med. 2022;18(3):871-876.

Robust Gaussian process regression based on iterative trimming

The Gaussian process (GP) regression can be severely biased when the data are contaminated by outliers. This paper presents a new robust GP regression algorithm that iteratively trims the most extreme data points. While the new algorithm retains the attractive properties of the standard GP as a nonparametric and flexible regression method, it can greatly improve the model accuracy for contaminated data even in the presence of extreme or abundant outliers. It is also easier to implement compared with previous robust GP variants that rely on approximate inference. Applied to a wide range of experiments with different contamination levels, the proposed method significantly outperforms the standard GP and the popular robust GP variant with the Student-t likelihood in most test cases. In addition, as a practical example in the astrophysical study, we show that this method can precisely determine the main-sequence ridge line in the color-magnitude diagram of star clusters.

Human ophthalmic artery as a sensor for non-invasive intracranial pressure monitoring: numerical modeling and in vivo pilot study

Intracranial pressure (ICP) monitoring is important in managing neurosurgical, neurological, and ophthalmological patients with open-angle glaucoma. Non-invasive two-depth transcranial Doppler (TCD) technique is used in a novel method for ICP snapshot measurement that has been previously investigated prospectively, and the results showed clinically acceptable accuracy and precision. The aim of this study was to investigate possibility of using the ophthalmic artery (OA) as a pressure sensor for continuous ICP monitoring. First, numerical modeling was done to investigate the possibility, and then a pilot clinical study was conducted to compare two-depth TCD-based non-invasive ICP monitoring data with readings from an invasive Codman ICP microsensor from patients with severe traumatic brain injury. The numerical modeling showed that the systematic error of non-invasive ICP monitoring was < 1.0 mmHg after eliminating the intraorbital and blood pressure gradient. In a clinical study, a total of 1928 paired data points were collected, and the extreme data points of measured differences between invasive and non-invasive ICP were − 3.94 and 4.68 mmHg (95% CI − 2.55 to 2.72). The total mean and SD were 0.086 ± 1.34 mmHg, and the correlation coefficient was 0.94. The results show that the OA can be used as a linear natural pressure sensor and that it could potentially be possible to monitor the ICP for up to 1 h without recalibration.

Cluster-Adjusted DEA Efficiency in the presence of Heterogeneity: An Application to Banking Sector

Abstract This paper improves on the issues of extreme data points and heterogeneity found in the linear programming data envelopment analysis (DEA) by presenting a cluster-adjusted DEA model (DEA with cluster approach). This analysis, based on efficiency, determines the number of clusters via Gap statistic and Elbow methods. We use the December quarterly panel data consisting of 122 U.S agricultural banks across 37 states from 2000 to 2017 to estimate the cluster-adjusted DEA model. Empirical results show differences in the estimated DEA efficiency measures with and without a clustering approach. Furthermore, using non-parametric tests, the results of Ansari-Bradley, Kruskal Wallis, and Wilcoxon Rank Sum tests suggest that the cluster-adjusted DEA model provides statistically better efficiency measures in comparison to the DEA model without a clustering approach.

An Electronic Screening and Brief Intervention for Hazardous and Harmful Drinking Among Swedish University Students: Reanalysis of Findings From a Randomized Controlled Trial Using a Bayesian Framework.

BackgroundDue to a resurgent debate on the misuse of P values, the Journal of Medical Internet Research is hosting a standing theme issue inviting the reanalysis of (primarily digital health) trial data using a Bayesian framework. This first paper in this series focuses on an electronic screening and brief intervention (eSBI), targeting harmful and hazardous alcohol consumption, which student health care centers across Sweden have routinely administerd to all students during the past decade. The second Alcohol Email Assessment and Feedback Study Dismantling Effectiveness for University Students (AMADEUS-2) trial aimed to assess the effect of the eSBI on alcohol consumption among students who were harmful and hazardous drinkers. A two-arm randomized controlled trial design was employed, randomizing eligible participants to either a waiting list or direct access to an eSBI. Follow-up assessments were conducted 2 months after randomization. Subsequent analysis of the trial followed the conventional null hypothesis approach, and no statistical significance was found between groups at follow-up with respect to the number of standard drinks consumed weekly. However, in an unspecified sensitivity analysis, it was discovered that removing three potential outliers made the difference between the groups significant.ObjectiveThe objective of this study is to reperform the primary and sensitivity analysis of the AMADEUS-2 trial using a Bayesian framework and to compare the results with those of the original analysis.MethodsThe same regression models used in the original analysis were employed in this reanalysis (negative binomial regression). Model parameters were given uniform priors. Markov chain Monte Carlo was used for Bayesian inference, and posterior probabilities were calculated for prespecified thresholds of interest.ResultsNull hypothesis tests did not identify a statistically significant difference between the intervention and control groups, potentially due to a few extreme data points. The Bayesian analysis indicated a 93.6% probability that there was a difference in grams of alcohol consumed at follow-up between the intervention and control groups and a 71.5% probability that the incidence rate ratio was <0.96. Posterior probabilities increased when excluding three potential outliers, yet such post hoc analyses were not necessary to show the preference toward offering an eSBI to harmful and hazardous drinkers among university students.ConclusionsThe null hypothesis framework relies on point estimates of parameters. P values can therefore swing heavily, depending on a single or few data points alone, casting doubt on the value of the analysis. Bayesian analysis results in a distribution over parameter values and is therefore less sensitive to outliers and extreme values. Results from analyses of trials of interventions where small-to-modest effect sizes are expected can be more robust in a Bayesian framework, making this a potentially better approach for analyzing digital health research data.Trial RegistrationInternational Standard Randomized Controlled Trial Number (ISRCTN) 02335307; http://www.isrctn.com/ISRCTN02335307

The power of outliers (and why researchers should ALWAYS check for them)

There has been much debate in the literature regarding what to do with extreme or influential data points. The goal of this paper is to summarize the various potential causes of extreme scores in a data set (e.g., data recording or entry errors, motivated mis-reporting, sampling errors, and legitimate sampling), how to detect them, and whether they should be removed or not. Another goal of this paper was to explore how significantly a small proportion of outliers can affect even simple analyses. The examples show a strong beneficial effect of removal of extreme scores. Accuracy tended to increase significantly and substantially, and errors of inference tended to drop significantly and substantially once extreme scores were removed. The presence of outliers can lead to inflated error rates and substantial distortions of parameter and statistic estimates when using either parametric or nonparametric tests (e.g., Zimmerman, 1994, 1995, 1998). Casual observation of the literature suggests that researchers rarely report checking for outliers of any sort. This inference is supported empirically by Osborne, Christiansen, and Gunter (2001), who found that authors reported testing assumptions of the statistical procedure(s) used in their studies--including checking for the presence of outliers--only 8% of the time. Given what we know of the importance of assumptions to accuracy of estimates and error rates, this in itself is alarming. There is no reason to believe that the situation is different in other social science disciplines. What are Outliers and Fringeliers and why do we care about them?

New computational methods for classification problems in the existence of outliers based on conic quadratic optimization

Most of the statistical research involves classification which is a procedure utilized to establish prediction models to set apart and classify new observations in the dataset from every fields of science, technology, and economics. However, these models may give misclassification results when dataset contains outliers (extreme data points). Therefore, we dealt with outliers in classification problem: firstly, by combining robustness of mean-shift outlier model and then stability of Tikhonov regularization based on continuous optimization method called Conic Quadratic Programming. These new methodologies are performed on classification dataset within the existence of outliers, and the results are compared with parametric model by using well-known performance measures.

Predicting soil carbon loss with warming.

Crowther et al. reported that the best predictor of surface soil carbon (top 10 cm) losses in response to warming is the size of the surface carbon stock in the soil (that is, carbon stocks in plots that have not been warmed), finding that soils that are high in soil carbon also lose more carbon under warming conditions. This relationship was based on a linear regression of soil carbon losses and soil carbon stocks in field warming studies, which was then used to project carbon losses over time and to generate a map of soil carbon vulnerability. However, a few extreme data points (high-leverage points) can strongly influence the slope of a regression line. Only 5 of the 49 sites analysed by Crowther et al.1 are in the upper half of the carbon stock range, which raises the possibility that the relationship they observed could be substantially altered by introducing data from sites with relatively high surface soil carbon stocks. There is a Reply to this Comment by Crowther, T. W. et al. Nature 554, 10.1038/nature25746 (2018).

Performance monitoring of a wind turbine using extreme function theory

A power curve relates the power produced by a wind turbine to the wind speed. Usually, such curves are unique to the various types of wind turbines, so that by monitoring the power curves, one may monitor the performance of the turbine itself. Most approaches to monitoring a system or a structure at a basic level, generally aim at differentiating between a normal and an abnormal state. Typically, the normal state is represented by a model, and then abnormal, or extreme data points are identified when they are compared to that model. This comparison is very often done pointwise on scalars in the univariate case, or on vectors, if multivariate features are available. Depending on the actual application, the pointwise approach may be limited, or highly prone to false identifications. This paper presents the use of extreme functions for the performance monitoring of wind turbines. Power curves from an actual wind turbine, are assessed as whole functions, and not individual datapoints, with the help of Gaussian process regression and extreme value distributions, with the ultimate aim of the performance monitoring of the wind turbine at a weekly resolution. The approach is compared to the more conventional pointwise method, and approaches which make use of multivariate features, and is shown to be superior in terms of the number of false identifications, with a significantly lower number of false-positives without sacrificing the sensitivity of the approach.

Archetypoid analysis for sports analytics

We intend to understand the growing amount of sports performance data by finding extreme data points, which makes human interpretation easier. In archetypoid analysis each datum is expressed as a mixture of actual observations (archetypoids). Therefore, it allows us to identify not only extreme athletes and teams, but also the composition of other athletes (or teams) according to the archetypoid athletes, and to establish a ranking. The utility of archetypoids in sports is illustrated with basketball and soccer data in three scenarios. Firstly, with multivariate data, where they are compared with other alternatives, showing their best results. Secondly, despite the fact that functional data are common in sports (time series or trajectories), functional data analysis has not been exploited until now, due to the sparseness of functions. In the second scenario, we extend archetypoid analysis for sparse functional data, furthermore showing the potential of functional data analysis in sports analytics. Finally, in the third scenario, features are not available, so we use proximities. We extend archetypoid analysis when asymmetric relations are present in data. This study provides information that will provide valuable knowledge about player/team/league performance so that we can analyze athlete’s careers.

Scalable and fast SVM regression using modern hardware

Support Vector Machine (SVM) regression is an important technique in data mining. The SVM training is expensive and its cost is dominated by: (i) the kernel value computation, and (ii) a search operation which finds extreme training data points for adjusting the regression function in every training iteration. Existing training algorithms for SVM regression are not scalable to large datasets because: (i) each training iteration repeatedly performs expensive kernel value computations, which is inefficient and requires holding the whole training dataset in memory; (ii) the search operation used in each training iteration considers the whole search space which is very expensive. In this article, we significantly improve the scalability and efficiency of SVM regression by exploiting the high performance of Graphics Processing Units (GPUs) and solid state drives (SSDs). Our key ideas are as follows. (i) To reduce the cost of repeated kernel value computations and avoid holding the whole training dataset in the GPU memory, we precompute all the kernel values and store them in the CPU memory extended by the SSD; together with an efficient strategy to read the precomputed kernel values, reusing precomputed kernel values with an efficient retrieval is much faster than computing them on-the-fly. This also alleviates the restriction that the training dataset has to fit into the GPU memory, and hence makes our algorithm scalable to large datasets, especially for large datasets with very high dimensionality. (ii) To enhance the performance of the frequently used search operation, we design an algorithm that minimizes the search space and the number of accesses to the GPU global memory; this optimized search algorithm also avoids branch divergence (one of the causes for poor performance) among GPU threads to achieve high utilization of the GPU resources. Our proposed techniques together form a scalable solution to the SVM regression which we call SIGMA. Our extensive experimental results show that SIGMA is highly efficient and can handle very large datasets which the state-of-the-art GPU-based algorithm cannot handle. On the datasets of size that the state-of-the-art GPU-based algorithm can handle, SIGMA consistently outperforms the state-of-the-art GPU-based algorithm by an order of magnitude and achieves up to 86 times speedup.

Compositional data for global monitoring: The case of drinking water and sanitation

IntroductionAt a global level, access to safe drinking water and sanitation has been monitored by the Joint Monitoring Programme (JMP) of WHO and UNICEF. The methods employed are based on analysis of data from household surveys and linear regression modelling of these results over time. However, there is evidence of non-linearity in the JMP data. In addition, the compositional nature of these data is not taken into consideration. This article seeks to address these two previous shortcomings in order to produce more accurate estimates. MethodsWe employed an isometric log-ratio transformation designed for compositional data. We applied linear and non-linear time regressions to both the original and the transformed data. Specifically, different modelling alternatives for non-linear trajectories were analysed, all of which are based on a generalized additive model (GAM). Results and discussionNon-linear methods, such as GAM, may be used for modelling non-linear trajectories in the JMP data. This projection method is particularly suited for data-rich countries. Moreover, the ilr transformation of compositional data is conceptually sound and fairly simple to implement. It helps improve the performance of both linear and non-linear regression models, specifically in the occurrence of extreme data points, i.e. when coverage rates are near either 0% or 100%.