Identification Of Outliers Research Articles

A meta-classification-based approach for outlier identification in GNSS networks

A meta-classification-based approach for outlier identification in GNSS networks

A robust distance-based approach for detecting multidimensional outliers

ABSTRACT Identifying outliers in data analysis is a critical task, as outliers can significantly influence the results and conclusions drawn from a dataset. This study explores the use of the Mahalanobis distance metric for detecting outliers in multivariate data, focusing on a novel approach inspired by the work of M. Falk, [On mad and comedians, Ann. Inst. Stat. Math. 49 (1997), pp. 615–644]. The proposed method is rigorously tested through extensive simulation analysis, where it demonstrates high True Positive Rates (TPR) and low False Positive Rates (FPR) when compared to other existing outlier detection techniques. Through extensive simulation analysis, we empirically evaluate the affine equivariance and breakdown properties of our proposed distance measure and it is evident from the outputs that our robust distance measure demonstrates effective results with respect to the measures FPR and TPR. The proposed method was applied to seven different datasets, showing promising true positive rates (TPR) and false positive rates (FPR), and it outperformed several well-known outlier identification approaches. We can effectively use our proposed distance measure in fields demanding outlier detection.

Empirical Inferences Under Bayesian Framework to Identify Cellwise Outliers

Outliers are typically identified using frequentist methods. The data are classified as “outliers” or “not outliers” based on a test statistic that measures the magnitude of the difference between a value and the majority part of the data. The threshold for a data value to be an outlier is typically defined by the user. However, a subjective choice of the threshold increases the uncertainty associated with outlier status for each data value. A cellwise outlier detection algorithm named FuzzyHRT is used to automate the editing process in repeated surveys. This algorithm uses Bienaymé–Chebyshev’s inequality and fuzzy logic to detect four different types of outliers resulting from format inconsistencies, historical, tail, and relational anomalies. However, fuzzy logic is not suited for probabilistic reasoning behind the identification of anomalous cells. Bayesian methods are well suited for quantifying the uncertainty associated with the identification of outliers. Although, as suggested by the literature, there exist well-developed Bayesian methods for record-level outlier detection, Bayesian methods for identifying outliers within individual records (i.e., at the cell level) remain unexplored. This paper presents two approaches from the Bayesian perspective to study the uncertainty associated with identifying outliers. A Bayesian bootstrap approach is explored to study the uncertainty associated with the output scores from the FuzzyHRT algorithm. Empirical likelihoods in a Bayesian setting are also considered for probabilistic reasoning behind the identification of anomalous cells. NASS survey data for livestock and major crop yield (such as corn) are considered for comparing the performances of the two proposed approaches with recent cellwise outlier methods.

No genetic causal relationship between lung function and osteoporosis ― evidence from a mendelian randomization study

For a long time, the decline in lung function has been regarded as a potential factor associated with the risk of osteoporosis (OP). Although several observational studies have investigated the relationship between lung function and OP, their conclusions have been inconsistent. Given that Mendelian randomization (MR) studies can help reduce the interference of confounding factors on outcomes, we adopted this approach to explore the causal relationship between lung function and OP at the genetic level. To investigate the potential causality between lung function (FVC, FEV1, FEV1/FVC, PEF) and OP, we conducted a MR analysis employing three approaches: inverse variance weighted (IVW), MR-Egger, and weighted median. We used Cochran’s Q test to detect potential heterogeneity, MR-Egger regression to evaluate directional pleiotropy, and the MR-PRESSO method to evaluate horizontal pleiotropy. In addition, we used MR-PRESSO and MR radial methods to exclude SNPs exhibiting pleiotropic outliers. Upon identification of potential outliers, we removed them and subsequently ran MR analysis again to assess the reliability of our findings. The MR analysis suggested that there was no causal effect of lung function (FVC, PEF, FEV1/FVC, FEV1) on OP, which is consistent with the. results after excluding potential outliers using MR-PRESSO and MR radial. methods. Sensitivity analysis confirmed the reliability and consistency of these. results. The study concluded that there is no causal link between lung function and OP. The association found in observational studies might be attributable to shared risk factors.

Computational modeling of air pollutants for aquatic risk: Prediction of ecological toxicity and exploring structural characteristics

Assessing the aquatic toxicity originating from air pollutants is essential in sustaining water resources and maintaining the ecosystem's safety. Quantitative structure-activity relationship (QSAR) models provide a computational tool for predicting pollutant toxicity, facilitating the identification/evaluation of the contaminants and identifying responsible structural fragments. One-vs-all (OvA) QSAR is a tailored approach to address multi-class QSAR problems. The study aims to determine five distinct levels of aquatic hazard categories for airborne pollutants using OvA-QSAR modeling containing 254 air contaminants. This QSAR analysis reveals the critical descriptors of air pollutants to target for molecular modification. Various factors, including the selection of relevant mechanistic descriptors, data quality, and outliers, determine the reliability of QSAR models. By employing feature selection and outlier identification approaches, the robustness and accuracy of our QSAR models were significantly increased, leading to more reliable predictions in chemical hazard assessment. The results revealed that models using the Random Forest algorithm performed the best based on the selected descriptors, with internal and external validation accuracy ranging from 71.90% to 97.53% and 76.47%–98.03%, respectively. This study indicated that the aquatic risk of air contaminants might be attributed predominantly to their sp3/sp2 carbon ratio, hydrogen-bond acceptor capability, hydrophilicity/lipophilicity, and van der Waals volumes. These structures can be critical in developing innovative strategies to mitigate or avoid the chemicals' harmful effects. Supporting air quality improvement, this study contributes to the rapid implementation of measures to protect aquatic ecosystems affected by air pollution.

Near‐Infrared Spectroscopy‐Based Chilled Fresh Lamb Quality Detection Using Machine Learning Algorithms

ABSTRACTTraditionally methods for assessing mutton quality rely on physical and chemical examination analyses that necessitate precise experimental environment conditions and specialized knowledge, often resulting in the compromise of the sample's structural integrity. To address these challenges, this study explores the application of near‐infrared spectroscopy (NIR) as a non‐destructive alternative for mutton quality evaluation, leveraging its operational simplicity, rapid analysis capabilities, and minimal requirement for technical expertise. Among various spectral data preprocessing techniques evaluated, multiple scattering correction (MSC) was found to significantly enhance model detection performance. Furthermore, principal component analysis (PCA) combined with the Mahalanobis Distance method was utilized for outlier identification. Finally, a mutton freshness detection model is constructed based on stacking ensemble learning, yielding an impressive accuracy rate of 0.976, outperforming other advanced approaches. In conclusion, our findings establish a robust theoretical framework for the rapid and non‐destructive assessment of meat freshness, contributing to advancements in meat quality detection.

Implementing Stacking with Cross - Validation for Heart Disease Prediction.

The objective of this work is to apply machine learning techniques for the prediction and early identification of cardiovascular disease, a major worldwide health problem. XGBoost, K-nearest neighbors (KNN), decision tree (DT), support vector machine (SVM), and StackingCVClassifier were among the ensemble algorithms used to anticipate cardiac disease using a dataset that came from the UCI ML database. To improve the quality of the data, an exploratory data analysis was performed on the dataset using methods including outlier identification and missing value imputation. Stacking CV Classifier attained the best accuracy rate of 92.20%, according to a comparative examination of pre-processing and post-processing findings. When compared to previous approaches, the suggested strategies performed better in terms of accuracy, recall, and f1-score. Moreover, the flexibility of the model indicates its possible application to other illnesses with comparable features.

Contrast-enhanced CT scan (CECT) for the detection of hollow viscus and mesenteric injuries in blunt trauma - an updated systematic review of the literature and meta-analysis of diagnostic test accuracy.

Despite improved awareness of blunt traumatic hollow viscus and mesenteric injuries (THVMI), the accuracy of contrast-enhanced CT (CECT) varies considerably among studies. This systematic review and meta-analysis of test accuracy aims to explore the diagnostic performance of CECT in detecting THVMI in blunt trauma. The study was conducted according to the Cochrane recommendations searching the PubMed, Scopus, and Cochrane Library datasets from 2000 to 8 September 2023 (PROSPERO ID: CRD42023473041). Surgical exploration, autopsy, and discharge from the hospital after monitoring were set as reference standard. To explore the diagnostic accuracy of CECT in detecting THVMI hierarchical models were developed. The risk of bias in individual studies was assessed with the QUADAS-2 tool. Sensitivity analysis was conducted to detect sources of heterogeneity. Twelve studies, for a total of 4537 patients, were deemed eligible. After identification of outliers and sensitivity analysis, the pooled sensitivity, specificity, positive and negative likelihood ratios, and diagnostic odds ratio were 0.85 (95% CI: 0.69-0.93), 0.94 (95% CI: 0.8-0.98), 14.65 (95% CI: 4.22-50.85), 0.16 (95% CI: 0.07-0.34), 92.3 (95% CI: 29.75-286.34), respectively. The Area under the HSROC curve was 0.95 (95% CI: 0.92-0.96). Meta-regression analysis identified the year of publication as a covariate significantly associated with heterogeneity. A high risk of bias was detected in the "patient selection" domains. CECT has a fundamental role in identifying THVMI with high specificity but suboptimal sensitivity. Clinical criteria are still of paramount importance, especially in cases of ambiguous initial CECT images.

Exploratory data analysis & EDS analysis on tool chip adhesion under dry and nano minimum quantity lubrication machining environment

Adhesion, abrasion, plowing and transverse displacement are the major causes of origin of friction. During dry machining of ductile materials, chips are fused with the rake face of the cutting tool and contribute to the formation of the built up edge. Built up edge protects the rake face of the tool from wear, but makes the machined surface rough. To improve the tribological properties and to decrease the tool-chip adhesion, cutting fluids are used. Cutting fluids are added to the machining zone to minimise cutting temperature and friction, but they result in environmental and health degradation. Surface texturing of tools is a potential way to modify the tribological properties of mating surfaces in order to increase the tribological qualities and decrease the tool-chip adhesion. The performance of surface textured tools depends upon orientation, dimple depth, width and pattern. Under lubrication regime, microholes in surface textured tools acts both as lubrication reservoir and traps wear debris to reduce abrasive wear. Nano Minimum Quantity Lubrication (nano-MQL) is an advanced lubrication technique used in machining processes, specifically designed to minimize the use of lubricants while maintaining effective lubrication. This method involves delivering an extremely small amount of high-performance lubricant directly to the cutting zone. The MoS2 lubricant with sunflower oil is typically in the form of nanodroplets, which are much smaller than conventional minimum quantity lubrication (MQL) droplets. The Exploratory Data Analysis (EDA) methodology enables the identification of patterns, trends, and outliers within the collected data, offering insights into the complex interplay of factors affecting the machining of Aluminium Alloy AA2024. At a spindle speed of 3500 rpm the feed force decreases by 37% under dry environment when compared with nano-MQL environment. Under dry environment average surface roughness reduces by 27% when compared with nano-MQL environment.

An outlier identification method based on parallel clustering and the Adaboost algorithm for power multi-source heterogeneous data

Abstract Influenced by the diversity of data scheduling and the multi-level differentiation of data distribution, the multi-source heterogeneous data of electric power have serious deviations in space and time scheduling, which makes it difficult to identify outliers. Therefore, a method for identifying outliers of multi-source heterogeneous data of electric power based on parallel clustering and the Adaboost algorithm is proposed. A sampling node model of heterogeneous data is constructed, and random numbers that obey Gaussian distribution are generated according to the mean value of data, and their mean square loss values are optimized to form an objective function of abnormal values of multi-source heterogeneous data of electric power. According to parallel clustering and the Adaboost algorithm, a unified format is created to obtain the load control results of abnormal values of multi-source heterogeneous data of electric power, so as to realize the identification of abnormal values of multi-source heterogeneous data of electric power. The experimental results show that at the time point of 18 s, the data identification quality of this method has reached 80, and with the increase of time, its data identification quality remains stable. It is demonstrated that the method has good recognition results.

Bioarchaeological Perspectives on Late Antiquity in Dalmatia: Paleogenetic, Dietary, and Population Studies of the Hvar—Radošević burial site

Late Antiquity Dalmatia was a time and place of political unrest in the Roman Empire that influenced the lives of those in that region. The Late Antique burial site of Hvar – Radošević, spanning the 3rd to 5th centuries CE, is located on the Croatian Dalmatian island of Hvar. Given the time frame and location on a busy marine trade route, the study of this burial site offers us a glimpse into the lives of the Late Antique population living on this island. It comprises 33 individuals, with 17 buried within a confined grave tomb and the remaining individuals buried in separate locations in the tomb's proximity. The study aims to provide a new perspective on the lives of people on the island during those times by studying ancestry, population structure, possible differences within the buried population, dietary habits, and general health. The genetic analysis of the ancestral origins of the individuals buried at Hvar – Radošević revealed a diverse population reflective of the era's genetic variability. The identification of genetic outliers suggests a range of ancestries from distinct regions of the Roman Empire, possibly linked to trade routes associated with the Late Antique port in ancient Hvar. Stable isotope ratio analysis (δ13C and δ15N) indicated a diet mainly consisting of C3 plants, with minimal consumption of marine foods. High childhood mortality rates, physiological stress markers, and dental diseases suggest a low quality of life in the population. Assessment of kinship and dietary patterns revealed no discernible distinctions between individuals buried within the tomb and those buried outside, indicative of an absence of differential burial practices based on social status and familial ties among this specific buried population.

Causality of inflammatory bowel disease and seborrheic keratosis: A bidirectional two-sample Mendelian randomization study.

Previous studies have revealed a potential link between inflammatory bowel disease (IBD) and seborrheic keratosis (SK). However, whether this association is causal or confounded remains unknown. We conducted this two-sample Mendelian randomization (TSMR) analysis to clarify bidirectional causality between IBD, including its two primary conditions Crohn's disease (CD) and ulcerative colitis (UC), and SK. The summary genetic data of IBD, CD, UC and SK were obtained from accessible genome-wide association studies (GWAS). This TSMR study was primarily performed using inverse-variance weighted (IVW) method, complemented by MR-Egger, weighted median (WM), Bayesian weighted MR (BWMR), MR-robust adjusted profile score (MR-RAPS), MR-pleiotropy residual sum and outlier (MR-PRESSO), and radial IVW MR analyses with modified second-order weights (IVW [Mod 2nd]) methods. Assessment of sensitivity and identification of potential outliers were subsequently conducted to aid interpretation of results. The forward MR results showed that IBD [odds ratio (OR)=1.068, 95% confidence interval (CI)=1.010-1.129, p=0.020) and its subtype CD (OR=1.088, 95%CI=1.038-1.139, p<0.001) increased the risk of SK. However, the occurrence of SK could not be affected by UC (OR=1.090, 95%CI=0.977-1.216, p=0.123). In the reverse analysis, no causal relationship between SK and IBD (OR=0.905, 95%CI=0.813-1.008, p=0.069), UC (OR=0.959, 95%CI=0.860-1.068, p=0.443), and CD (OR=0.933, 95%CI=0.846-1.029, p=0.165) was identified. These findings demonstrate that IBD and its subtype CD could increase the incidence of SK in European populations, whereas SK does not affect IBD occurrence.

Wastewater-based surveillance of SARS-CoV-2: Short-term projection (forecasting), smoothing and outlier identification using Bayesian smoothing

BackgroundDay-to-day variation in the measurement of SARS-CoV-2 in wastewater can challenge public health interpretation. We assessed a Bayesian smoothing and forecasting method previously used for surveillance and short-term projection of COVID-19 cases, hospitalizations, and deaths. MethodsSARS-CoV-2 viral measurement from the sewershed in Ottawa, Canada, sampled at the municipal wastewater treatment plant from July 1, 2020, to February 15, 2022, was used to assess and internally validate measurement averaging and prediction. External validation was performed using viral measurement data from influent wastewater samples from 15 wastewater treatment plants and municipalities across Ontario. ResultsPlots of SARS-CoV-2 viral measurement over time using Bayesian smoothing visually represented distinct COVID-19 “waves” described by case and hospitalization data in both initial (Ottawa) and external validation in 15 Ontario communities. The time-varying growth rate of viral measurement in wastewater samples approximated the growth rate observed for cases and hospitalization. One-week predicted viral measurement approximated the observed viral measurement throughout the assessment period from December 23, 2020, to August 8, 2022. An uncalibrated model showed underprediction during rapid increases in viral measurement (positive growth) and overprediction during rapid decreases. After recalibration, the model showed a close approximation between observed and predicted estimates. ConclusionBayesian smoothing of wastewater surveillance data of SARS-CoV-2 allows for accurate estimates of COVID-19 growth rates and one- and two-week forecasting of SARS-CoV-2 in wastewater for 16 municipalities in Ontario, Canada. Further assessment is warranted in other communities representing different sewersheds and environmental conditions.

Data Visualization Dashboard using Python

This study presents the development of a Data Visualization Dashboard using Python, aimed at providing comprehensive insights into complex datasets. Leveraging Python libraries such as Matplotlib, Seaborn, and Plotly, alongside frameworks like Dash and Streamlit, the dashboard offers an intuitive and interactive interface for data exploration and analysis. The visualization components include various charts, graphs, and maps tailored to depict diverse data types, facilitating the identification of patterns, trends, and outliers. Moreover, advanced features such as dynamic filtering, drill-down capabilities, and real-time updates enhance the dashboard's functionality, enabling users to delve deeper into the data and extract valuable insights efficiently. The dashboard's modular architecture ensures scalability and flexibility, allowing for seamless integration with different data sources and adaptability to evolving analytical requirements. Through case studies and performance evaluations, the effectiveness and usability of the Data Visualization Dashboard are demonstrated, highlighting its potential as a powerful tool for decision- making, reporting, and storytelling in diverse domains

Causal relationships between inflammatory cytokines and myopia: an analysis of genetic and observational studies.

This study aims to explore the causal relationship between inflammatory markers and myopia through the use of bidirectional Mendelian randomization (MR) and myopia animal models. The authors utilized data from a comprehensive and publicly accessible genome-wide association study (GWAS) for our analysis, which includes 460536 European ancestry control subjects and 37362 myopia patients. Utilizing a two-sample Mendelian randomization analysis framework, 27 inflammatory markers were investigated as exposure variables with myopia serving as the outcome variable. Nine MR analysis techniques were employed, with inverse-variance weighting (IVW) as the principal MR analysis method. Heterogeneity was assessed using Cochrane's Q test. The identification of single-nucleotide polymorphisms (SNPs) and outliers linked to myopia was achieved via MR-PRESSO. The expression of interleukin-2 (IL-2) in the vitreous of guinea pigs subjected to experimentally induced form-deprivation myopia (FDM) was examined. Elevated concentrations of IL-2 and IL-2ra were found to be associated [IVW estimate odds ratio (OR): 1.003, 95% CI: 1.001-1.005, P=0.001] and strongly associated (IVW estimate OR: 1.002, 95% CI: 1.000-1.003, P=0.049) with an increased risk of myopia, respectively. Conversely, lower levels of C-reactive protein (CRP) (IVW estimate OR: 0.996, 95% CI: 0.994-0.999, P=0.002) and tumour necrosis factor alpha (IVW estimate OR: 0.995, 95% CI: 0.994-0.996, P<0.001) were robustly linked to a heightened risk of myopia. IL-2 expression was notably upregulated in the vitreous of guinea pigs with experimentally induced FDM. Elevated levels of inflammatory factors, especially IL-2 and IL-2ra, have a potential causal relationship with myopia susceptibility, providing new insights into the pathogenesis of myopia.

Can Machine Learning Algorithms Contribute to the Initial Screening of Hip Prostheses and Early Identification of Outliers?

Can Machine Learning Algorithms Contribute to the Initial Screening of Hip Prostheses and Early Identification of Outliers?

Outlier identification and adjustment for time series

Identification and replacement of erroneous data is of fundamental importance for the quality of statistical surveys. If statistical units are continuously sampled over an extended period, time series methods can facilitate this task. Numerous outlier identification and replacement procedures are accessible for this particular purpose, like RegArima Approaches within the seasonal adjustment procedures in X13-Arima or Tramo/Seats. These algorithms can be used to identify different types of outliers, like additive outliers, level shifts or transitory changes. In this paper an alternative outlier identification procedure is proposed which is based on a nonlinear model estimated with support vector regressions. The focus of this procedure is on the identification of additive outliers and on the applicability for short time series with less than 3 years of observations.

Rainfall-Runoff Modeling Using Artificial Neural Network for Batu Pahat River Basin

This research delves into the effectiveness of Artificial Neural Networks with Multilayer Perceptron (ANN-MLP) and Nonlinear AutoRegressive with eXogenous inputs (NARX) models in predicting short-term rainfall-runoff patterns in the Batu Pahat River Basin. This study aims to predict river water levels using historical rainfall and river level data for future intervals of 1, 3, and 6 hours. Data preprocessing techniques, including the management of missing values, identification of outliers, and reduction of noise, were applied to enhance the accuracy and dependability of the models. This study assessed the performance of the models for ANN-MLP and NARX by comparing their effectiveness across various forecast timeframes and evaluating their performance in different scenarios. The findings of the study revealed that the ANN-MLP model showed robust performance in short-term prediction. On the contrary, the NARX model exhibited higher accuracy, particularly in capturing intricate temporal relationships and external impacts on river behavior. The ANN-MLP produces 99% accuracy for 1-hour prediction, and NARX yields 98% accuracy with 0.3245 Root Mean Squared Error and 0.1967 Mean Absolute Error. This study makes a valuable contribution to hydrological forecasting by presenting a rigorous and precise modeling methodology.

Methods of Assigning Labels to Detect Outliers

Outlier identification is a crucial field within data mining that focuses on identifying data points that significantly depart from other patterns in the data. Outlier identification may be categorized into formal and informal procedures. This article discusses informal approaches, sometimes known as labelling methods. The study focused on the analysis of real-time medical data to identify outliers using outlier labelling techniques. Various labelling approaches are used to calculate realistic situations in the dataset. Ultimately, using the anticipated outcomes of the outliers is a more suitable approach for addressing the needs of the larger populations.

The Environmental Impact on Economically Significant Traits in Central Anatolian Merino Sheep

This research delves into the pre-weaning growth and fleece characteristics of the Central Anatolian Merino sheep breed, focusing on Türkiye's National Community-Based Small Ruminant Breeding Program. The study encompasses Central Anatolian Merino lambs born between 2016 and 2021 across 22 farms in Ankara province, amassing a dataset of around 35,344 observations. Economically important traits such as birth weight (BW), weaning weight (WW), average daily weight gain (ADWG), Kleiber ratio at weaning (KR), fibre diameter, and fibre length (measured in approximately 4,809 observations) were scrutinized. Rigorous statistical analyses, including outlier identification, normality assessment, and the development of linear mixed models, were employed to unravel the impact of environmental factors on these traits. Significant findings emerged, indicating that birth weight, weaning weight, and the Kleiber ratio were substantially influenced by variables such as sex, birth type, birth season, birth year, and flock size. Moreover, average daily weight gain exhibited noteworthy variations attributed to gender, birth type, birth season, birth year, flock size, and other environmental factors, underscoring their collective impact on growth. Fleece traits displayed considerable diversity influenced by gender, birth type, birth season, birth year, and flock size. This study sheds light on the intricate interplay between environmental factors and pre-weaning growth traits within the Central Anatolian Merino sheep breed. Beyond its scientific contributions, this research provides valuable insights aimed at bolstering productivity. The adaptability of the Central Anatolian Merino breed to arid climates and challenging pasture conditions positions it as a key player in Türkiye's broader agricultural development.