Outlier Diagnosis Research Articles

Quantitative structure‐property relationship techniques for predicting carbon dioxide solubility in ionic liquids using machine learning methods

AbstractIonic liquids (ILs) are considered unique and attractive types of solvents with great potential to capture carbon dioxide (CO2) and reduce its emissions into the atmosphere. On the other hand, carrying out experimental measurements of CO2 solubility for each new IL is time‐consuming and expensive. Whereas, the possible combinations of cations and anions are numerous. Therefore, the preparation and design of such processes requires simple and accurate models to predict the solubility of CO2 as a greenhouse gas. In the present study, two different models, namely: artificial neural network (ANN) and support vector machine optimized with dragonfly algorithm (DA‐SVM) were used in order to establish a quantitative structure–property relationship (QSPR) between the molecular structures of cations and anions and the CO2 solubility. More than 10 116 CO2 solubility data measured in various ionic liquids (ILs) at different temperatures and pressures were collected. 13 significant PaDEL descriptors (E2M, MATS8S, TDB6I, TDB1S, ATSC4V, MATS8M, ATSC7V, Gats2S, Gats5S, Gats5C, ATSC6V, DE, and Lobmax), temperature and pressure were considered as the model input data. For the test set data (2023 data point), the estimated mean absolute error (MAE) and R2 for the ANN model are of 0.0195 and 0.9828 and 0.0219 and 0.9745 for the DA‐SVM model. The results obtained showed that both models can reliably predict the solubility of CO2 in ILs with a slight superiority of the ANN model. Examination of sensitivity and outlier diagnosis examinations confirmed that the QSPR model optimized using the ANN algorithm is better suited to correlate and predict this property.

Enhancing smartphone precise point positioning to sub-meter accuracy in suburban environments: a new stochastic model and outlier diagnosis

Enhancing smartphone precise point positioning to sub-meter accuracy in suburban environments: a new stochastic model and outlier diagnosis

Check your outliers! An introduction to identifying statistical outliers in R with easystats.

Beyond the challenge of keeping up to date with current best practices regarding the diagnosis and treatment of outliers, an additional difficulty arises concerning the mathematical implementation of the recommended methods. Here, we provide an overview of current recommendations and best practices and demonstrate how they can easily and conveniently be implemented in the R statistical computing software, using the {performance} package of the easystats ecosystem. We cover univariate, multivariate, and model-based statistical outlier detection methods, their recommended threshold, standard output, and plotting methods. We conclude by reviewing the different theoretical types of outliers, whether to exclude or winsorize them, and the importance of transparency. A preprint of this paper is available at: 10.31234/osf.io/bu6nt.

A LOF-IDW based data cleaning method for quality assessment in intelligent compaction of soils

Intelligent compaction (IC) is attracting increasing interest in construction engineering involving earthwork compaction. However, outliers may exist in data sets collected during IC, which can lead to misinterpretation of the compaction status of soils and may further result in erroneous results for compaction quality assessment. This study proposes a novel method for cleaning the measured data sets by combining the density-based local outlier factor (LOF) and inverse distance weighted (IDW) method. In the proposed LOF-IDW based method, the effect of spatial variation in soil properties is taken into account, while a boxplot-based method is proposed to dynamically determine the suitable threshold for outlier diagnosis. The capability and performance of the proposed method are verified against three data sets collected from construction sites. The results indicate that compared with the commonly used 3σ criterion, the proposed method not only exhibits a better performance in outlier diagnosis but also can rehabilitate the relevant data. In addition, the proposed method is demonstrated to be capable of significantly reducing the coefficient of variation of the measured data sets, which provides a more reliable support for the quality assessment and decision-making in IC.

Discrimination of Free-Range and Caged Eggs by Chemometrics Analysis of the Elemental Profiles of Eggshell.

As one of the foods commonly eaten all over the world, eggs have attracted more and more attention for their quality and price. A method based on elemental profiles and chemometrics to discriminate between free-range and caged eggs was established. Free-range (n1 = 127) and caged (n2 = 122) eggs were collected from different producing areas in China. The content of 16 elements (Zn, Pb, Cd, Co, Ni, Fe, Mn, Cr, Mg, Cu, Se, Ca, Al, Sr, Na, and K) in the eggshell was determined using a inductively coupled plasma atomic emission spectrometer (ICP-AES). Outlier diagnosis is performed by robust Stahel-Donoho estimation (SDE) and the Kennard and Stone (K-S) algorithm for training and test set partitioning. Partial least squares discriminant analysis (PLS-DA) and least squares support vector machine (LS-SVM) were used for classification of the two types of eggs. As a result, Cd, Mn, Mg, Se, and K make an important contribution to the classification of free-range and caged eggs. By combining column-wise and row-wise rescaling of the elemental data, the sensitivity, specificity, and accuracy were 91.9%, 91.1%, and 92.7% for PLS-DA, while the results of LS-SVM were 95.3%, 95.6%, and 95.1%, respectively. The result indicates that chemometrics analysis of the elemental profiles of eggshells could provide a useful and effective method to discriminate between free-range and caged eggs.

H‐type indices with applications in chemometrics II: h‐outlyingness index

AbstractAn outlier is generally considered as a data point that deviates from the “bulk” of all the data points. For outlier diagnosis, two questions could be asked: (1) How far is an object from the bulk? and (2) how many data points do the “bulk” include? To simultaneously deal with the above two questions, the h‐outlyingness index (HOI) is defined as suppose a given data point in a data set of N data points, if at most M% of all the (N − 1) one‐to‐rest distances is no less than M% of all the N(N − 1)/2 pairwise distances, the HOI value for the given data point will be M%. For applications, HOI was used for outlier diagnosis in simulated and real data sets, and the results were compared with those obtained by some robust statistical methods. Compared with the traditional methods, HOI gained similar results. For high‐dimensional data, it was wise to compute HOI based on dimension reduction methods such as principal component analysis (PCA). HOI was demonstrated to be a simple, easy‐to‐compute, robust and effective index for outlier diagnosis. Moreover, HOI is a nonparametric method that has no underlying assumptions on data distribution, which will be useful in chemometrics for multivariate outlier diagnosis.

1H NMR spectroscopy, one-class classification and outlier diagnosis: A powerful combination for adulteration detection in paprika powder

Paprika powder is a widely consumed high-priced product and therefore particularly prone to fraudulent practices. Nuclear magnetic resonance spectroscopy combined with one-class classification was applied for non-targeted detection of paprika adulteration and further chemometric tools were tested for diagnostic purposes. The 1H NMR spectra of 186 commercial paprika powders and 216 spiked samples were used to develop and comprehensively validate a data-driven soft independent modelling of class analogy model. The established one-class model yielded 92% sensitivity and exemplary adulterants were detected with 100% specificity at concentration levels of 0.1%, 0.1%, 10% and 20% by weight for Azorubine, Ponceau 4R, beetroot and sumac powder, respectively. After successful classification, visualization tools of robust principal component analysis and orthogonal partial least squares analysis were explored to uncover fingerprints of unusual (atypical and spiked) paprika powders. One-class classifiers based on 1H NMR spectroscopic data seem to be suitable for adulteration screening and, in combination with an outlier diagnosis, may improve prioritization of suspicious products for further confirmatory analysis in food authentication.

A simple multiple linear regression model in near infrared spectroscopy for soluble solids content of pomegranate arils based on stability competitive adaptive re-weighted sampling

The objective of this study was to develop a multiple linear regression (MLR) model using near infrared (NIR) spectroscopy combined with chemometric techniques for soluble solids content (SSC) in pomegranate samples at different storage periods. A total of 135 NIR diffuse reflectance spectra with the wavelength range of 950-1650 nm were acquired from pomegranate arils. Based upon sampling error profile analysis, outlier diagnosis was conducted to improve the stability of the model, and four outliers were removed. Several pretreatment and variable selection methods were compared using partial least squares (PLS) regression models. The overall results demonstrated that the pretreatment using the first derivative (1D) was very effective and the variable selection method of stability competitive adaptive re-weighted sampling (SCARS) was powerful for extracting feature variables. The equilibrium performance of 1D-SCARS-PLS regression model over ten repeats was similar to 1D-PLS regression model, so that the advantage of wavelength selection was inconspicuous in PLS regression model. However, the number of variables selected by 1D-SCARS was less than 9, which was enough to establish a simple MLR model. The performance of MLR model for SSC of pomegranate arils based on 1D-SCARS achieved a root-mean-square error of calibration of 0.29% and prediction of 0.31%. This strategy combining variable selection method with MLR may have a broad prospect in the application of NIR spectroscopy due to its simplicity and robustness.

Comparison of kriging, machine learning algorithms and classical thermodynamics for correlating the formation conditions for CO2 gas hydrates and semi-clathrates

The correlating capabilities of four machine learning methods, the ordinary kriging method, an adaptive neuro-fuzzy interference system (ANFIS), a multi-layer artificial neural network (ANN) and a Hybrid of Fuzzy logic and Genetic Algorithm (HFGA), as well as the thermodynamics-based approach of van der Waals-Platteeuw (vdWP) are compared for CO2 gas hydrates formed in the presence of thermodynamic promoters as well as for semi-clathrates formed from CO2. These systems were chosen for testing the three methods due to their potential relevance in CO2 capture and due to the expectation of them being computationally challenging. This is the first time that kriging has been tested for correlating gas hydrate equilibrium conditions. Different statistical indices, including the mean square error (MSE), an average absolute relative deviation (AARD), a correlation coefficient, and minimum and maximum errors, are employed to evaluate the performance of these methods. According to these performance indices, the ANFIS method performed the best among these methods; it predicted the equilibrium pressure with the highest accuracy. Finally, an outlier diagnosis is applied to the generated results to specify the reliability and uncertainty of the machine learning-based models. The simple-to-use machine learning tools are shown to be acceptable alternative to the vdWP methods and can be easily coupled with commercial simulation software to reduce calculation times while maintaining the accuracy.

RESEARCH ON GPS HEIGHT FITTING BASED ON LINEAR REGRESSION MODEL

Abstract. This paper mainly expounds the parameter estimation method, the outlier diagnosis and the establishment of the optimal regression equation in the linear regression model theory, the analysis of the principle of the polynomial fitting model, the derivation of the algorithm process, and the research on the accuracy evaluation method.The GPS survey area is fitted and calculated. The fitting model is analyzed and compared in detail. The better parameter values and regression equation models of the planar region are estimated. The fitting accuracy meets the requirements of the fourth level measurement, which can be used in actual engineering. Replace the fourth level measurement in the application.

An integrated spectroscopic strategy to trace the geographical origins of emblic medicines: Application for the quality assessment of natural medicines

Emblic medicine is a popular natural source in the world due to its outstanding healthcare and therapeutic functions. Our preliminary results indicated that the quality of emblic medicines might have an apparent regional variation. A rapid and effective geographical traceability system has not been designed yet. To trace the geographical origins so that their quality can be controlled, an integrated spectroscopic strategy including spectral pretreatment, outlier diagnosis, feature selection, data fusion, and machine learning algorithm was proposed. A featured data matrix (245 × 220) was successfully generated, and a carefully adjusted RF machine learning algorithm was utilized to develop the geographical traceability model. The results demonstrate that the proposed strategy is effective and can be generalized. Sensitivity (SEN), specificity (SPE) and accuracy (ACC) of 97.65%, 99.85% and 97.63% for the calibrated set, as well as 100.00% predictive efficiency, were obtained using this spectroscopic analysis strategy. Our study has created an integrated analysis process for multiple spectral data, which can achieve a rapid, nondestructive and green quality detection for emblic medicines originating from seventeen geographical origins.

Design and application of real-time monitoring system for service status of continuously welded turnout on the high-speed railway bridge

A turnout is a key piece of equipment and one of the weakest links of the railway infrastructures. Continuously welded turnouts (CWTs) have been widely laid on high-speed railway bridges because of the topographic and environmental limitations, where special attention is required to ensure the safety of the high-speed railways. A real-time monitoring system will be interesting for the service status of the CWT on the bridge, which was designed and established for the first time in China. Real-time data of the service status indicators were obtained by fiber grating and video perception technologies. The data features of the monitoring results were analyzed. A regression model and backpropagation neural network model were adopted to predict the key indicators. An alarm mechanism with a static threshold and outlier diagnosis was established. The monitoring system has been in good operation state for more than 4 years, effectively ensuring railway safety and security.

Diagnosis of Outliers and Cyber Attacks in Dynamic PMU-Based Power State Estimation

The paper introduces a robust method to detect random errors and cyber-attacks targeting alternating current (AC) dynamic state estimation through false data injection. The dynamic state estimator uses measurements that are delivered at a high reporting rate from synchrophasors or phasor measurement units (PMUs). A statistical outlier detection algorithm using the S-estimator is implemented on successive batch-mode regression representations of the extended Kalman filter (EKF). The S-estimator is a robust regression estimator that resists the effect of outliers and bad measurements. Theoretically, an S-estimator offers an improved detection of outliers compared with other methods implemented in the smart grid literature such as the Huber M-estimator and the Generalized Huber M-estimator (GM-estimator). The S-estimator has the advantage of offering a high breakdown point and can resist both observation outliers and leverage points. The introduced S-based EKF accurately tracks the fast variations of states when the PMU data is either clean or contaminated by bad measurements and cyber attacks. The proposed filter also accurately tracks the states when errors and cyber attacks target the topology of the grid. The simulations are conducted on the IEEE 14-bus system.

Fusion of elemental profiles and chemometrics: Discrimination of organic and conventional green teas

The feasibility of using elemental profiles and chemometrics for rapid discrimination of organic and conventional green teas was investigated in this work. Organic (n1 = 50) and ordinary (n2 = 50) green teas were collected from the same tea plant. The contents of 16 elements in green teas, including Al, Cr, Mg, Pb, Zn, Ca, Cu, Mn, Se, Cd, Fe, Na, Sr, Co, K and Ni, were analyzed using inductively coupled plasma atomic emission spectrometry (ICP-AES) after microwave-aided digestion with nitric acid. After outlier diagnosis by robust principal components analysis, the DUPLEX algorithm was performed to split the total data into training and test sets. Partial least squares discriminant analysis (PLSDA) and least squares support vector machine (LS-SVM) were used for classification of the two types of green teas. As a result, the sensitivity, specificity and accuracy were 0.90, 0.95, and 0.85 for PLSDA, and 0.975, 1.00, and 0.95 for LS-SVM, respectively. The results indicate that it is possible to rapidly distinguish organic and traditional green teas by combination of elemental profiles and chemometrics methods.

Similarity-aware data aggregation using fuzzy c-means approach for wireless sensor networks

For resource-constrained IoT systems, data collection is one of the fundamental operations to reduce the energy dissipation of sensor nodes and improve the network lifetime. However, an anomaly or deviation will exert a great influence on the quality of data collected, especially for a data aggregation scheme. By taking into account data-aware clustering and detection of anomalous events, a similarity-aware data aggregation using a fuzzy c-means approach for wireless sensor networks is proposed. Firstly, by using a fuzzy c-means approach, the clustering process can be performed to organize sensors into clusters based on data similarity. Next, an effective support degree function is defined for further outlier diagnosis. Afterwards, the appropriate weight of valid data can be obtained by taking advantage of the probability distribution characteristics of normal samples within a certain period. Finally, the aggregation result in the cluster can be estimated. Practical database-based simulations have confirmed that the proposed data aggregation method can achieve better performance than traditional methods in terms of data outlier detection accuracy and relative recovery error.

Application of a new approach for modeling the oil field formation damage due to mineral scaling

Mineral scaling has been considered a great concern for developing the oil production from the underground petroleum reservoirs. One of the main causes of this phenomenon is known as the chemical incompatibility of injected brine, frequently sea water, with the reservoir brine leading to the deposition of various supersaturated salts such as calcium carbonate, calcium sulfate and barium sulfate. In present communication, an evolutionary approach namely, Gene Expression Programming (GEP), was employed for rigorous modeling of formation damage by mineral scaling of mixed sulfate salt deposition. At first, a large databank of damaged permeability datapoints as a function of injected volume, injection flowrate, temperature, differential pressure and ionic concentrations of the existing chemical species in the porous media was employed. In this regard, a user-friendly correlation was extended for the first time by the aforementioned technique in the literature. Professional evaluation of the suggested GEP-based model was implemented by different statistical parameters and appealing visualization tools. Having proposed the GEP-based correlation, statistical parameters of the Average Absolute Relative Deviation Percent (AARD%) of 0.640% and determination coefficient (R2) of 0.984 was calculated. Accordingly, it is demonstrated that the proposed model has a superior performance and great potential for efficient prediction of damaged permeability due to the mixed sulfate salt scaling. Moreover, the implemented outlier diagnosis technique verified the validity of the databank used for modeling, as well as the high robustness of the suggested model was confirmed. In conclusion, the developed correlation in this work can be of enormous practical value for skillful engineers and scientists in any academic study and industrial applications dealing with mixed salt deposition.

Determination of Total Flavonoids Contents and Antioxidant Activity of Ginkgo biloba Leaf by Near-Infrared Reflectance Method.

Background Total flavonoids content (TFC) is one of the most important quality indexes of Ginkgo biloba leaf, and it is concerned with total antioxidant activity. Near-infrared spectroscopy (NIR) method has showed its advantages in fast, accurate, qualitative, and quantitative analysis of various components in many quality control researches. In this study, a calibration model was built by partial least squares regression (PLSR) coupling with NIR spectrum to quantitatively analyze the TFC and total antioxidant activity of Ginkgo biloba leaf. Results During the model establishing, some spectrum pretreatment and outlier diagnosis methods were optimized to establish the final model. The coefficients of determination (R2) for TFC and total antioxidant activity prediction were 0.8863 and 0.8486, respectively; and the root mean square errors of prediction (RMSEP) were 2.203 mg/g and 0.2211 mM/g, respectively. Conclusion These results showed that NIR method combined with chemometrics is suitable for quantitative analysis of main components and their activities and might be applied to quality control of relevant products.

Artificial neural network for the second virial coefficient of organic and inorganic compounds: An ANN for B of organic and inorganic compounds

ABSTRACTAn artificial neural network (ANN) to estimate the second virial coefficient, valid for organic and inorganic compounds, is presented. First, we statistically analyzed 6,531 experimental points, belonging to 234 fluids, collected from literature. The data were investigated with a factor analysis approach to identify the most significant parameters that influence the second virial coefficient. The factor analysis, combined with physical considerations, allowed to find four (Tr, Tc, Pc, ω) or five (μr) parameters as input variables for the ANN, according to the specific chemical family. The architecture of the proposed multi-layers perceptron (MLP) neural network consists of one input layer with five input variables (Tr, Tc, Pc, ω, μr), one output layer with one neuron (B) and two-hidden-layers with 19 neurons each. We trained, validated and tested several configurations of the neural network to obtain this network topology that minimizes the deviations between experimental and calculated points. Results show that the ANN is able to calculate the second virial coefficient with greater accuracy (RMSE = 29.38 cm3/mol) than that of correlations available in literature. To identify the outliers and applicability domain of the proposed MLP neural network, an outlier diagnosis based on the Leverage approach was performed. This analysis shows that the model is statistically valid.

Combinatorial Algorithm In Linear Model

Gauss-jacobi combinatorial algorithm is an alternative approach to traditional iterative numerical methods, which is primary oriented for parameter estimation in nonlinear models. The combinatorial algorithm is often exploited for outlier diagnosis in nonlinear models, where the other parameter estimation methods lose their efficiency. The paper describes comparison of both of gauss-jacobi combinatorial and gauss-markov models executed on parameter estimation process of levelling network for the reason to find the efficiency of combinatorial algorithm in simply linear model.

Fault and defect diagnosis of battery for electric vehicles based on big data analysis methods

This paper presents a novel fault diagnosis method for battery systems in electric vehicles based on big data statistical methods. According to machine learning algorithm and 3σ multi-level screening strategy (3σ-MSS), the abnormal changes of cell terminal voltages in a battery pack can be detected and calculated in the form of probability. Applying the neural network algorithm, this paper combines fault and defect diagnosis results with big data statistical regulation to construct a more complete battery system fault diagnosis model. Through analyzing the abnormalities hidden beneath the surface, researchers can see the design flaws in battery systems and provide feedback on the upstream of designing. Furthermore, the local outlier factor (LOF) algorithm and clustering outlier diagnosis algorithm are applied to verifying the calculation results. To further validate the effectiveness of the diagnosis method, a corresponding analysis between statistical diagnosis results and actual vehicle is given. To test the big data diagnosis model, the diagnosis results based on the actual vehicle operating data for the whole year is shown.