Outlier Detection Techniques Research Articles

Outlier Detection Technique for Wireless Sensor Network Using GAN with Autoencoder to Increase the Network Lifetime

In wireless sensor networks (WSN), sensor nodes are expected to operate autonomously in a human inaccessible and the hostile environment for which the sensor nodes and communication links are therefore, prone to faults and potential malicious attacks. Sensor readings that differ significantly from the usual pattern of sensed data due to faults in sensor nodes, unreliable communication links, and physical and logical malicious attacks are considered as outliers. This paper presents an outlier detection technique based on deep learning namely, generative adversarial networks (GANs) with autoencoder neural network. The two-level architecture proposed for WSN makes the proposed technique robust. The simulation result indicates improvement in detection accuracy compared to existing state-of-the-art techniques applied for WSNs and increase of the network lifetime. Robustness of outlier detection algorithm with respect to channel fault and robustness concerning different types of distribution of faulty communication channel is analyzed.

Robust spectrum sensing against malicious users using particle swarm optimization

The different issues in the wireless channel, such as multipath fading, shadowing, and hidden terminal problems, limit the sensing abilities of the single user in a cognitive radio network (CRN). In this connection, it is more suitable to follow cooperative spectrum sensing (CSS). However, the various categories of malicious users (MUs) pose threats to the performance of CSS. These MUs are eager to report false sensing data to the fusion center (FC) to deceive its decision. This work considered the CSS performance in the presence of always yes MU (AYMU), opposite MU (OMU), and always no MU (ANMU). Moreover, the considered particle swarm optimization (PSO) algorithm takes out reports of the MUs using one to many sensing distances (OTMSD) and z-score outlier detection techniques before the final decision. Simulation results validate the effectiveness of the proposed scheme in the presence of MUs.

A Robust Tuned Random Forest Classifier Using Randomized Grid Search to Predict Coronary Artery Diseases

Coronary artery disease (CAD) is one of the most authentic cardiovascular afflictions because it is an uncommonly overwhelming heart issue. The breakdown of coronary cardiovascular disease is one of the principal sources of death all over the world. Cardiovascular deterioration is a challenge, especially in youthful and rural countries where there is an absence of human-trained professionals. Since heart diseases happen without apparent signs, high-level detection is desirable. This paper proposed a robust and tuned random forest model using the randomized grid search technique to predict CAD. The proposed framework increases the ability of CAD predictions by tracking down risk pointers and learning the confusing joint efforts between them. Nowadays, the healthcare industry has a lot of data but needs to gain more knowledge. Our proposed framework is used for extracting knowledge from data stores and using that knowledge to help doctors accurately and effectively diagnose heart disease (HD). We evaluated the proposed framework over two public databases, Cleveland and Framingham datasets. The datasets were pre-processed by using a cleaning technique, a normalization technique, and an outlier detection technique. Secondly, the principal component analysis (PCA) algorithm was utilized to lessen the feature dimensionality of the two datasets. Finally, we used a hyperparameter tuning technique, randomized grid search, to tune a random forest (RF) machine learning (ML) model. The randomized grid search selected the best parameters and got the ideal CAD analysis. The proposed framework was evaluated and compared with traditional classifiers. Our proposed framework’s accuracy, sensitivity, precision, specificity, and f1-score were 100%. The evaluation of the proposed framework showed that it is an unrivaled perceptive outcome with tuning as opposed to other ongoing existing frameworks.

An Effective Heart Disease Prediction Framework based on Ensemble Techniques in Machine Learning

To design a framework for effective prediction of heart disease based on ensemble techniques, without the need of feature selection, incorporating data balancing, outlier detection and removal techniques, with results that are still at par with cutting-edge research. In this study, the Cleveland dataset, which has 303 occurrences, is used from the UCI repository. The dataset comprises 76 raw attributes, however only 14 of them are listed by the UCI repository as significant risk factors for heart disease when the dataset is uploaded as an open source dataset. Data balancing strategies, such as random over sampling, are used to address the issue of unbalanced data. Additionally, an isolation forest is used to find outliers in multivariate data, which has not been explored in previous research. After eliminating anomalies from the data, ensemble techniques such as bagging, boosting, voting, stacking are employed to create the prediction model. The potential of the proposed model is assessed for accuracy, sensitivity, and specificity, positive prediction value (PPV), negative prediction value (NPV), F1 score, ROC-AUC and model training time. For the Cleveland dataset, the performance of the suggested methodology is superior, with 98.73% accuracy, 98% sensitivity, 100% specificity, 100% PPV, 97% NPV, 1 as F score, and AUC as 1 with comparatively very less training time. The results of this study demonstrate that our proposed approach significantly outperforms the existing scholarly work in terms of accuracy and all the stated performance metrics. No earlier research has focused on these many performance parameters.

Intelligent Deep Learning Enabled Human Activity Recognition for Improved Medical Services

Human Activity Recognition (HAR) has been made simple in recent years, thanks to recent advancements made in Artificial Intelligence (AI) techniques. These techniques are applied in several areas like security, surveillance, healthcare, human-robot interaction, and entertainment. Since wearable sensor-based HAR system includes in-built sensors, human activities can be categorized based on sensor values. Further, it can also be employed in other applications such as gait diagnosis, observation of children/adult’s cognitive nature, stroke-patient hospital direction, Epilepsy and Parkinson’s disease examination, etc. Recently-developed Artificial Intelligence (AI) techniques, especially Deep Learning (DL) models can be deployed to accomplish effective outcomes on HAR process. With this motivation, the current research paper focuses on designing Intelligent Hyperparameter Tuned Deep Learning-based HAR (IHPTDL-HAR) technique in healthcare environment. The proposed IHPTDL-HAR technique aims at recognizing the human actions in healthcare environment and helps the patients in managing their healthcare service. In addition, the presented model makes use of Hierarchical Clustering (HC)-based outlier detection technique to remove the outliers. IHPTDL-HAR technique incorporates DL-based Deep Belief Network (DBN) model to recognize the activities of users. Moreover, Harris Hawks Optimization (HHO) algorithm is used for hyperparameter tuning of DBN model. Finally, a comprehensive experimental analysis was conducted upon benchmark dataset and the results were examined under different aspects. The experimental results demonstrate that the proposed IHPTDL-HAR technique is a superior performer compared to other recent techniques under different measures.

Geospatial Big Data Analytics for Quality Control of Surveys

Geospatial big data analytics allows survey quality control analysts to draw important conclusions about survey data quality that otherwise would take excessive time and resources. In this work, we explored two algorithmic methods that can help ensure reliability of survey interviews by detecting geospatial outliers. Focusing on geospatial data collected from surveys, we implemented outlier detection techniques with two different distance metrics to identify statistical anomalies in real-world datasets that may have qualitative interpretations. We found that one algorithm, which considers the local distribution of points in a dataset, identifies a different set of outliers when compared to another method, which considers the global distribution of points. Since there was a small overlap (10-19%) of flagged points between the two algorithms implemented, it may be helpful for analysts to focus on the fewer “outlier” points that are flagged by both methods rather than all the “outlier” points that are flagged by each algorithm. Finally, analysts should consider the computational costs, as the algorithms differ significantly.

Automated Identification of Overheated Belt Conveyor Idlers in Thermal Images with Complex Backgrounds Using Binary Classification with CNN.

Mechanical industrial infrastructures in mining sites must be monitored regularly. Conveyor systems are mechanical systems that are commonly used for safe and efficient transportation of bulk goods in mines. Regular inspection of conveyor systems is a challenging task for mining enterprises, as conveyor systems' lengths can reach tens of kilometers, where several thousand idlers need to be monitored. Considering the harsh environmental conditions that can affect human health, manual inspection of conveyor systems can be extremely difficult. Hence, the authors proposed an automatic robotics-based inspection for condition monitoring of belt conveyor idlers using infrared images, instead of vibrations and acoustic signals that are commonly used for condition monitoring applications. The first step in the whole process is to segment the overheated idlers from the complex background. However, classical image segmentation techniques do not always deliver accurate results in the detection of target in infrared images with complex backgrounds. For improving the quality of captured infrared images, preprocessing stages are introduced. Afterward, an anomaly detection method based on an outlier detection technique is applied to the preprocessed image for the segmentation of hotspots. Due to the presence of different thermal sources in mining sites that can be captured and wrongly identified as overheated idlers, in this research, we address the overheated idler detection process as an image binary classification task. For this reason, a Convolutional Neural Network (CNN) was used for the binary classification of the segmented thermal images. The accuracy of the proposed condition monitoring technique was compared with our previous research. The metrics for the previous methodology reach a precision of 0.4590 and an F1 score of 0.6292. The metrics for the proposed method reach a precision of 0.9740 and an F1 score of 0.9782. The proposed classification method considerably improved our previous results in terms of the true identification of overheated idlers in the presence of complex backgrounds.

Towards a deep learning-based outlier detection approach in the context of streaming data

Uncommon observations that significantly vary from the norm are referred to as outliers. Outlier detection, which aims to detect unexpected behavior, is a critical topic that has attracted significant attention in a wide range of research areas and application domains, including video surveillance, network intrusion detection, disease outbreak detection, and others. Deep learning-based techniques for outlier detection have currently outperformed machine learning and shallow approaches on streaming data, which are big and complicated datasets. Despite the fact that deep learning has been successfully applied in a variety of application domains, developing an effective and appropriate model is a difficult task due to the dynamic nature and variations of real-world applications and data. Hence, this research proposes a novel deep learning model based on a deep neural network (DNN) to handle the outlier detection problem in the context of streaming data. The proposed DNN model is developed with multiple hidden layers to improve feature abstraction and capabilities. Extensive experiments performed on four real-world outlier benchmark datasets, available at the UCI repository, and comparisons to state-of-the-art approaches are used to evaluate the proposed model's performance. Experiment results demonstrate that it outperforms both machine learning algorithms and deep learning competitors, resulting in significant performance gains. Particularly, when compared to other algorithms, the evaluation results clearly demonstrated the efficacy of the proposed approach, with much higher accuracy, recall and f1-score rates of 99.63%, 99.014% and 99.437%, respectively.

Intelligent prediction of rockburst in tunnels based on back propagation neural network integrated beetle antennae search algorithm.

Rockburst is one of the major engineering geological disasters of underground engineering. Accurate rockburst intensity level prediction is vital for disaster control during underground tunnel construction. In this work, a hybrid model integrating the back propagation neural network (BPNN) with beetle antennae search algorithm (BAS) has been developed for rockburst prediction. Before model building, 173 groups of rockburst dataset were collected. Six geological parameters are selected as predictors for rockburst, including the maximum tangential stress of the surrounding rock σθ, the uniaxial compressive strength of rock σc, the tensile strength of rock σt, the stress ratio σθ/σc, the rock brittleness ratio σc/σt, and the elastic energy index Wet. After preprocessed by outlier detection and synthetic minority oversampling technique (SMOTE), the new dataset was divided into training and test parts. BAS could optimize the weights and biases of BPNN from the training process. Then the established hybrid model was applied to the test samples with predicted accuracy of 94.3%, proving that the hybrid model has practical value in researching rockburst prediction.

Impact of Dimensionality Reduction on Outlier Detection: an Empirical Study.

Outlier detection is a fundamental data analytics technique often used for many security applications. Numerous outlier detection techniques exist, and in most cases are used to directly identify outliers without any interaction. Typically the underlying data used is often high dimensional and complex. Even though outliers may be identified, since humans can easily grasp low dimensional spaces, it is difficult for a security expert to understand/visualize why a particular event or record has been identified as an outlier. In this paper we study the extent to which outlier detection techniques work in smaller dimensions and how well dimensional reduction techniques still enable accurate detection of outliers. This can help us to understand the extent to which data can be visualized while still retaining the intrinsic outlyingness of the outliers.

SMART ENVIRONMENT INDEX PREDICTION OF SMART CITY USING POLYNOMIAL REGRESSION

Smart Environment refers to environment where pollution is detected, predicted, classified and solved using smart tools and technology such as using Internet of Things (IoT) sensors, cloud service, and machine learning algorithms. Episodic environment index prediction allows governments and local city agency to detect pollution in environment at a premature stage to initiate proper steps. Advancements in machine learning, sensor, and camera technology have endorsed us to do that, which can benefit millions of cities and its citizens indeed. This paper describes a new method for smart environment index prediction based on index of five others smart city components using polynomial regression. This research mainly works with the Smart Cities Index dataset. We attempted numerous steps to select feature and to filter its contents to make them more apposite for selected regression algorithm. Our approach unites several feature selection techniques, outlier detection technique, polynomial degree selection technique, and random state selection technique, which outcomes in a reduction of Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE) to 0.0000000140, 0.000000000000000322, and 0.0000000179 respectively and rise R2 to 1.0 for prediction. This method can be a valuable tool for smart environment index prediction from smartness of various components of a city, thus drastically reducing survey or other related works.

A Review on Outliers in IoT

In recent decades, the Internet of Things (IoT) has grown rapidly, attracting the attention of scientists and businesspeople. In extreme conditions, autonomously scattered sensor nodes pose a high risk of failure and intrusion into the IoT, skewing sensor values. Abnormal data, anomalies, or outliers are sensor values that depart from norms. When abnormalities are factored into data analytics, the ultimate judgment is affected. Using data-driven algorithms for IoT outlier detection is a cutting-edge tactic in Machine Learning (ML). However, evaluating the effectiveness of implemented ML techniques for outlier detection in IoT, which have the minimal processing power and power sources to ensure data quality, raises several difficulties that have just recently begun to be addressed in the academic literature. This paper analyses the cutting-edge architecture, type, degree, technique, and detection mode of AI and statistical outlier detection strategies in IoTs. Also, each of the ways to find outliers is talked about in detail, along with ways to make them better.

An Integrated Classification and Association Rule Technique for Early-Stage Diabetes Risk Prediction.

The number of diabetic patients is increasing yearly worldwide, requiring the need for a quick intervention to help these people. Mortality rates are higher for diabetic patients with other serious health complications. Thus, early prediction for such diseases positively impacts healthcare quality and can prevent serious health complications later. This paper constructs an efficient prediction system for predicting diabetes in its early stage. The proposed system starts with a Local Outlier Factor (LOF)-based outlier detection technique to detect outlier data. A Balanced Bagging Classifier (BBC) technique is used to balance data distribution. Finally, integration between association rules and classification algorithms is used to develop a prediction model based on real data. Four classification algorithms were utilized in addition to an a priori algorithm that discovered relationships between various factors. The named algorithms are Artificial Neural Network (ANN), Decision Trees (DT), Support Vector Machines (SVM), and K Nearest Neighbor (KNN) for data classification. Results revealed that KNN provided the highest accuracy of 97.36% compared to the other applied algorithms. An a priori algorithm extracted association rules based on the Lift matrix. Four association rules from 12 attributes with the highest correlation and information gain scores relative to the class attribute were produced.

Intrusion Detection in the IoT Under Data and Concept Drifts: Online Deep Learning Approach

Although the existing machine learning-based intrusion detection systems in the Internet of Things (IoT) usually perform well in static environments, they struggle to preserve their performance over time, in dynamic environments. Yet, the IoT is a highly dynamic and heterogeneous environment, leading to what is known as data drift and concept drift. Data drift is a phenomenon which embodies the change that happens in the relationships among the independent features, which is mainly due to changes in the data quality over time. Concept drift is a phenomenon which depicts the change in the relationships between input and output data in the machine learning model over time. To detect data and concept drifts, we first propose a drift detection technique that capitalizes on the principal component analysis (PCA) method to study the change in the variance of the features across the intrusion detection data streams. We also discuss an online outlier detection technique that identifies the outliers that diverge both from historical and temporally close data points. To counter these drifts, we discuss an online deep neural network (DNN) that dynamically adjusts the sizes of the hidden layers based on the Hedge weighting mechanism, thus enabling the model to steadily learn and adapt as new intrusion data come. Experiments conducted on an IoT-based intrusion detection data set suggest that our solution stabilizes the performance of the intrusion detection on both the training and testing data compared to the static DNN model, which is widely used for intrusion detection.

An Integration of Genetic Feature Selector, Histogram-Based Outlier Score, and Deep Learning for Wind Turbine Power Prediction

ABSTRACT During the last decades, the importance of clean energy resources is being increased. Wind is one of the most significant clean energy resources. Forecasting the output power of wind turbines is important for turbine control to improve power grids’ performance and maintenance. In this study, a novel method for predicting the power of wind turbines is proposed based on integrating data preprocessing, re-sampling, feature selection (genetic algorithm), and outlier detection (Histogram-Based Outlier Score) techniques to prepare the data for the deep learning (DL) algorithms. The results show that, after removing features chosen by the genetic algorithm (GA) method, the mean absolute error (MAE) reduced considerably to 333.7. Integrating Histogram-Based Outlier Score (HBOS) with genetic algorithm (GA) significantly decreased the MAE to 488. Comparing the results with benchmark machine learning algorithms, namely Light Gradient Boosting Machine (LGBM), Extreme Gradient Boosting Regression (×GBR), K-Nearest Neighbor (KNN), Support Vector Regression (SVR), and Recurrent Neural Networks (RNN) models, shows a remarkable improvement in the accuracy of turbine power prediction for about 78.7, 944.9, 104.7, 1456.6, and 17.1 in mean absolute error (MAE), respectively.

Automated data preparation for in vivo tumor characterization with machine learning.

BackgroundThis study proposes machine learning-driven data preparation (MLDP) for optimal data preparation (DP) prior to building prediction models for cancer cohorts.MethodsA collection of well-established DP methods were incorporated for building the DP pipelines for various clinical cohorts prior to machine learning. Evolutionary algorithm principles combined with hyperparameter optimization were employed to iteratively select the best fitting subset of data preparation algorithms for the given dataset. The proposed method was validated for glioma and prostate single center cohorts by 100-fold Monte Carlo (MC) cross-validation scheme with 80-20% training-validation split ratio. In addition, a dual-center diffuse large B-cell lymphoma (DLBCL) cohort was utilized with Center 1 as training and Center 2 as independent validation datasets to predict cohort-specific clinical endpoints. Five machine learning (ML) classifiers were employed for building prediction models across all analyzed cohorts. Predictive performance was estimated by confusion matrix analytics over the validation sets of each cohort. The performance of each model with and without MLDP, as well as with manually-defined DP were compared in each of the four cohorts.ResultsSixteen of twenty established predictive models demonstrated area under the receiver operator characteristics curve (AUC) performance increase utilizing the MLDP. The MLDP resulted in the highest performance increase for random forest (RF) (+0.16 AUC) and support vector machine (SVM) (+0.13 AUC) model schemes for predicting 36-months survival in the glioma cohort. Single center cohorts resulted in complex (6-7 DP steps) DP pipelines, with a high occurrence of outlier detection, feature selection and synthetic majority oversampling technique (SMOTE). In contrast, the optimal DP pipeline for the dual-center DLBCL cohort only included outlier detection and SMOTE DP steps.ConclusionsThis study demonstrates that data preparation prior to ML prediction model building in cancer cohorts shall be ML-driven itself, yielding optimal prediction models in both single and multi-centric settings.

Machine Learning Tool to Predict Student Categories After Outlier Removal

Statistical outlier detection techniques uses academic performance oriented results to find the truly brilliant as well as the weakest amongst a colony of students. Machine Learning allows further partitions within the remaining student community, based on both merit and personality. Present work proposes a decision tree model for predicting three more appropriate categories. It utilizes Text Analytic tools to assess student characteristic traits from their textual responses and feedbacks. The cream of the general pool is chosen to belong to a top class comprising the mentor group, provided they can academically assist the weaker of the lot. But all on the top may not be suited for mentor-ship role - textual assessment data delves to reveal character orientations favouring such decisions. The bulk who can manage their own forms the second class. The bottom of the pool benefits with assistance from the mentor group and comprise the third class.

A comparative study on outlier detection techniques for noisy production data from unconventional shale reservoirs

Decline curve analyses (DCA) and rate transient analyses (RTA) are widely used to characterize the fluid flow through porous media and forecast future production. Oil and gas production data are routinely analyzed for history matching and optimizing the well stimulation methods in hydrocarbon exploration and production lifecycle. However, outliers add significant uncertainty and non-uniqueness to results from production data analysis. This study provides a structured and comprehensive overview of five widely used outlier detection (OD) techniques for identifying and removing outliers in production data. Each OD technique measures deviation differently and, therefore, has a different outcome even when applied to the same dataset, creating the need to test several methods and find the optimal technique for identifying and removing outliers from production data. First, we generated production data from a typical multi-fractured horizontal well using a numerical reservoir simulator and added random noise to the data. Then, we used five different OD techniques to identify the prelabeled outliers from the synthetic production data. Finally, we identified the best-performing OD algorithm by comparing the various evaluation metrics such as the mean absolute error (MAE), precision, sensitivity, and F1 score. Results showed that the angle-based OD (ABOD) had the best MAE, precision, sensitivity, and F1 score of 8%, 85%, 98%, and 0.90, respectively. The next best-performing OD technique was distance-based OD (DBOD), with MAE, precision, sensitivity, and F1 score of 16%, 71%, 100%, and 0.83, respectively. We tested the ABOD method on several field production datasets by assuming different outlier thresholds (fraction of the data points likely to be outliers). Visual inspection of the processed data showed that the ABOD method effectively identified and removed the outliers from a relatively clean dataset (outlier threshold of 20%) and a highly noisy dataset (outlier threshold of 80%). This algorithm is intuitive and can effectively identify and remove outliers from the field production data to improve production forecasting, reserves estimation, and rate transient analysis for multi-fractured horizontal oil and gas reservoirs.

SVM-CRF Method and Box Plot Technique for Outlier Detection of Lidar Point Cloud

مجموعة داده‌های ابر نقاط لیدار و مدل‌های سه‌بعدی (3-D) در استخراج عوارض شهری، مدیریت جنگل‌داری، شهری و گردشگری، رباتیک، تولید بازی‌های رایانه‌ای و موارد دیگر کاربرد گسترده‌ای دارد. از سویی، وجود نقاط پرت در ابر نقاط لیدار اجتناب‌ناپذیر است؛ بنابراین تشخیص نقاط پرت و حذف آن‌ از ابر نقاط لیدار به‌منزلة گامی ضروری در پردازش ابر نقاط لیدار شناخته شده است. طی دهه‌های گذشته، چندین تکنیک تشخیص نقاط پرت در منابع این موضوع معرفی شده است اما بیشتر آنها از نظر زمانی هزینه‌برند و به نیروی متخصص انسانی نیاز دارند. به‌منظور کاهش این محدودیت‌ها، این مقاله رویکرد خودکار جدیدی برای تشخیص نقاط پرت، با استفاده از تکنیک میدان تصادفی شرطی برپایة ماشین بردار پشتیبان (SVM-CRF) و روش نمودار جعبه‌ای، معرفی کرده است. رویکرد نمودار جعبه‌ای بردار انرژی خروجی SVM-CRF را برای تشخیص نقاط پرت تجزیه و تحلیل می‌کند. این روش‌ به‌کمک مجموعه دادة محک ISPRS که برای مجموعه دادة وهینگن، با هدف طبقه‌بندی سه‌بعدی و بازسازی سه‌بعدی ساختمان ایجاد شده بود، ارزیابی شد. به‌منظور ارزیابی این روش، ابتدا نقاط پرتی به‌صورت دستی به مجموعه داده افزوده شد؛ با این تمرکز که این نقاط جزء نقاط پرت چسبیده به اشیا باشند. سپس مراحل تحقیق برای ارزیابی توانایی روش پیشنهادی در تشخیص نقاط پرت انجام شد. نتایج این تحقیق عملکرد مدل پیشنهادی را با دقت کلی 62% نشان داد. اگرچه الگوریتم RANSAC عملکردی بهتر از مدل پیشنهادی دارد، تکنیک زمان‌بر و پرهزینه‌تری در مقایسه با تکنیک تشخیص نقاط پرت پیشنهادی است.

Breaking the Curse of Class Imbalance: Bangla Text Classification

This article addresses the class imbalance issue in a low-resource language called Bengali. As a use-case, we choose one of the most fundamental NLP tasks, i.e., text classification, where we utilize three benchmark text corpora: fake-news dataset, sentiment analysis dataset, and song lyrics dataset. Each of them contains a critical class imbalance. We attempt to tackle the problem by applying several strategies that include data augmentation with synthetic samples via text and embedding generation in order to augment the proportion of the minority samples. Moreover, we apply ensembling of deep learning models by subsetting the majority samples. Additionally, we enforce the focal loss function for class-imbalanced data classification. We also apply the outlier detection technique, data resampling, and hidden feature extraction to improve the minority-f1 score. All of our experimentations are entirely focused on textual content analysis, which results in a more than90%minority f1 score for each of the three tasks. It is an excellent outcome on such highly class-imbalanced datasets.