Efficient Anomaly Detection Research Articles

Deep Learning-based Hybrid Model for Efficient Anomaly Detection

It is common among security organizations to run processes system call trace data to predict its anomalous behavior, and it is still a dynamic study region. Learning-based algorithms can be employed to solve such problems since it is typical pattern recognition problem. With the advanced progress in operating systems, some datasets became outdated and irrelevant. System calls datasets such as Australian Defense Force Academy Linux Dataset (ADFA-LD) are amongst the current cohort containing labeled data of system call traces for normal and malicious processes on various applications. In this paper, we propose a hybrid deep learning-based anomaly detection system. To advance the detection accurateness and competence of anomaly detection systems, Convolution Neural Network (CNN) with Long Short Term Memory (LSTM) is employed. The raw sequence of system call trace is fed to the CNN network first, reducing the traces' dimension. This reduced trace vector is further fed to the LSTM network to learn the sequences of the system calls and produce the concluding detection outcome. Tensorflow-GPU was used to implement and train the hybrid model and evaluated on the ADFA-LD dataset. Experimental results showed that the proposed method had reduced training time with an enhanced anomaly detection rate. Therefore, this method lowers the false alarm rates.

LightAnomalyNet: A Lightweight Framework for Efficient Abnormal Behavior Detection.

The continuous development of intelligent video surveillance systems has increased the demand for enhanced vision-based methods of automated detection of anomalies within various behaviors found in video scenes. Several methods have appeared in the literature that detect different anomalies by using the details of motion features associated with different actions. To enable the efficient detection of anomalies, alongside characterizing the specificities involved in features related to each behavior, the model complexity leading to computational expense must be reduced. This paper provides a lightweight framework (LightAnomalyNet) comprising a convolutional neural network (CNN) that is trained using input frames obtained by a computationally cost-effective method. The proposed framework effectively represents and differentiates between normal and abnormal events. In particular, this work defines human falls, some kinds of suspicious behavior, and violent acts as abnormal activities, and discriminates them from other (normal) activities in surveillance videos. Experiments on public datasets show that LightAnomalyNet yields better performance comparative to the existing methods in terms of classification accuracy and input frames generation.

Efficient anomaly detection from medical signals and images with convolutional neural networks for Internet of medical things (IoMT) systems.

Deep learning is one of the most promising machine learning techniques that revolutionalized the artificial intelligence field. The known traditional and convolutional neural networks (CNNs) have been utilized in medical pattern recognition applications that depend on deep learning concepts. This is attributed to the importance of anomaly detection (AD) in automatic diagnosis systems. In this paper, the AD is performed on medical electroencephalography (EEG) signal spectrograms and medical corneal images for Internet of medical things (IoMT) systems. Deep learning based on the CNN models is employed for this task with training and testing phases. Each input image passes through a series of convolution layers with different kernel filters. For the classification task, pooling and fully-connected layers are utilized. Computer simulation experiments reveal the success and superiority of the proposed models for automated medical diagnosis in IoMT systems.

Blockchain-Based Data Collection With Efficient Anomaly Detection for Estimating Battery State-of-Health

The number of electric vehicles in various countries has shown exponential growth so that the related industries to face the tremendous pressure of power batteries disposal. Efficient secondary use and recycling of power batteries require effective collection of battery data and reasonable estimation of battery state-of-health (SOH). In this paper, we propose a framework to collect battery charging data from different stakeholders with an anomaly detection method based on Isolation Forest with two features. Besides a score-based mechanism is adopted to do data screening and capture the data with good quality. Unlike prior works, our proposed method can exploit crowdsourced data to reduce the significant effort of battery data sensing and provide a data source scoring mechanism based on blockchain to improve the data quality and meet the requirement of reasonable estimation. In order to verify the effectiveness of the proposed collection method, a charge data test set is constructed based on the NASA battery data set. The simulation results indicate that the method increases the F-measure criteria up to 25.65% compared to the well-known anomaly detection algorithms. In addition, the proposed collection method outperforms the traditional method up to 10.9% in reducing the relative error when being used for SOH estimation.

Efficient Anomaly Detection for High-Dimensional Sensing Data With One-Class Support Vector Machine

This paper addresses the problem of anomaly detection for high-dimensional sensing data. The one-class support vector machine (OCSVM) is one of the most popular unsupervised methods for anomaly detection. When data are high dimensional and large scale, however, the efficiency of OCSVM-based methods in anomaly detection suffers. Although dimensionality-reduction tools, such as deep belief networks, can be applied to compress the high-dimensional data to alleviate the problem, the accuracy and timely detection are still hard to improve due to the inherent features of OCSVM. In this paper, we propose a new form of OCSVM model based on the structure of the compressed data and the characteristics of OCSVM. Based on the new model, we design both optimal and approximate methods for model training and testing. We evaluate the performance of our methods with extensive experiments on four real-world datasets. The experimental results demonstrate that our new methods, both optimal and approximate ones, not only significantly outperform the state-of-the-art in accuracy and efficiency, but also achieve the good performance without the need of manual parameter tuning. In addition, our approximate training and testing mechanism can reduce the computing time by three orders of magnitude with a negligible loss in accuracy.

DeL-IoT: A deep ensemble learning approach to uncover anomalies in IoT

Internet of Things (IoT) devices are inherently vulnerable due to insecure design, implementation, and configuration. Aggressive behavior changes, due to increased attacker’s sophistication, and the heterogeneity of the data in IoT have proven that securing IoT devices trigger multiple challenges. It includes complex and dynamic attack detection, data imbalance, data heterogeneity, real-time response, and prediction capability. Most researchers are not focusing on the class imbalance, dynamic attack detection, and data heterogeneity problems together in Software-Defined Networking (SDN) enabled IoT anomaly detection. Thus, to address these challenging tasks, we propose DeL-IoT, a deep ensemble learning framework for IoT anomaly detection and prediction using SDN, having three primary modules including anomaly detection, intelligent flow management, and device status forecasting. The DeL-IoT employs deep and stacked autoencoders to extract handy features for stacking into an ensemble learning model. This framework yields efficient detection of anomalies, manages flows dynamically, and forecasts both short and long-term device status for early action. We validate the proposed DeL-IoT framework with testbed and benchmark datasets. We demonstrate that in even a 1% imbalanced dataset, the performance of our proposed method, deep feature extraction with a deep ensemble learning model, is around 3% better than the single model. The extensive experimental results show that our models have a better and more reliable performance than the competing models showcased in the relevant related work.

Unsupervised Anomaly Detection in Stream Data with Online Evolving Spiking Neural Networks

Unsupervised anomaly discovery in stream data is a research topic with many practical applications. However, in many cases, it is not easy to collect enough training data with labeled anomalies for supervised learning of an anomaly detector in order to deploy it later for identification of real anomalies in streaming data. It is thus important to design anomalies detectors that can correctly detect anomalies without access to labeled training data. Our idea is to adapt the Online evolving Spiking Neural Network (OeSNN) classifier to the anomaly detection task. As a result, we offer an Online evolving Spiking Neural Network for Unsupervised Anomaly Detection algorithm (OeSNN-UAD), which, unlike OeSNN, works in an unsupervised way and does not separate output neurons into disjoint decision classes. OeSNN-UAD uses our proposed new two-step anomaly detection method. Also, we derive new theoretical properties of neuronal model and input layer encoding of OeSNN, which enable more effective and efficient detection of anomalies in our OeSNN-UAD approach. The proposed OeSNN-UAD detector was experimentally compared with state-of-the-art unsupervised and semi-supervised detectors of anomalies in stream data from the Numenta Anomaly Benchmark and Yahoo Anomaly Datasets repositories. Our approach outperforms the other solutions provided in the literature in the case of data streams from the Numenta Anomaly Benchmark repository. Also, in the case of real data files of the Yahoo Anomaly Benchmark repository, OeSNN-UAD outperforms other selected algorithms, whereas in the case of Yahoo Anomaly Benchmark synthetic data files, it provides competitive results to the results recently reported in the literature.

Efficient Anomaly Detection for Smart Hospital IoT Systems.

In critical Internet of Things (IoT) application domains, such as the Defense Industry and Healthcare, false alerts have many negative effects, such as fear, disruption of emergency services, and waste of resources. Therefore, an alert must only be sent if triggered by a correct event. Nevertheless, IoT networks are exposed to intrusions, which affects event detection accuracy. In this paper, an Anomaly Detection System (ADS) is proposed in a smart hospital IoT system for detecting events of interest about patients’ health and environment and, at the same time, for network intrusions. Providing a single system for network infrastructure supervision and e-health monitoring has been shown to optimize resources and enforce the system reliability. Consequently, decisions regarding patients’ care and their environments’ adaptation are more accurate. The low latency is ensured, thanks to a deployment on the edge to allow for a processing close to data sources. The proposed ADS is implemented and evaluated while using Contiki Cooja simulator and the e-health event detection is based on a realistic data-set analysis. The results show a high detection accuracy for both e-health related events and IoT network intrusions.

Oops! I Shrunk the Sample Covariance Matrix Again: Blockbuster Meets Shrinkage

Abstract Existing shrinkage techniques struggle to model the covariance matrix of asset returns in the presence of multiple-asset classes. Therefore, we introduce a Blockbuster shrinkage estimator that clusters the covariance matrix accordingly. Besides the definition and derivation of a new asymptotically optimal linear shrinkage estimator, we propose an adaptive Blockbuster algorithm that clusters the covariance matrix even if the (number of) asset classes are unknown and change over time. It displays superior all-around performance on historical data against a variety of state-of-the-art linear shrinkage competitors. Additionally, we find that for small- and medium-sized investment universes the proposed estimator outperforms even recent nonlinear shrinkage techniques. Hence, this new estimator can be used to deliver more efficient portfolio selection and detection of anomalies in the cross-section of asset returns. Furthermore, due to the general structure of the proposed Blockbuster shrinkage estimator, the application is not restricted to financial problems.

RETRACTED: Efficient anomaly detection in surveillance videos based on multi layer perception recurrent neural network

Surveillance frameworks actualized in true environment are strong in nature. As the environment is uncertain and dynamic, the surveillance turns out to be increasingly perplexing when contrasted with a static and controlled environment. Effective anomaly identification in the video surveillance is a difficult issue because of spilling, video noise, anomalies, and goals. This examination work proposes a background deduction approach dependent on Maximally Stable Extremal Region (MSER) highlight extraction technique with the ongoing profound learning structure of Multi-layer perception recurrent neural network (MLP-RNN) that is fit for distinguishing multiple objects of various sizes by pixel-wise foreground investigating framework. The proposed algorithm takes as information a reference (without anomaly) and an objective edge, both transiently adjusted, and outputs a segmentation guide of same spatial goals where the featured pixels meaning the recognized anomalies, which ought to be all the components not present in the reference outline. Besides, examine the advantages of various remaking strategies to the reestablish unique picture goals and exhibit the improvement of leftover designs over the littler and more straightforward models proposed by past comparable works. The simulation results are shows serious execution in the tried dataset, just as constant handling ability as compared with existing methods.

Efficient anomaly detection on sampled data streams with contaminated phase I data

Control chart algorithms aim to monitor a process over time. This process consists of two phases. Phase I, also called the learning phase, estimates the normal process parameters, then in Phase II, anomalies are detected. However, the learning phase itself can contain contaminated data such as outliers. If left undetected, they can jeopardize the accuracy of the whole chart by affecting the computed parameters, which leads to faulty classifications and defective data analysis results. This problem becomes more severe when the analysis is done on a sample of the data rather than the whole data. To avoid such a situation, Phase I quality must be guaranteed. The purpose of this paper is to introduce a new approach for applying EWMA chart to obtain accurate anomaly detection results over sampled data even if contaminations exist in Phase I. The new chart is applied to a real dataset, and its performance is evaluated on both sampled and not sampled data according to several criteria.

Efficient Anomaly Detection with Generative Adversarial Network for Breast Ultrasound Imaging.

We aimed to use generative adversarial network (GAN)-based anomaly detection to diagnose images of normal tissue, benign masses, or malignant masses on breast ultrasound. We retrospectively collected 531 normal breast ultrasound images from 69 patients. Data augmentation was performed and 6372 (531 × 12) images were available for training. Efficient GAN-based anomaly detection was used to construct a computational model to detect anomalous lesions in images and calculate abnormalities as an anomaly score. Images of 51 normal tissues, 48 benign masses, and 72 malignant masses were analyzed for the test data. The sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) of this anomaly detection model were calculated. Malignant masses had significantly higher anomaly scores than benign masses (p < 0.001), and benign masses had significantly higher scores than normal tissues (p < 0.001). Our anomaly detection model had high sensitivities, specificities, and AUC values for distinguishing normal tissues from benign and malignant masses, with even greater values for distinguishing normal tissues from malignant masses. GAN-based anomaly detection shows high performance for the detection and diagnosis of anomalous lesions in breast ultrasound images.

Hybrid Model for Efficient Anomaly Detection in Short-timescale GWAC Light Curves and Similar Datasets

Early warning during sky survey provides a crucial opportunity to detect low-mass, free-floating planets. In particular, to search short-timescale microlensing (ML) events from high-cadence and wide- field survey in real time, a hybrid method which combines ARIMA (Autoregressive Integrated Moving Average) with LSTM (Long-Short Time Memory) and GRU (Gated Recurrent Unit) recurrent neural networks (RNN) is presented to monitor all observed light curves and identify ML events at their early stages. Experimental results show that the hybrid models perform better in accuracy and less time consuming of adjusting parameters. ARIMA+LSTM and ARIMA+GRU can achieve improvement in accuracy by 14.5% and 13.2%, respectively. In the case of abnormal detection of light curves, GRU can achieve almost the same result as LSTM with less time by 8%. The hybrid models are also applied to MIT-BIH Arrhythmia Databases ECG dataset which has the similar abnormal pattern to ML. The experimental results from both data sets show that the hybrid model can save up to 40% of researchers' time in model adjusting and optimization to achieve 90% accuracy.

Efficient anomaly detection from medical signals and images

Anomaly detection is a very vital area in medical signal and image processing due to its importance in automatic diagnosis. This paper presents three efficient anomaly detection approaches for applications related to Electroencephalogram (EEG) signal processing and retinal image processing. The first approach depends on the utilization of Scale-Invariant Feature Transform (SIFT) for automatic seizure detection. The second one is based on the utilization of digital filters in a statistical framework for seizure prediction. Finally, an automated Diabetic Retinopathy (DR) diagnosis approach is presented based on the segmentation and detection of anomalous objects from retinal images. The presented simulation results reveal the success of the proposed techniques towards automated medical diagnosis.

Oops! I Shrunk the Sample Covariance Matrix Again: Blockbuster Meets Shrinkage

Existing shrinkage techniques struggle to model the covariance matrix of asset returns in the presence of multiple-asset classes. Therefore, we introduce a Blockbuster shrinkage estimator that clusters the covariance matrix accordingly. Besides the definition and derivation of a new asymptotically optimal linear shrinkage estimator we propose an adaptive Blockbuster algorithm that clusters the covariance matrix even if the (number of) asset classes are unknown and change over time. It displays superior all-around performance on historical data against a variety of state-of-the-art linear shrinkage competitors. Additionally, we find that for small and medium-sized investment universes the proposed estimator outperforms even recent nonlinear shrinkage techniques. Hence, this new estimator can be used to deliver more efficient portfolio selection and detection of anomalies in the cross-section of asset returns. Furthermore, due to the general structure of the proposed Blockbuster shrinkage estimator the application is not restricted to financial problems.

Factor Models for Portfolio Selection in Large Dimensions: The Good, the Better and the Ugly

Abstract This paper injects factor structure into the estimation of time-varying, large-dimensional covariance matrices of stock returns. Existing factor models struggle to model the covariance matrix of residuals in the presence of time-varying conditional heteroskedasticity in large universes. Conversely, rotation-equivariant estimators of large-dimensional time-varying covariance matrices forsake directional information embedded in market-wide risk factors. We introduce a new covariance matrix estimator that blends factor structure with time-varying conditional heteroskedasticity of residuals in large dimensions up to 1000 stocks. It displays superior all-around performance on historical data against a variety of state-of-the-art competitors, including static factor models, exogenous factor models, sparsity-based models, and structure-free dynamic models. This new estimator can be used to deliver more efficient portfolio selection and detection of anomalies in the cross-section of stock returns.

Гибридная модель для эффективного обнаружения аномалий в кратковременных последовательностях кривых блеска GWAC и аналогичных наборах данных

Early warning during sky survey provides a crucial opportunity to detect low-mass, free-floating planets. In particular, to search short-timescale microlensing (ML) events from high-cadence and wide- field survey in real time, a hybrid method which combines ARIMA (Autoregressive Integrated Moving Average) with LSTM (Long-Short Time Memory) and GRU (Gated Recurrent Unit) recurrent neural networks (RNN) is presented to monitor all observed light curves and identify ML events at their early stages. Experimental results show that the hybrid models perform better in accuracy and less time consuming of adjusting parameters. ARIMA+LSTM and ARIMA+GRU can achieve improvement in accuracy by 14.5% and 13.2%, respectively. In the case of abnormal detection of light curves, GRU can achieve almost the same result as LSTM with less time by 8%. Same models are also applied to MIT-BIH Arrhythmia Databases ECG dataset with similar abnormal pattern and we yield from both sets that we can reduce up to 40% of time consuming for researchers to adjust the model to 90% accuracy.

Factor Models for Portfolio Selection in Large Dimensions: The Good, the Better and the Ugly

This paper injects factor structure into the estimation of time-varying, large-dimensional covariance matrices of stock returns. Existing factor models struggle to model the covariance matrix of residuals in the presence of time-varying conditional heteroskedasticity in large universes. Conversely, rotation-equivariant estimators of large-dimensional time-varying covariance matrices forsake directional information embedded in market-wide risk factors. We introduce a new covariance matrix estimator that blends factor structure with time-varying conditional heteroskedasticity of residuals in large dimensions up to 1000 stocks. It displays superior all-around performance on historical data against a variety of state-of-the-art competitors, including static factor models, exogenous factor models, sparsity-based models, and structure-free dynamic models. This new estimator can be used to deliver more efficient portfolio selection and detection of anomalies in the cross-section of stock returns.

Detecting anomalies in time series data via a deep learning algorithm combining wavelets, neural networks and Hilbert transform

Design of a transferable time series anomaly detection method.Novel deep neural network structure facilitates learning short and long-term pattern interdependencies.Detection of anomalies in the Seismic Electrical Signal for predicting earthquake activity.Detection of road anomalies using smartphone data, facilitating crowdsourcing applications. The quest for more efficient real-time detection of anomalies in time series data is critically important in numerous applications and systems ranging from intelligent transportation, structural health monitoring, heart disease, and earthquake prediction. Although the range of application is wide, anomaly detection algorithms are usually domain specific and build on experts knowledge. Here a new signal processing algorithm inspired by the deep learning paradigm is presented that combines wavelets, neural networks, and Hilbert transform. The algorithm performs robustly and is transferable. The proposed neural network structure facilitates learning short and long-term pattern interdependencies; a task usually hard to accomplish using standard neural network training algorithms. The paper provides guidelines for selecting the neural network's buffer size, training algorithm, and anomaly detection features. The algorithm learns the system's normal behavior and does not require the existence of anomalous data for assessing its statistical significance. This is an essential attribute in applications that require customization. Anomalies are detected by analysing hierarchically the instantaneous frequency and amplitude of the residual signal using probabilistic Receiver Operating Characteristics. The method is shown to be able to automatically detect anomalies in the Seismic Electrical Signal that could be used to predict earthquake activity. Furthermore, the method can be used in combination with crowdsourcing of smartphone data to locate road defects such as potholes and bumps for intervention and repair.

Health Assessment of Railway Turnouts: A Case Study

Within railway infrastructure, railway point systems are among the most critical equipment, not only due to accidents and delays caused by their failures but also due to maintenance costs. The detection of early signs of degradation and the ability to identify the maintenance actions required to prevent a failure are key aspects of a successful and advantageous health assessment strategy. While studies focusing on the detection and prognostics of railway point systems exist, few or none address the correlation between environment, field layout and the point system behavior. This paper aims to consider the interaction between these factors and the point system behavior, and compare a fleet-based approach to an asset-based approach for the point systems health assessment, highlighting the influence of the field configuration on the effectiveness of the two methods. The proposed methods exploit Self-Organizing Maps (SOMs) to construct a health indicator for both the detection and the diagnosis of railway point systems. The approaches are applied to a case study for the on-line health assessment of 20 electro-mechanical point systems operating on a main line over the course of 6 months. The results show how an asset-based monitoring system is necessary in order to maintain a level of information which enables to achieve an efficient detection of anomalies and a correct identification of degradation mechanisms. In addition, fleet-based health assessment leads to a higher percentage of missed alarms, due to the intrinsic hypothesis of considering all point systems as operating in the same context and mission profile.