Anomaly Detection In Data Research Articles

A cognitive digital twin for process chain anomaly detection and bottleneck analysis

ABSTRACT Bottleneck detection and management plays a significant role in the context of Industry 4.0, wherein process chains have become more intricate. The dynamic nature of process chains shifts the bottleneck location, which requires an integrated methodology capable of identifying current as well as predicting future bottlenecks. The paper proposes a cognitive digital twin (CDT) with a novel explainable artificial intelligence (XAI) model. The proposed CDT is capable of (i) detecting existing bottlenecks, (ii) detecting data anomalies and process chain anomalies (iii) estimating shifting bottlenecks due to anomalies, (iv) predicting near future bottlenecks, and (v) the XAI model supports operational and strategic decision making. The usefulness of proposed CDT is demonstrated and validated experimentally on an industry 4.0 compliant learning factory. The proposed novel CDT effectively addresses the process chain bottlenecks (existing, shifting, and future) while the XAI model enhances transparency and trustworthiness for practical implementation.

Multi-Attack Intrusion Detection for In-Vehicle CAN-FD Messages.

As an enhanced version of standard CAN, the Controller Area Network with Flexible Data (CAN-FD) rate is vulnerable to attacks due to its lack of information security measures. However, although anomaly detection is an effective method to prevent attacks, the accuracy of detection needs further improvement. In this paper, we propose a novel intrusion detection model for the CAN-FD bus, comprising two sub-models: Anomaly Data Detection Model (ADDM) for spotting anomalies and Anomaly Classification Detection Model (ACDM) for identifying and classifying anomaly types. ADDM employs Long Short-Term Memory (LSTM) layers to capture the long-range dependencies and temporal patterns within CAN-FD frame data, thus identifying frames that deviate from established norms. ACDM is enhanced with the attention mechanism that weights LSTM outputs, further improving the identification of sequence-based relationships and facilitating multi-attack classification. The method is evaluated on two datasets: a real-vehicle dataset including frames designed by us based on known attack patterns, and the CAN-FD Intrusion Dataset, developed by the Hacking and Countermeasure Research Lab. Our method offers broader applicability and more refined classification in anomaly detection. Compared with existing advanced LSTM-based and CNN-LSTM-based methods, our method exhibits superior performance in detection, achieving an improvement in accuracy of 1.44% and 1.01%, respectively.

Research on noise reduction and data mining techniques for pavement dynamic response signals

Purpose This study aims to ensure reliable analysis of dynamic responses in asphalt pavement structures. It investigates noise reduction and data mining techniques for pavement dynamic response signals. Design/methodology/approach The paper conducts time-frequency analysis on signals of pavement dynamic response initially. It also uses two common noise reduction methods, namely, low-pass filtering and wavelet decomposition reconstruction, to evaluate their effectiveness in reducing noise in these signals. Furthermore, as these signals are generated in response to vehicle loading, they contain a substantial amount of data and are prone to environmental interference, potentially resulting in outliers. Hence, it becomes crucial to extract dynamic strain response features (e.g. peaks and peak intervals) in real-time and efficiently. Findings The study introduces an improved density-based spatial clustering of applications with Noise (DBSCAN) algorithm for identifying outliers in denoised data. The results demonstrate that low-pass filtering is highly effective in reducing noise in pavement dynamic response signals within specified frequency ranges. The improved DBSCAN algorithm effectively identifies outliers in these signals through testing. Furthermore, the peak detection process, using the enhanced findpeaks function, consistently achieves excellent performance in identifying peak values, even when complex multi-axle heavy-duty truck strain signals are present. Originality/value The authors identified a suitable frequency domain range for low-pass filtering in asphalt road dynamic response signals, revealing minimal amplitude loss and effective strain information reflection between road layers. Furthermore, the authors introduced the DBSCAN-based anomaly data detection method and enhancements to the Matlab findpeaks function, enabling the detection of anomalies in road sensor data and automated peak identification.

CutPaste-ROI: An Industrial Anomaly Data Detection Method based on Self-supervised Learning

CutPaste-ROI: An Industrial Anomaly Data Detection Method based on Self-supervised Learning

Deep learning-based optimization method for detecting data anomalies in power usage detection devices

Abstract In this paper, the self-attention layer of a graph convolutional neural network is first constructed to output the important information in the network structure. The migration learning network model is established, and the sample data are preprocessed and trained sequentially. The final processing results are used as the initial data for abnormal power consumption detection. Introduce Bayes’ theorem to optimize the hyperparameters of the model. The optimized model is applied in the abnormal power consumption detection system to identify abnormal power consumption events and provide specific processing solutions. Through the detection of the system, it was found that the voltage of the test user dropped from a 100V cliff to about 20V in late November, which was determined by the system to be a power consumption abnormality, and, therefore, an operation and maintenance order was issued. The site survey revealed that the data was in line with the system detection. Calculating the power consumption information of another user, the phase voltage of this user stays around 85-100V, far below 150V, so the undercounting of power is verified for the user, and the amount of power that should be recovered is 201.22kW.

Enterprise financial anomaly data detection method based on improved support vector machine

Enterprise financial anomaly data detection method based on improved support vector machine

Real-time anomaly data detection method based on mixed kernel function PSO-SVR

To avoid the influence of abnormal data of micro-plant factory sensors on the control system, an abnormal data detection method based on mixed kernel function particle swarm optimization (PSO)-support vector regression (SVR) is proposed. First, the mixed kernel function constructed by the polynomial kernel and radial basis kernel is used as the kernel function of SVR, and PSO is introduced to optimize the hyperparameters of the SVR model and establish a prediction model. Then, the model completes a step-by-step prediction based on the data in the sliding window and calculates confidence intervals. Finally, the data are judged to be abnormal based on whether the measured values exceed the confidence interval, and the abnormal values are replaced with the predicted values. The results show that the mean absolute percentage error of the model is 0.0063, the accuracy is 99.63%, the prediction accuracy and detection accuracy are better than the comparison model, and the abnormal data in the sensor data flow can be effectively detected and processed.

BADM-Net: Hierarchical Classification Network for Identifying Anomalous Trends in Bridge Monitoring Data Patterns

SHM systems have been widely implemented in long-span bridges, and seas of field measurement data have been accumulated. Due to the imperfect sensors, data transmission and acquisition, various anomalies inevitably exist in the SHM data, which may lead to unreliable structural condition assessment. Thus, an effective approach for detecting data anomalies is highly desirable. Due to the imbalanced data, some anomalous patterns are undertrained in popular end-to-end deep neural network models, resulting in a reduction in detection precision. In this paper, a hierarchical classification model with deep neural network tree is proposed for imbalanced data. The DNN tree contains three levels: (1) CNN to divide seven types of data into four categories (134, 2, 5, 67), denoted as C4; (2) two DNNs to classify to two classes separately (1, 34, 6, 7), denoted as D2D2; (3) DNNs to classify to two classes (3, 4). So, the DNN tree is presented as C4_D2D2_D2. The DNN tree is an open framework and can be defined based on the data characteristics. In the data processing, three data sets are built for training, namely single-channel data set, dual-channel data set and statistical data set. To validate our work, we considered the effects of balanced and imbalanced training sets and training ratios. The results show that our model can detect the multi-pattern anomalies of SHM data efficiently with 95.5% high accuracy. Besides, the proportion of abnormal data classified to normal data has been reduced, especially 3-minor. This model successfully solves the problem in a simple and easy to understand way, which has certain reference significance for the bridge structure anomaly judgment in the future.

Detecting Data Anomalies from Their Formal Specifications: A Case Study in IoT Systems

We present in this paper a new method in detecting anomalies in datasets representing systems behaviour, which is based on comparing a dataset to the data blueprint of the system representing its normal behaviour. This method removes some of the need for applying complex machine learning algorithms that aim at detecting abnormalities in such datasets and gives a more assured outcome of the presence of abnormalities. Our method first models a system using the formal language of the π-calculus, and then applies an abstract interpretation that ultimately generates an abstract multiset representing the messages exchanged in the system model. We term this multiset as the data blueprint of the system, and it represents the normal behaviour expected. We apply this method to the case of a recent study in literature, which attempts to analyse normal and abnormal behaviour in datasets representing runs of the MQTT protocol, both under attack and no attack conditions. We show that our method is able to detect these conditions in an easier and more straightforward manner than the original case study attempts to.

Simultaneous detection for multiple anomaly data in internet of energy based on random forest

Anomaly Data Detection plays a core role in defensing cyber-attacks, ensuring reliability, protecting security and improving robustness in smart grid. This study constructs a fast method to detect multi-node anomaly data including outlier and implicit characteristics. The outliers are determined according historical datasets, while implicit features invoked are summed up from satisfying with operation requirements of smart grid. For accomplishment of simultaneous detection for multiple abnormal data, Improved Random Forest Algorithm (IRFA) is proposed through modifying the following aspects: data set reconstruction, bootstrap sampling, decision tree generation and majority voting. In addition, parallel strategies are designed to improve algorithmic efficiency. Plenty of simulations are used to evaluate performance of the developed method through plenty of loads information from various transformers. Here not only explore the impact of various features and decision trees on algorithm’s performances, but also investigate the correspondence between node number and decision tree. And the accuracy in this study is compared with that of BP Neural Network and Support Vector Machine. Simulations demonstrate that here the algorithm has excellent detection accuracy and efficiency, i.e., the detection accuracy for single-node is 97%, while that for multiple-node are higher than 95%. It is versatile and universal, and very suitable for abnormal data identification from multiple-node in smart grid with the cyber–physical systems interactions.

An Approach for Detecting Data Anomalies at Permanent Cycling Count Stations

An Approach for Detecting Data Anomalies at Permanent Cycling Count Stations

Labeling Expert: A New Multi-Network Anomaly Detection Architecture Based on LNN-RLSTM

In network edge computing scenarios, close monitoring of network data and anomaly detection is critical for Internet services. Although a variety of anomaly detectors have been proposed by many scholars, few of these take into account the anomalies of the data in business logic. Expert labeling of business logic exceptions is also very important for detection. Most exception detection algorithms focus on problems, such as numerical exceptions, missed exceptions and false exceptions, but they ignore the existence of business logic exceptions, which brings a whole new challenge to exception detection. Moreover, anomaly detection in the context of big data is limited to the need to manually adjust detector parameters and thresholds, which is constrained by the physiological limits of operators. In this paper, a neural network algorithm based on the combination of Labeling Neural Network and Relevant Long Short-Term Memory Neural Network is proposed. This is a semi-supervised exception detection algorithm that can be readily extended with business logic exception types. The self-learning performance of this multi-network is better adapted to the big data anomaly detection scenario, which further improves the efficiency and accuracy of network data anomaly detection and considers business scenario-based anomaly data detection. The results show that the algorithm achieves 96% detection accuracy and 97% recall rate, which are consistent with the business logic anomaly fragments marked by experts. Both theoretical analysis and simulation experiments verify its effectiveness.

Fuzzy rule-based anomaly detectors construction via information granulation

It is meaningful for data-driven studies to detect and remove the anomaly data from a given raw dataset. Considering the existing anomaly data detection (ADD) methods have some drawbacks, e.g. confidence and uncertainty in anomaly description, computation cost of kernel learning and deep learning in ADD, therefore this paper proposes an advanced approach for constructing low-cost and effective ADD detectors. Firstly, making use of the good data description ability of information granules, granular data descriptors are constructed for anomaly and normal data description. Then, based on these data descriptors, reconstruction-based strategy is applied to model anomaly detection. Subsequently, with consideration of anomaly data’s uncertainty, fuzzy rules are formed and used to realize the final anomaly data detection. In this study, both synthetic and publicly available datasets are considered in experimental analysis. Numerical results illustrate the proposed method has a superiority ranging from 8.60% to 32.58% to conventional models on ADD performance, verifying the feasibility and effectiveness the proposed methods via granular data descriptors. Moreover, through the discussion on computation complexity, the proposed method is also demonstrated efficient on detecting anomalies.

Research on unsupervised anomaly data detection method based on improved automatic encoder and Gaussian mixture model

With the development of cloud computing, more and more security problems like “fuzzy boundary” are exposed. To solve such problems, unsupervised anomaly detection is increasingly used in cloud security, where density estimation is commonly used in anomaly detection clustering tasks. However, in practical use, the excessive amount of data and high dimensionality of data features can lead to difficulties in data calibration, data redundancy, and reduced effectiveness of density estimation algorithms. Although auto-encoders have made fruitful progress in data dimensionality reduction, using auto-encoders alone may still cause the model to be too generalized and unable to detect specific anomalies. In this paper, a new unsupervised anomaly detection method, MemAe-gmm-ma, is proposed. MemAe-gmm-ma generates a low-dimensional representation and reconstruction error for each input sample by a deep auto-encoder. It adds a memory module inside the auto-encoder to better learn the inner meaning of the training samples, and finally puts the low-dimensional information of the samples into a Gaussian mixture model (GMM) for density estimation. MemAe-gmm-ma demonstrates better performance on the public benchmark dataset, with a 4.47% improvement over the MemAe model standard F1 score on the NSL-KDD dataset, and a 9.77% improvement over the CAE-GMM model standard F1 score on the CIC-IDS-2017 dataset.

Situation-Aware Multivariate Time Series Anomaly Detection Through Active Learning and Contrast VAE-Based Models in Large Distributed Systems

The massive amounts of monitoring data in network applications bring an urgent need for intelligent operation in large distributed systems. The key problem is precisely detecting anomalies in multivariate time series (MTS) monitoring metrics with the awareness of different application scenarios. Unsupervised MTS anomaly detection methods aim at detecting data anomalies from historical MTS without considering the out-of-band information (including user feedback and background information like code deployment status), which leads to poor performance in practice. To take advantage of the out-of-band information, we propose ACVAE, an MTS anomaly detection algorithm through active learning and contrast VAE-based detection models, which simultaneously learns MTS data’s normal and anomalous patterns for anomaly detection. We also use a learnable prior to capture system status from the background information. Moreover, we propose a query model for VAE-based methods, which can learn to query labels of the most useful instances to train the detection model. We evaluate our algorithm on three different monitoring situations in eBay’s search back-end systems. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ACVAE</i> achieves a range F1 score of 0.68~0.96 with only 3% labels, significantly outperforming the best competing methods by 0.18~0.50, and even better than a supervised ensemble method designed by domain experts in eBay.

Internet of Things Information Network Security Situational Awareness Based on Machine Learning Algorithms

In order to accurately predict the security situation of Internet of Things information network, a research method based on machine learning algorithm for security situational awareness of Internet of Things information network is proposed. The perception result is represented by the perception model, the sample data are preprocessed based on the linear discriminant analysis method, the sample data are optimized to obtain the combined features, and then the processed data are used as the training input of the RBF neural network to find out the mapping relationship with the network situation value, so as to quantify the security posture of the system. The results show that automatic discovery and classification of seven violations is achieved. Since the platform was launched in China Mobile, more than 65,000 suspected illegal IoT cards have been discovered, effectively monitoring and controlling the operation and behavior of IoT cards. The processing efficiency of IoT card violations has been increased by more than 20 times. After the system is put into use, the discovery of suspected illegal users and behaviors can be realized through full automation, and the process of analysis, confirmation, and disposal can be shortened to 2 hours, which effectively reduce the false alarm rate and reduce operator costs. In previous monitoring of IoT cards, the false-positive rate of illegal IoT cards was about 83%, while the false-positive rate of existing algorithms dropped to 20%.Monitoring shows that business abuse monitoring detects the highest proportion of illegal IoT cards, 59%; infractions of Internet of Things cards detected through Internet abuse monitoring accounted for 20%; the proportion of Internet of Things card with machine card separation is 8%; in the information security risk monitoring (including spam text messages and harassing phone calls), the number of illegal Internet of Things cards found is small, only 3% and 2%, respectively; other infractions, including unauthorized use in locations and user complaints, accounted for 8%. It can effectively improve the ability to discover illegal IoT cards, greatly improve the accuracy of judgment, and improve the efficiency of disposal. The comparison verifies that the method is reliable and effective in the security situation awareness of the Internet of Things information network. Using the Internet of Things information security management system software based on the machine learning algorithm, the system software suitable for anomaly data detection is trained by adjusting the main parameters of the algorithm, which improves the automation and intelligence degree of the system software.

Anomaly Data Detection of Rolling Element Bearings Vibration Signal Based on Parameter Optimization Isolation Forest

Anomaly data detection is not only an important part of the condition monitoring process of rolling element bearings, but also the premise of data cleaning, compensation and mining. Aiming at the abnormal data segment detection of the vibration signals of a rolling element bearing, this paper proposes an abnormal data detection model based on comprehensive features and parameter optimization isolation forest (CF-POIF), which can adaptively identify abnormal data segments. First, in order to extract the mutation feature of vibration signals more accurately, the concept of comprehensive feature is proposed, which integrates the time domain and wavelet packet energy features. Then, the particle swarm optimization (PSO) algorithm is used to optimize the rectangular window length and sub sample set capacity in the isolation forest for anomaly detection. Finally, three real cases concerning abnormal data are used to verify the effectiveness of the proposed method. The results demonstrate that the proposed method is able to detect missing data, drift data and external interference data effectively, and it has a higher F1 score and accuracy compared to other methods.

GPS spoofing detection algorithm for quad-rotor helicopter based on speed/height comparison

In order to detect whether the GPS is abnormal in the flight state of small size quad-rotor, a GPS spoofing detection algorithm based on data fusion is proposed. Firstly, the velocity data from the monocular camera is calculated by using the self-velocity measurement algorithm based on the improved pyramid optical flow method. Secondly, the speed and altitude data of the IMU sensor unit of the small size quad-rotor were obtained, exactly the speed data were fused with the velocity data measured by using the optical flow method to obtain the optimal velocity estimation at that time. Finally, the small size quad-rotor GPS sensor data information is obtained, the aircraft speed and altitude data are calculated, and comparing with the data calculated in the previous step, to determine whether the GPS data at this time is abnormal. The experimental results on Anafi show that the present GPS spoofing detection algorithm can quickly detect data anomalies during flight. It can be seen that the present algorithm can provide strong technical support for GPS spoofing detection of quad-rotor.

Analysis of Abnormal Detection Data of Fire Accident in Power Cable Tunnel and Field Test Study on Characteristic Parameters of Tunnel Fire

With the continuous expansion of urban cable scale and the increase in service time, the fire accident risk of cable tunnels also increases. Defects in the configuration of fire prevention measures lead to many fire accidents and seriously affect the reliability of power supply. First, the detection data of a typical tunnel fire accident are analyzed in this study. The results show that the main problems are the unreasonable installation position of fire extinguishers, the lack of fire prevention measures between intermediate joints, and the insufficient fire prevention performance of interlayer fire barriers. Then, the field test of cable tunnel fire is carried out in a cable tunnel to be put into operation. The test data show that the high-temperature area is mainly concentrated on the top of the tunnel, and the insufficient suspension height of the fire extinguisher will affect the trigger time. The energy of a short circuit accident is large, the cable burns and spreads rapidly, and the delayed triggering may lead to large fire to extinguish effectively. Comparing the simulation and test results of abnormal characteristic parameters in fire, the temperature error is less than 4%. The fire power development curve is obtained through simulation, and the burner power is set to test the fire prevention and extinguishing devices. Finally, the optimal configuration scheme of the fire anomaly data detection equipment and fire-extinguishing products is proposed. The temperature drop rate increases by 135%.

Research on anomaly detection of wireless data acquisition in power system based on spark

In the era of big data, the network data of power system is more and more complex. Due to the limitation of data storage and processing capacity, the abnormal data detection of power grid terminal information system has the problems of low accuracy and high false alarm rate. The original machine learning algorithm with good detection effect is limited by the processing capacity and storage space of the traditional platform, and the detection effect and efficiency are significantly reduced. This paper takes improving the detection accuracy of abnormal data as the main research target, and designs an abnormal data behavior analysis program based on the Internet of Things under the Spark framework combined with improved Support Vector Machine (SVM) and random forest algorithm. The parallel SA_SVM_RF anomaly data behavior detection model based on Spark is mainly studied and applied to real-time detection. Combined with the respective advantages of Internet of Things technology and machine learning in anomaly data detection, the detection capability and rate of power grid anomaly data detection model are further improved. Experimental tests show that the proposed program is superior to traditional methods in data anomaly detection efficiency and quality, and has certain research significance in the field of power grid security.