Anomaly Detection Techniques Research Articles

Deep Learning for Time Series Anomaly Detection: A Survey

Time series anomaly detection is important for a wide range of research fields and applications, including financial markets, economics, earth sciences, manufacturing, and healthcare. The presence of anomalies can indicate novel or unexpected events, such as production faults, system defects, and heart palpitations, and is therefore of particular interest. The large size and complexity of patterns in time series data have led researchers to develop specialised deep learning models for detecting anomalous patterns. This survey provides a structured and comprehensive overview of state-of-the-art deep learning for time series anomaly detection. It provides a taxonomy based on anomaly detection strategies and deep learning models. Aside from describing the basic anomaly detection techniques in each category, their advantages and limitations are also discussed. Furthermore, this study includes examples of deep anomaly detection in time series across various application domains in recent years. Finally, it summarises open issues in research and challenges faced while adopting deep anomaly detection models to time series data.

Hyperspectral anomaly detection based on weighted low-rank sparse dictionary learning

The geometric model-based hyperspectral anomaly detection techniques have garnered a lot of interest recently. These methods are predicated on the assumption that the background can be represented by a dictionary of spectral vectors, but the anomaly cannot. Building an objective model with high representational capabilities and obtaining precise background modeling are therefore key aspects of this type of method. The background dictionary is also readily tainted by anomalies due to the limitations of the current approaches, making the detection findings less reliable. To address the above problems, this paper proposes a weighted low-rank sparse dictionary learning method (WLSDL). This model organically combines sparse representation with low-rank representation in order to effectively depict the intricate background distribution. The weights relating to the eigenvalues of the image are designed by mining the physical characteristics of the HSI. This can enhance the nuclear norm’s capacity for representation, resulting in a background dictionary with more representativeness. Furthermore, the capped norm constraint is incorporated in the objective function to decrease the effect of anomalies and noises on background modeling, hence minimizing anomaly pollution on the background dictionary. The residual image effectively increases the accuracy of anomaly identification since it makes it easier to distinguish between anomalies and background. We use three hyperspectral datasets to evaluate the proposed method. The experimental findings demonstrate the effectiveness and superiority of the proposed method over state-of-the-art methods.

Efficient fabric anomaly detection: A transfer learning framework with expedited training times

In industrial quality control, anomaly detection plays a critical role in identifying defective products. However, because of the rarity and time-consuming nature of defect collection, training models often rely solely on defect-free samples. This necessitates the use of unsupervised anomaly-detection techniques trained exclusively on defect-free data. Alternatively, defect data can be synthesized to augment the dataset with defective samples. In the textile industry, expeditious model training is crucial to ensure a smooth production flow. Unfortunately, most unsupervised methods require extensive training time. This paper proposes a novel transfer learning approach designed to achieve training times in seconds while effectively adapting the model to the target domain of fabric anomaly detection. The key contributions of our method include significantly reduced training times, up to 10 times faster than current state-of-the-art methods, and comparable performance in anomaly detection, achieving results on par with state-of-the-art approaches on benchmark datasets (MVTEC Anomaly Detection, TILDA, AITEX and DAGM). Additionally, our approach improves inference times, ensuring expedited and efficient anomaly detection during production. The proposed method offers a practical and efficient solution for real-time industrial quality control.

Automated PIM Detection in Wireless Networks via Smart Data-Driven Techniques

As the number of base stations keeps increasing due to ever-growing wireless services and pervasive connectivity, the operational costs for network service providers rises proportionally. In addition, with the evolution of wireless technologies and the development of next generation networks such as 5G and 6G, network operators are inevitably expected to encounter more complex challenges. One such network issue is the Passive Intermodulation (PIM) problem that is observed in both 4G and 5G networks. It degrades the user experience and radio resource efficiency while leading to an operational overhead for detecting and mitigating it on the operator side. Although there is a significant body of work regarding PIM detection and cancellation methods, the majority of such studies depends on hardware solutions and manual investigation by network engineers, which is costly in terms of time and labor. In this paper, we propose two methods using unsupervised and semi-supervised Machine Learning (ML) approaches, namely a time-series-based anomaly detection technique and an autoencoder-based one, for identifying PIM problems in network sites. The proposed solutions utilize a set of Key Performance Indicator (KPI) data of base stations obtained from network management systems for a significantly long time interval and detect possible PIM problems without the need for a human in the loop. We measure and analyze the performance of our solutions, with the guidance of experienced network engineers, on our collected dataset.

Enhancing training performance of convolutional neural network algorithm through an autoencoder-based unsupervised labeling framework for mineral exploration targeting

Convolutional neural networks (CNNs), a prominent deep learning approach, have garnered significant interest in the field of mineral potential mapping (MPM) due to their capability to capture and learn spatial features that traditional algorithms tend to overlook. The effectiveness of CNNs is closely tied to the quantity of training data available, thereby impacting the outcomes of MPM. Moreover, uncertainties arising from delineation of negative samples can compromise the reliability of MPM assessments. To deal with these challenges, we propose the utilization of an autoencoder-based anomaly detection technique for the purpose of annotating locations in an unsupervised manner. Subsequently, a CNN is trained using the unsupervised annotated data to generate a Fe prospectivity model within a specific region in Iran. To validate the effectiveness of the proposed method, we first perform an MPM with insufficient positive training samples that extract from the location of known occurrences. We then execute another MPM on a set of samples labeled based on the reconstruction error of an autoencoder network. When comparing the two prospectivity models, namely using augmented data or inadequate samples, it is evident that modeling with augmented data outperforms the MPM model trained with insufficient samples. This confirms the effectiveness of the adopted approach and shows that the unsupervised labeling technique proposed in this work can significantly improve the performance of the CNN in MPM.

Anomaly detection using unsupervised machine learning algorithms: A simulation study

This study presents a comprehensive evaluation of five prominent unsupervised machine learning anomaly detection algorithms: One-Class Support Vector Machine (One-Class SVM), One-Class SVM with Stochastic Gradient Descent (SGD), Isolation Forest (iForest), Local Outlier Factor (LOF), and Robust Covariance (Elliptic Envelope). Through systematic analysis on a synthetically simulated dataset, the study assessed each algorithm’s predictive performance using accuracy, precision, recall, and F1 score specifically for outlier detection. The evaluation reveals that One-Class SVM, Isolation Forest, and Robust Covariance are more effective in identifying outliers in the synthetic simulated dataset, with Isolation Forest slightly outperforming the other algorithms in terms of balancing precision and recall. One-Class SVM with SGD shows promise in precision but needs adjustment to improve recall. Local Outlier Factor may require parameter tuning or may not be as suitable for this particular dataset’s characteristics. The findings reveal significant variations in performance, highlighting the strengths and limitations of each method in identifying anomalies. This research contributes to the field of machine learning by demonstrating that the selection of an anomaly detection algorithm should be a considered decision, taking into account the specific characteristics of the data and the operational context of its application. Future work should explore parameter optimization, the impact of dataset characteristics on model performance, and the application of these models to real-world datasets to validate their efficacy in practical anomaly detection scenarios.

Fuzzy Ranking for Enhanced Security: Optimizing Feature Selection in VANET Intrusion Detection

In Vehicular Ad-Hoc Networks (VANETs), ensuring efficient intrusion detection while minimizing false alarms is paramount for maintaining network security and reliability. Traditional methods often struggle to strike this balance effectively. This study proposes a novel approach that employs fuzzy ranking to enhance intrusion detection efficiency in VANETs while mitigating false positive and negative factors. At the core of our approach lies the development of an innovative fuzzy ranking mechanism to identify optimal features extracted from VANET nodes. These features serve as critical indicators of potential security threats within the network. By systematically collecting and analyzing data from various sources including vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications, our approach aims to capture nuanced patterns and anomalies indicative of malicious behavior. The fuzzy ranking technique enables us to prioritize features based on their relevance to intrusion detection, thereby facilitating more effective feature selection. This process ensures that only the most informative features are utilized, enhancing the overall efficiency of the intrusion detection system. Features such as packet transmission rates, signal strength variations, route deviation patterns, and network topology characteristics are carefully evaluated and ranked to optimize detection accuracy. To train our intrusion detection model, we leverage machine learning algorithms such as deep neural networks, support vector machines, and decision trees. By utilizing labeled datasets containing examples of both benign and malicious network activities, our model learns to distinguish between normal and anomalous behavior, thus enabling timely threat detection. Furthermore, we integrate fuzzy logic-based anomaly detection techniques to identify previously unseen or zero-day attacks, enhancing the system's robustness. In addition to improving detection accuracy, our approach focuses on preventing harmful warnings caused by false alarms. We achieve this by implementing a comprehensive filtering mechanism that distinguishes between transient anomalies and genuine security threats. Contextual factors such as traffic conditions, environmental variables, and historical network behavior are considered to reduce false positives while ensuring reliable threat detection.Through extensive experimentation and evaluation, we demonstrate the effectiveness of our approach in enhancing intrusion detection efficiency and reducing false positive/negative factors in VANETs. The proposed methodology offers a promising solution to the security challenges faced by VANETs, paving the way for safer and more secure vehicular communication systems in the future.

Impact of log parsing on deep learning-based anomaly detection

Software systems log massive amounts of data, recording important runtime information. Such logs are used, for example, for log-based anomaly detection, which aims to automatically detect abnormal behaviors of the system under analysis by processing the information recorded in its logs. Many log-based anomaly detection techniques based on deep learning models include a pre-processing step called log parsing. However, understanding the impact of log parsing on the accuracy of anomaly detection techniques has received surprisingly little attention so far. Investigating what are the key properties log parsing techniques should ideally have to help anomaly detection is therefore warranted. In this paper, we report on a comprehensive empirical study on the impact of log parsing on anomaly detection accuracy, using 13 log parsing techniques, seven anomly detection techniques (five based on deep learning and two based on traditional machine learning) on three publicly available log datasets. Our empirical results show that, despite what is widely assumed, there is no strong correlation between log parsing accuracy and anomaly detection accuracy, regardless of the metric used for measuring log parsing accuracy. Moreover, we experimentally confirm existing theoretical results showing that it is a property that we refer to as distinguishability in log parsing results—as opposed to their accuracy—that plays an essential role in achieving accurate anomaly detection.

Electric vehicle supply equipment monitoring and early fault detection through autoencoders

This paper presents a novel approach to detecting anomalies in Electric Vehicle charging unit power profiles using a combination of Autoencoders with LSTM techniques. This study presents a robust methodology, combining the two Machine Learning techniques, for early fault estimation in a real-world case study. The proposed methodology offers significant advantages over existing methods by providing a more comprehensive analysis of anomalous trends. To validate the effectiveness of the proposed methodology, the authors tested it on real Electric Vehicles charging power curves provided by an Italian Distribution System Operator recorded on a historical database and compared the performances with the ones of a traditional anomaly detection technique. The results of the study, tested on Electric Vehicles Supply Equipment or charging stations, demonstrate that the proposed approach is highly effective in detecting anomalous trends in Electric Vehicles charging profiles.

Energy Consumption Outlier Detection with AI Models in Modern Cities: A Case Study from North-Eastern Mexico

The development of smart cities will require the construction of smart buildings. Smart buildings will demand the incorporation of elements for efficient monitoring and control of electrical consumption. The development of efficient AI algorithms is needed to generate more accurate electricity consumption predictions; therefore; anomaly detection in electricity consumption predictions has become an important research topic. This work focuses on the study of the detection of anomalies in domestic electrical consumption in Mexico. A predictive machine learning model of future electricity consumption was generated to evaluate various anomaly-detection techniques. Their effectiveness in identifying outliers was determined, and their performance was documented. A 30-day forecast of electrical consumption and an anomaly-detection model have been developed using isolation forest. Isolation forest successfully captured up to 75% of the anomalies. Finally, the Shapley values have been used to generate an explanation of the results of a model capable of detecting anomalous data for the Mexican context.

Deep Learning for Anomaly Detection in IoT Systems: Techniques, Applications, and Future Directions

The proliferation of Internet of Things (IoT) devices has revolutionized various sectors by enabling real-time data collection and processing, leading to unprecedented levels of efficiency and convenience. However, this increased connectivity and data flow also introduce significant security risks, particularly in the realm of anomaly detection. This study delves into the use of deep learning techniques for identifying abnormalities in IoT systems, providing a thorough analysis of state-of-the-art methods, assessing their effectiveness across different IoT scenarios, and exploring future research directions. We review current deep learning-based anomaly detection techniques, examine their applications in real-world settings, and discuss potential improvements and innovations. By synthesizing the latest research and developments, this paper aims to offer a comprehensive understanding of how deep learning can bolster the security and performance of IoT systems. Our findings highlight the importance of robust anomaly detection mechanisms in safeguarding IoT networks and underscore the need for continued advancements in this area. Through this study, we seek to contribute valuable insights into the application of deep learning for enhancing IoT system security and reliability, ultimately supporting the sustainable growth and integration of IoT technologies across various domains.

A Critical Review of Common Log Data Sets Used for Evaluation of Sequence-Based Anomaly Detection Techniques

A Critical Review of Common Log Data Sets Used for Evaluation of Sequence-Based Anomaly Detection Techniques

Advancing medical imaging with GAN-based anomaly detection

Anomaly detection in medical imaging is a complex challenge, exacerbated by limited annotated data. Recent advancements in generative adversarial networks (GANs) offer potential solutions, yet their effectiveness in medical imaging remains largely uncharted. We conducted a targeted exploration of the benefits and constraints associated with GAN-based anomaly detection techniques. Our investigations encompassed experiments employing eight anomaly detection methods on three medical imaging datasets representing diverse modalities and organ/tissue types. These experiments yielded notably diverse results. The results exhibited significant variability, with metrics spanning a wide range (area under the curve (AUC): 0.475-0.991; sensitivity: 0.17-0.98; specificity: 0.14-0.97). Furthermore, we offer guidance for implementing anomaly detection models in medical imaging and anticipate pivotal avenues for future research. Results unveil varying performances, influenced by factors like dataset size, anomaly subtlety, and dispersion. Our findings provide insights into the complex landscape of anomaly detection in medical imaging, offering recommendations for future research and deployment.

Detecting Financial Fraud Using Anomaly Detection Techniques: A Comparative Study of Machine Learning Algorithms

Detecting Financial Fraud Using Anomaly Detection Techniques: A Comparative Study of Machine Learning Algorithms

Advancements in Anomaly Detection Techniques in Network Traffic: The Role of Artificial Intelligence and Machine Learning

Purpose: This paper examines the most recent techniques for identifying irregularities in network data, with an emphasis upon machine learning (ML) and artificial intelligence (AI). Understanding how these technologies improve anomaly detection and overall network security is the goal of the study.Design/Methodology/Approach: A thorough examination of scholarly works, business analyses, and conference proceedings from the previous ten years was carried out. The study looks into supervised learning, unsupervised learning, and deep learning, among other AI and ML approaches. In order to evaluate these techniques' efficacy, advantages, and disadvantages in network anomaly detection, a comparative analysis was carried out.Findings/Results: The analysis shows that the identification of anomalies in network traffic is greatly enhanced by the use of AI and ML approaches. Methods such as Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have shown to be very successful in recognizing intricate patterns. Nonetheless, issues with data quality, computational complexity, and interpretability of models continue to exist.Originality/Value: This paper offers a current assessment of machine learning and artificial intelligence applications in network anomaly detection, emphasizing emerging trends and areas for further study. For academics and practitioners looking to improve network security using sophisticated detection methods, it provides insightful information.

Forecasting dynamics by an incomplete equation of motion and an auto-encoder Koopman operator

The governing equation of mass–spring–damper dynamics usually consists of a known structure and undetermined nonlinear components. This work proposes a physics-informed method to learn the unknown component by modeling its evolution with an auto-encoder Koopman operator. The Koopman operator is well known for its global linearization of dynamics and sensitivity to spectral modes. The undetermined component in an oscillatory equation of motion (EOM) is first reconstructed by a time-delay neural network whose inputs stack past sequences of observables and the control. Then, the evolution of the undetermined component is globally linearized and propagated by the Koopman operator in a latent space created by an auto-encoder neural network. Ultimately, observables at the future timestep are forecasted by timestepping the original EOM. Uniting an incomplete EOM and the Koopman operator, the proposed physics-informed Koopman neural ordinary differential equation (PiKNODE) method shows competitive edges compared with several black- and grey-box benchmarks in four synthetic and experimental case studies. Additionally, an anomaly-detection technique based on the Jensen–Shannon distance is proposed to boost the practicality of the PiKNODE for structural health monitoring.

Detecting faults in the cooling systems by monitoring temperature and energy

The cooling systems contribute to 40% of overall building energy consumption. Out of which, 40% is wasted because of faulty parts that cause anomalies in the cooling systems. We propose a three-stage, non-invasive part-level anomaly detection technique to identify anomalies in both cooling systems, a ducted-centralized and a ductless-split. We use COTS sensors to monitor temperature and energy without invading the cooling system. After identifying the anomalies, we find the cause of the anomaly. Based on the anomaly, the solution recommends a fix. If there is a technical fault, our proposed technique informs the technician regarding the faulty part, reducing the cost and time needed to repair it. In the first stage, we propose a domain-inspired time-series statistical technique to identify anomalies in cooling systems. We observe an AUC-ROC score of more than 0.93 in simulation and experimentation. In the second stage, we propose using a rule-based technique to identify the cause of the anomaly. We classify causes of anomalies into three classes. We observe an AUC-ROC score of 1. Based on the anomaly classification, we identify the faulty part of the cooling system in the third stage. We use the Nearest-Neighbour Density-Based Spatial Clustering of Applications with Noise (NN-DBSCAN) algorithm with transfer learning capabilities to train the model only once, where it learns the domain knowledge using the simulated data. The trained model is used in different environmental scenarios with both types of cooling systems. The proposed algorithm shows an accuracy score of 0.82 in simulation deployment and 0.88 in experimentation. In the simulation we used both ducted-centralized and ductless-split cooling systems and in the experimentation we evaluated the solution with ductless-split cooling systems. The overall accuracy of the three-stage technique is 0.82 and 0.86 in simulation and experimentation, respectively. We observe energy savings of up to 68% in simulation and 42% during experimentation, with a reduction of ten days in the cooling system’s downtime and up to 75% in repair cost.

Anomaly Detection in Smart Grid Application

The smart grid, a modernized electrical grid integrating advanced communication and information technologies, enables bidirectional flow of electricity and information, offering numerous benefits such as enhanced reliability, efficiency, and sustainability. However, the complexity and scale of the smart grid introduce new challenges, including the need for effective anomaly detection to ensure its secure and reliable operation. This research paper presents a comprehensive review of state-of-the-art anomaly detection techniques in smart grid applications. We explore various methodologies, including statistical methods, machine learning algorithms, and hybrid approaches, highlighting their strengths and limitations. Furthermore, we discuss the importance of accurate data preprocessing and feature selection to enhance anomaly detection performance. Through a comparative analysis, we demonstrate the effectiveness of different techniques in detecting anomalies in smart grid datasets. Our findings provide valuable insights for researchers and practitioners in developing robust anomaly detection systems for smart grid applications, ultimately contributing to the advancement of smart grid technology.

Uncovering time series anomaly using deep learning technique

Time series data, characterized by its sequential and temporal nature, plays a crucial role in various domains such as finance, healthcare, and industrial processes. Identifying anomalies within time series data is a critical task with applications ranging from fault detection to fraud prevention. Traditional anomaly detection techniques often struggle to capture complex temporal patterns and dependencies in time series data. This study presents a novel time series anomaly detection method using long short-term memory (LSTM) neural networks. LSTMs are a type of recurrent neural networks (RNNs) designed to model long-term dependencies, making them suitable for capturing the complex temporal relationships present in time series data. Here we use a financial dataset featuring opening and closing time, we predict the volume of money and the current price of that money at a specific time. The results demonstrate the efficacy of the deep learning-based approach in detecting anomalies within time series data, outperforming traditional methods in terms of sensitivity and adaptability to complex temporal patterns. The proposed methodology presents a promising solution for real-world applications where early detection of anomalies is crucial for proactive decision-making and system integrity.

A study on the auto encoder-based anomaly detection technique for pipeline inspection

A study on the auto encoder-based anomaly detection technique for pipeline inspection