Traffic Anomalies Research Articles

Evaluation of the Versatility of a Traffic Anomaly Detection Method during Heavy Rainfall

AbstractDuring heavy rains, traffic monitoring is limited, as the affected areas are monitored by reporting and patrolling. In this study, a method for detecting traffic anomalies during heavy rainfall events was established, and a model that uses probe vehicle data to detect traffic anomalies during a disaster (an event in which vehicles make U-turns in front of a damaged area) was proposed. In addition, a parameter calibration method was developed for the model using past disaster-related data. The generalizability of the calibrated model was evaluated by applying it to other disasters. According to the results, the proposed model exhibited good generalizability.

Diagnosing urban traffic anomalies by integrating geographic knowledge and tensor theory

ABSTRACT Urban traffic anomaly diagnosis is crucial for urban road management and smart city construction. Most existing methods perform anomaly detection from a data-driven perspective and ignore the unique spatiotemporal characteristics of traffic anomalies, resulting in reduced accuracy or incorrect extraction of anomalies. In this study, we integrate geographic methods with high-dimensional tensor theory and develop a comprehensive framework for traffic anomaly diagnosis. The framework is divided into three parts: road traffic status acquisition, traffic anomaly detection, and urban anomaly analysis. First, we perform filtering and map matching on the trajectory data to obtain the traffic states of urban roads and construct them into a tensor. For traffic anomaly detection, a novel Spatio-Temporal constrained Low-Rank Sparse Tensor (ST-LRST) method is constructed to decompose urban traffic data into normal and anomalous components. Specifically, ST-LRST introduces a truncated nuclear norm, as a tighter nonconvex alternative to the tensor low-rank function, to more accurately estimate daily traffic patterns, thus improving the extraction accuracy of the sparse anomaly tensor. The spatiotemporal features of traffic anomalies are embedded into the sparse tensor to reduce the fragmentation and error rate of urban anomaly identification. To analyze anomalies, we study the characteristics of the anomalies detected by ST-LRST, such as duration, impact scope, and impact intensity. Subsequently, combined with crowdsourced geographic information, we comprehensively analyze the anomalous spatiotemporal variations in the road network and explain the factors that induce these urban anomalies. Experiments are performed using traffic data from Xi’an, China, and the results indicate that ST-LRST achieves the most accurate and comprehensive diagnosis of urban anomalies.

On-Line Network Traffic Anomaly Detection Based on Tensor Sketch

On-Line Network Traffic Anomaly Detection Based on Tensor Sketch

Hybrid resampling and weighted majority voting for multi-class anomaly detection on imbalanced malware and network traffic data

In a large skewed dataset, the data imbalance is severe and the classifier's accuracy is biased towards the majority class. Insufficient data makes it challenging for the classifier to learn the feature of the minority classes. Moreover, some existing techniques of binary classification cannot directly apply to multi-classification. Plain oversampling may generate redundant and irrelevant data for the minority classes while undersampling may eliminate too many features for the majority class. This paper proposes a combination of constrained undersampling, oversampling, noise cleaning, and a weighted majority voting classifier (WMVC) to detect multi-classification obfuscated malware via memory and network traffic anomalies. As a constraint on undersampling and oversampling, the proposed framework divides the total sample number by the class number to obtain the average class size. According to the constraint, one or more majority classes are down-sampled using Random Undersampling, while all the minority classes are up-sampled using Adaptive Synthetic Sampling Approach, which is followed by Tomek Link to remove the noisy data. Then a weighted majority voting classifier aggregated tree-based ensemble algorithms is designed and compared to the XGBoost and Convolutional Neural Network (CNN) classifiers and six state-of-the-art ensemble algorithms. The comparison results show that the performance of the classifiers on balanced data outperforms those on imbalanced data, and the WMVC outperforms XGBoost, CNN, and the six other ensemble algorithms. Our approach can alleviate the classifier's bias towards the majority class while improving its performance for the difficult minority class.

An intelligent zero trust secure framework for software defined networking

Software-defined networking (SDN) faces many of the same security threats as traditional networks. The separation of the SDN control plane and data plane makes the controller more vulnerable to cyber attacks. The conventional “perimeter defense” network security model cannot prevent lateral movement attacks caused by malicious insider users or hardware and software vulnerabilities. The “zero trust architecture” has become a new security network model to protect enterprise network security. In this article, we propose an intelligent zero-trust security framework IZTSDN for the software-defined networking by integrating deep learning and zero-trust architecture, which adopts zero-trust architecture to protect every resource and network connection in the network. IZTSDN uses a traffic anomaly detection mode CALSeq2Seql based on a deep learning algorithm to analyze users’ network behavior in real-time and achieve continuous tracking and analysis of users, restrict malicious users from accessing network resources, and realize the dynamic authorization process. Finally, the Mininet simulation platform is extended to build the simulation platform MiniIZTA supporting zero-trust architecture and the proposed security framework IZTSDN is experimentally analyzed. The experimental results show that the IZTSDN security framework can provide about 80.5% of throughput when the network is attacked. The accuracy of abnormal traffic detection reaches 99.56% on the SDN dataset, which verifies that the reliability and availability of the IZTSDN security framework are verified.

Network Traffic Anomaly Detection Model Based on Feature Reduction and Bidirectional LSTM Neural Network Optimization

Aiming at the problems of large data dimension, more redundant data, and low accuracy in network traffic anomaly detection, a network traffic anomaly detection model (FR-APPSO BiLSTM) based on feature reduction and bidirectional long short-term memory (LSTM) neural network optimization is proposed. First, the feature dimensions are divided by hierarchical clustering according to the similarity distance between data features, and the features with high correlation are divided into the same feature subset. Second, an automatic encoder is used to reduce each feature subset, eliminating redundant information, and reducing the computational complexity of the detection data. Then, a particle swarm optimization algorithm based on adaptive updating of variables and dynamic adjustment of parameters (APPSO) is proposed, which is used to optimize the parameters of the bidirectional LSTM neural network (BiLSTM). Finally, the optimized BiLSTM is used as a classifier to model network traffic anomaly detection using the reduced feature data. Experiments based on NSL-KDD, UNSW-NB15, and CICIDS-2017 datasets show that the proposed FR-APPSO-BiLSTM model can effectively reduce data features, improve the accuracy of detection, and the performance of network traffic anomaly detection.

GMAT-DU: Traffic Anomaly Prediction With Fine Spatiotemporal Granularity in Sparse Data

The fine-grained prediction of traffic anomalies is crucial for Traffic Management Bureau to alleviate congestion and avoid public safety incidents. While in practice, the fine-grained prediction is very challenging due to two issues. 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Data sparsity</i> . At the fine-grained setting, missing data is inevitable and widespread on spatial and temporal dimension. Existing methods have weak performance as they do not handle missing data properly. 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Data distribution mutation</i> . At the fine-grained setting, the traffic conditions of adjacent road segments are sometimes completely different, invalidating existing spatiotemporal smoothing-based methods. This paper proposes GMAT-DU, a novel model that aims to predict traffic anomaly from sparse data in fine-grained manner. To solve the first issue, we propose a Decay Unrolling (DU) mechanism to make the model applicable to sparse datasets. The performance will be progressively enhanced by the spatiotemporal unrolling of high-impact neighbors. For the second issue, we combine the meta-features of roads with correlations between roads, which are learnt from road semantic information and historical spatiotemporal data, and make the model focusing on the high-impact neighbors by a Graph Meta-features based ATtention (GMAT) mechanism. Extensive experiments on two real-world datasets validate the effectiveness of our method. The experiment results show the significant advantages against the state-of-the-art models.

Design and implement a real-time network traffic management system using SNMP protocol

The research focuses on designing and implementing a system for managing incoming server traffic using the Simple Network Management Protocol (SNMP). The essence of the method is to provide a real-time monitoring system that empowers network managers to oversee all network activities efficiently. The key difference of the developed system lies in using the SNMP protocol to monitor and control the network without needing a third-party tool. The proposed approach addresses administrators' common network challenges, including inefficient bandwidth utilization and complex network resource monitoring. Upon implementation, the system gathered essential network equipment information from the Management Information Base (MIB). The research results indicate that the SNMP-based network management system prevents network devices from reaching their allotted bandwidth limits. The system notably reduces CPU utilization from 28 % to 8 % and decreases bandwidth usage from 1.7 MiB/s to 260 bytes/s. Additionally, the system's alerting capability enhances the network's resilience against incoming traffic anomalies that may otherwise interrupt server operations. An example of using the proposed system to manage incoming server traffic in small networks. It has several practical uses in managing and maintaining network infrastructure in different fields. Also, it enables real-time monitoring of network devices, resource management, fault detection, and performance optimization. The system can be used in small organizations to monitor network performance, track faults, and manage network resources efficiently. In addition, SNMP-based network monitoring systems can find applications in almost any field where networked devices and infrastructure require monitoring, management, and optimization

Class Scatter Ratio Based Mahalanobis Distance Approach for Detection of Internet of Things Traffic Anomalies

Class Scatter Ratio Based Mahalanobis Distance Approach for Detection of Internet of Things Traffic Anomalies

NETWORK TRAFFIC ANOMALY DETECTION BY DIMENSIONALITY REDUCTION METHODS

Anomaly in network traffic can arise from various cyberattacks, malfunction network devices or software flaws. Therefore traffic anomaly detection is an important practical problem especially in the issue of cybersecurity. Many recent anomaly detection techniques are based on machine learning methods. Unsupervised machine learning algorithms generally are more preferable than supervised learning because they can be trained on raw undimensioned data that greatly reduce training expenses. Dimensionality reduction methods are unsupervised learning algorithms that can be used for traffic anomaly detection. This work presents an application of such traffic anomaly detection methods, when faults are caused by some network attacks.

A traffic anomaly detection approach based on unsupervised learning for industrial cyber–physical system

In Industrial Cyber–Physical Systems (ICPSs), the attacker can intrude into the cyber system through many penetration tools and attack the physical system. Payload-based traffic anomaly detection is a popular technique against these attacks. Due to the imbalanced distribution of normal and attack samples in ICPS, existing payload-based detection methods are mostly implemented based on unsupervised learning, typically comprising a word segmentation model and an unsupervised classifier. However, existing methods may disrupt semantic correlations and face challenges in extracting complex payload dependence relationships. To address these issues, this paper proposes a traffic anomaly detection approach, which consists of a data preprocessing model, an unsupervised word segmentation model, and an unsupervised classification model based on autoencoder. The unsupervised word segmentation model utilizes Long Short-Term Memory (LSTM) to calculate the probability of each word segmentation combination, effectively addressing the issue of inaccurate segmentation results in existing payload segmentation models. The unsupervised classification model, which combines 1D-Convolutional Neural Network (1D-CNN) and Bidirectional Encoder Representation from Transformers (BERT), addresses the challenge of extracting complex payload dependence relationships in existing classification models. The proposed detection approach is evaluated using a Cyber–Physical Attack Dataset (CPAD). Compared with the state-of-the-art detection approaches, the proposed approach has shown a significant improvement in Precision, with an increase of 18.83%. Additionally, the Recall has also been substantially enhanced, with a gain of 22.3%. Overall, the F1 has demonstrated a comprehensive improvement of 20.60%.

Generic and Sensitive Anomaly Detection of Network Covert Timing Channels

Network covert timing channels can be maliciously used to exfiltrate secrets, coordinate attacks and propagate malwares, posing serious threats to cybersecurity. Current covert timing channels normally conduct small-volume transmission under the covers of various disguising techniques, making them hard to detect especially when a detector has little priori knowledge of their traffic features. In this paper, we propose a generic and sensitive detection approach, which can simultaneously (i) identify various types of channels without their traffic knowledge and (ii) maintain reasonable performance on small traffic samples. The basis of our approach is the finding that the short-term timing behavior of covert and legitimate traffic is significantly different from the perspective of inter-packet delays' variation. This phenomenon can be a generic reference to detect various channels because it is resistant to major channel disguising techniques which only mimic long-term traffic features, while it is also a sensitive reference to spot small-volume covert transmission since it can capture traffic anomalies in a fine-grained manner. To obtain the inner patterns of inter-packet delays' variation, we design a context-sensitive feature-extraction technique. This technique transforms each raw inter-packet delay into a discrete counterpart based on its contextual properties, thus extracting its variation features and reducing traffic data complexity. Then we learn legitimate variation patterns using a neural network model, and identify samples showing anomalous variation as covert. The experimental results show that our approach effectively detects all currently representative channels in the absence of their knowledge, presenting once to twice higher sensitivity than the state-of-the-art solutions.

A Network Traffic Abnormal Detection Method: Sketch-Based Profile Evolution

Network anomaly detection faces unique challenges from dynamic traffic, including large data volume, few attributes, and human factors that influence it, making it difficult to identify typical behavioral characteristics. To address this, we propose using Sketch-based Profile Evolution (SPE) to detect network traffic anomalies. Firstly, the Traffic Graph (TG) of the network terminal is generated using Sketch to identify abnormal data flow positions. Next, the Convolutional Neural Network and Long Short-Term Memory Network (CNN-LSTM) are used to develop traffic behavior profiles, which are then continuously updated using Evolution to detect behavior pattern changes in real-time data streams. SPE allows for direct processing of raw traffic datasets and continuous detection of constantly updated data streams. In experiments using real network traffic datasets, the SPE algorithm was found to be far more efficient and accurate than PCA and Basic Evolution for outlier detection. It is important to note that the value of φ can affect the results of anomaly detection.

Sensing Travel Source–Sink Spatiotemporal Ranges Using Dockless Bicycle Trajectory via Density-Based Adaptive Clustering

The travel source–sink phenomenon is a typical urban traffic anomaly that reflects the imbalanced dissipation and aggregation of human mobility activities. It is useful for pertinently balancing urban facilities and optimizing urban structures to accurately sense the spatiotemporal ranges of travel source–sinks, such as for public transportation station optimization, sharing resource configurations, or stampede precautions among moving crowds. Unlike remote sensing using visual features, it is challenging to sense imbalanced and arbitrarily shaped source–sink areas using human mobility trajectories. This paper proposes a density-based adaptive clustering method to identify the spatiotemporal ranges of travel source–sink patterns. Firstly, a spatiotemporal field is utilized to construct a stable neighborhood of origin and destination points. Then, binary spatiotemporal statistical hypothesis tests are proposed to identify the source and sink core points. Finally, a density-based expansion strategy is employed to detect the spatial areas and temporal durations of sources and sinks. The experiments conducted using bicycle trajectory data in Shanghai show that the proposed method can accurately extract significantly imbalanced dissipation and aggregation events. The travel source–sink patterns detected by the proposed method have practical reference, meaning that they can provide useful insights into the redistribution of bike-sharing and station resources.

BAE: Anomaly Detection Algorithm Based on Clustering and Autoencoder

In this paper, we propose an outlier-detection algorithm for detecting network traffic anomalies based on a clustering algorithm and an autoencoder model. The BIRCH clustering algorithm is employed as the pre-algorithm of the autoencoder to pre-classify datasets with complex data distribution characteristics, while the autoencoder model is used to detect outliers based on a threshold. The proposed BIRCH-Autoencoder (BAE) algorithm has been tested on four network security datasets, KDDCUP99, UNSW-NB15, CICIDS2017, and NSL-KDD, and compared with representative algorithms. The BAE algorithm achieved average F-scores of 96.160, 81.132, and 91.424 on the KDDCUP99, UNSW-NB15, and CICIDS2017 datasets, respectively. These experimental results demonstrate that the proposed approach can effectively and accurately detect anomalous data.

A Deep Learning Approach for Network Intrusion Detection Using a Small Features Vector

With the growth in network usage, there has been a corresponding growth in the nefarious exploitation of this technology. A wide array of techniques is now available that can be used to deal with cyberattacks, and one of them is network intrusion detection. Artificial Intelligence (AI) and Machine Learning (ML) techniques have extensively been employed to identify network anomalies. This paper provides an effective technique to evaluate the classification performance of a deep-learning-based Feedforward Neural Network (FFNN) classifier. A small feature vector is used to detect network traffic anomalies in the UNSW-NB15 and NSL-KDD datasets. The results show that a large feature set can have redundant and unuseful features, and it requires high computation power. The proposed technique exploits a small feature vector and achieves better classification accuracy.

Machine Learning Based Classification Model for Network Traffic Anomaly Detection

In current days, cloud environments are facing a huge challenge from the attackers in terms of various attacks thrown to the cloud service providers. In both industry and academics, the problem of detection and mitigation of DDoS attacks is now a challenging issue. Detecting Distributed Denial of Service (DDos) threats is mainly a classification problem that can be addressed using data mining, machine learning and deep learning techniques. DDoS attacks can occur in any of the seven-layer OSI model's network. Hence, detecting the DDoS attacks is an important task for cloud service providers to overcome dangerous attacks and loss incurred to stake holders and also the provider.

Correction: Riaz et al. Traffic Anomaly Prediction System Using Predictive Network. Remote Sens. 2022, 14, 447

Since the article “Traffic Anomaly Prediction System Using Predictive Network” by Riaz et al. [...]

Graph Spatiotemporal Pattern Learning Network for Real-Time Road Network Traffic Abnormal Incident Detection

To improve the efficiency of detecting abnormal traffic incidents on the road network and reduce the false alarm rate, a real-time traffic anomaly detection framework based on a graph spatiotemporal pattern learning (GSTPL) network is proposed. In this framework, a traffic pattern search algorithm based on a fluctuation similarity measure is designed to screen traffic flow data with the same traffic pattern, and a traffic pattern graph tuple is constructed as the input of the network model to avoid the sample imbalance problem and the effect of single-sample randomness for traffic pattern learning. Then the GSTPL network is designed to extract, unsupervised, the traffic spatiotemporal pattern features and make a reasonable prediction of future traffic parameters as the basis for anomaly evaluation. To further restrain the effect of random fluctuations in traffic flow parameters, an abnormal state evaluation method is designed to calculate the anomaly state likelihood by prediction error distribution learning. The overall detection framework realizes stable prediction of network key node traffic parameters by using spatiotemporal pattern features to construct the traffic pattern graph tuple, and gives incident evaluation results in real time by combination with the detection data. The experiment uses I90 and I405 highway traffic data in Seattle, WA, from 2015. Through comparative analysis, the proposed incident detection method based on GSTPL has a higher detection rate and lower false alarm rate, can adaptively learn dynamic changes of the traffic pattern, and has strong adaptability and stability to different traffic environments.

Research on Anomaly Network Detection Based on Self-Attention Mechanism.

Network traffic anomaly detection is a key step in identifying and preventing network security threats. This study aims to construct a new deep-learning-based traffic anomaly detection model through in-depth research on new feature-engineering methods, significantly improving the efficiency and accuracy of network traffic anomaly detection. The specific research work mainly includes the following two aspects: 1. In order to construct a more comprehensive dataset, this article first starts from the raw data of the classic traffic anomaly detection dataset UNSW-NB15 and combines the feature extraction standards and feature calculation methods of other classic detection datasets to re-extract and design a feature description set for the original traffic data in order to accurately and completely describe the network traffic status. We reconstructed the dataset DNTAD using the feature-processing method designed in this article and conducted evaluation experiments on it. Experiments have shown that by verifying classic machine learning algorithms, such as XGBoost, this method not only does not reduce the training performance of the algorithm but also improves its operational efficiency. 2. This article proposes a detection algorithm model based on LSTM and the recurrent neural network self-attention mechanism for important time-series information contained in the abnormal traffic datasets. With this model, through the memory mechanism of the LSTM, the time dependence of traffic features can be learned. On the basis of LSTM, a self-attention mechanism is introduced, which can weight the features at different positions in the sequence, enabling the model to better learn the direct relationship between traffic features. A series of ablation experiments were also used to demonstrate the effectiveness of each component of the model. The experimental results show that, compared to other comparative models, the model proposed in this article achieves better experimental results on the constructed dataset.