Network Data Streams Research Articles

The detection method of continuous outliers in complex network data streams based on C-LSTM

The detection method of continuous outliers in complex network data streams based on C-LSTM

A clustering-based method for outlier detection under concept drift

The ongoing challenge against network security issues persists, necessitating the exploration of alternative approaches. Anomaly-based strategies, diverging from traditional signature-based methods, gain popularity for their effectiveness in detecting new attacks. However, accurately defining normal network behavior becomes increasingly challenging due to data fluctuations. This study introduces a two-step process for recognizing evolving anomalies in streaming network data. Initially, clusters are updated incrementally upon new data arrival (the updating phase). Subsequently, anomalies are identified by discerning outer and inner outliers using minimum and maximum density thresholds. A buffer concept temporarily stores incoming data to prevent misclassification of normal network samples as anomalies. Performance evaluation in Python 3 assesses the impact on detection rate, false positives, and accuracy using two popular streaming datasets (NSL-KDD and UNSWNB-15). The algorithm achieves notable results, with a detection rate of 99.12% on UNSWNB-15 and a 7.9% false positive rate on NSL-KDD, marking significant progress. The proposed approach CADSD (Cluster-based Anomaly Detection with Streaming Data), operates in real-time without pre-training. However, challenges may arise from assuming the majority of data comprises normal instances, particularly during sudden spikes in attack data, potentially diminishing algorithm effectiveness. Nonetheless, the method shows the potential to enhance network security by promptly identifying emerging anomalies in real-time streaming data. The incorporation of a buffer concept to prevent the misidentification of normal network samples as anomalies underscores the innovative nature of this approach.

GBDT-IL: Incremental Learning of Gradient Boosting Decision Trees to Detect Botnets in Internet of Things.

The rapid development of the Internet of Things (IoT) has brought many conveniences to our daily life. However, it has also introduced various security risks that need to be addressed. The proliferation of IoT botnets is one of these risks. Most of researchers have had some success in IoT botnet detection using artificial intelligence (AI). However, they have not considered the impact of dynamic network data streams on the models in real-world environments. Over time, existing detection models struggle to cope with evolving botnets. To address this challenge, we propose an incremental learning approach based on Gradient Boosting Decision Trees (GBDT), called GBDT-IL, for detecting botnet traffic in IoT environments. It improves the robustness of the framework by adapting to dynamic IoT data using incremental learning. Additionally, it incorporates an enhanced Fisher Score feature selection algorithm, which enables the model to achieve a high accuracy even with a smaller set of optimal features, thereby reducing the system resources required for model training. To evaluate the effectiveness of our approach, we conducted experiments on the BoT-IoT, N-BaIoT, MedBIoT, and MQTTSet datasets. We compared our method with similar feature selection algorithms and existing concept drift detection algorithms. The experimental results demonstrated that our method achieved an average accuracy of 99.81% using only 25 features, outperforming similar feature selection algorithms. Furthermore, our method achieved an average accuracy of 96.88% in the presence of different types of drifting data, which is 2.98% higher than the best available concept drift detection algorithms, while maintaining a low average false positive rate of 3.02%.

AE‐Integrated: Real‐time network intrusion detection with Apache Kafka and autoencoder

SummaryUnknown cyber‐attack detection in network traffic streams is challenging but crucial to ensure network security. It is observed that new security threats occur on a daily basis and make cyberspace vulnerable. In the literature, machine learning and deep learning‐based network intrusion detection systems have gained a lot of success but still face many challenges in detecting new security threats and unknown cyber‐attacks in real‐time. Additionally, high false alarm rates and real‐time detection in constantly evolving high‐dimensional network data streams are open issues for the research community. To address this issue, a DL‐based solution is developed to detect real‐time network anomalies in streaming data with high detection accuracy, precision, recall and low false negative and positive scenarios. The proposed novel algorithm, AE‐Integrated, is developed and evaluated on the latest CICIDS‐2017 dataset. The AE‐Integrated is updated with the newest network traffic data stream by the human administrator after a certain period to maintain its prediction accuracy for future inference. The simulation study is conducted with the Apache Kafka and Slack API to get real‐time anomaly alerts. Finally, we compared the result with recent state‐of‐the‐art research to evaluate the significance of the proposed algorithm. It is concluded that combining multiple lightweight autoencoders into a single large architecture provides optimal results. The accuracy, recall, and AUC of AE‐Integrated obtained are 99.54%, 99.53%, and 0.998, respectively.

A novel hybrid ensemble learning for anomaly detection in industrial sensor networks and SCADA systems for smart city infrastructures

Critical Infrastructures (CIs) use Supervisory Control and Data Acquisition (SCADA) systems for monitoring and remote control. Sensor networks are being integrated into all areas of the infrastructures of smart cities. The sensor network data stream contains information that can be utilized to model and control the activity of these infrastructures. However, SCADA systems are constantly exposed to a variety of diverse intrusions, making detection with traditional intrusion detection systems (IDS) extremely difficult. Due to their unique specifications, conventional security solutions, like antivirus and firewall software, are unsuitable for properly securing SCADA systems. In addition, anomaly detection in industrial sensor networks (ISNs) should occur in real time. Therefore, effectively identifying cyberattacks in major SCADA systems is unquestionably essential for enhancing their resilience, ensuring safe operations, and avoiding expensive maintenance. We developed a novel hybrid ensemble model approach to address these issues. This paper's primary objective is to detect hostile intrusions that have already circumvented firewalls and typical IDS. In this paper, we propose a hybrid Ensemble Learning Model (ELM) for intrusion detection in SCADA systems with ISNs utilizing a tangible data gathered from a gas pipeline system given by Mississippi State University (MSU), the water system, and the high-dimensional University of New South Wales-NB 2015 (UNSW-NB15) data that reflects a typical attack in the Internet of Things (IoT) environment. The unity normalization method was adopted for data preprocessing, and the Principal Component Analysis (PCA) was utilized for feature extraction of the high-dimensional datasets. Grey Wolf Optimizer (GWO) was used for optimizing the bagging, stacking, Adaboost, and an ensemble of classifiers Naive Bayes and Support Vector Machine with a majority voting technique. Then, we utilized the proposed approach founded on the bijective soft-set approach for efficient ELM selection. The experiment was conducted in two phases: Initially, without PCA + GWO for feature extraction and selection on the ELM, and subsequently, with PCA + GWO for feature extraction and selection on the ELM. PCA + GWO on the ensemble of classifiers NB + SVM provided an accuracy of 99%, precision of 100%, recall of 100%, and detection rate of 99.90%, outpacing the ensemble of classifiers without PCA feature extraction and GWO optimization approaches.

An integrity-preserving technique for range queries over data streams in two-tier sensor networks

Two-tier sensor networks enable energy and computation saving due to the introduction of non-constrained storage nodes acting as intermediates between the sensors, which provide data, and the sink, which needs to access these data by submitting queries to the storage nodes. However, the storage nodes maintain a lot of data coming from different sensors and represent a single point of failure targeted by the attackers. In this scenario, robust guarantees about the integrity of the query result (i.e., completeness, freshness, and correctness) should be provided. This is a very well-known problem, traditionally faced through Merkle-Hash-Tree-based data structures. This paper presents a technique more efficient than the state-of-the-art methods for data insertions and deletions, supporting range queries over even non-temporal dimensions. Thus, the proposed solution appears suitable to the considered scenario, in which data can be updated with high frequency and insertions and deletions are executed by constrained devices.

A Comprehensive Survey on RF Energy Harvesting: Applications and Performance Determinants.

There has been an explosion in research focused on Internet of Things (IoT) devices in recent years, with a broad range of use cases in different domains ranging from industrial automation to business analytics. Being battery-powered, these small devices are expected to last for extended periods (i.e., in some instances up to tens of years) to ensure network longevity and data streams with the required temporal and spatial granularity. It becomes even more critical when IoT devices are installed within a harsh environment where battery replacement/charging is both costly and labour intensive. Recent developments in the energy harvesting paradigm have significantly contributed towards mitigating this critical energy issue by incorporating the renewable energy potentially available within any environment in which a sensor network is deployed. Radio Frequency (RF) energy harvesting is one of the promising approaches being investigated in the research community to address this challenge, conducted by harvesting energy from the incident radio waves from both ambient and dedicated radio sources. A limited number of studies are available covering the state of the art related to specific research topics in this space, but there is a gap in the consolidation of domain knowledge associated with the factors influencing the performance of RF power harvesting systems. Moreover, a number of topics and research challenges affecting the performance of RF harvesting systems are still unreported, which deserve special attention. To this end, this article starts by providing an overview of the different application domains of RF power harvesting outlining their performance requirements and summarizing the RF power harvesting techniques with their associated power densities. It then comprehensively surveys the available literature on the horizons that affect the performance of RF energy harvesting, taking into account the evaluation metrics, power propagation models, rectenna architectures, and MAC protocols for RF energy harvesting. Finally, it summarizes the available literature associated with RF powered networks and highlights the limitations, challenges, and future research directions by synthesizing the research efforts in the field of RF energy harvesting to progress research in this area.

An Incremental Learning Method Based on Dynamic Ensemble RVM for Intrusion Detection

Due to the dynamic changes of network data over time, static intrusion detection systems cannot adapt well to the behavioral characteristics of the input network data, resulting in reduced detection accuracy. In addition, continuous input data streams will bring huge challenges to resource storage and computing costs. Therefore, we propose an intrusion detection method of dynamic ensemble incremental learning (DEIL-RVM), and realize a dynamically adjusted ensemble intrusion detection model. In which a new overall misclassification probability weight value (OMPW) based on incremental set or data chunk is designed as the basis for updating the ensemble model, and it can be used to prune and replace the poor base component in the ensemble model. We presented a probabilistic decision function taking into account the posterior probability of each base RVM model dividing the sample into each category. The RVM with high sparsity is used as the base component to obtain the good balance between the accuracy, robustness and resource consumption, which can sacrifice less time and storage cost in ensemble incremental learning while achieving higher detection accuracy and stability in network data streams.

Signal-to-noise-ratio and maximum-signal-to-noise-ratio detection statistics in template-bank searches for exotic physics transients with networks of quantum sensors

Quantum sensor networks such as the existing networks of atomic clocks and magnetometers offer intriguing capabilities in searches for transient signals such as dark matter or fields sourced by powerful astrophysical events. The common matched-filter technique relies on the signal-to-noise-ratio (SNR) detection statistic to probe such transients. For macroscopic dark matter objects, the network would register a sweeping signal as the object propagates through at galactic velocities. A potential event is registered when the SNR from a specific template exceeds a threshold set by a desired false positive rate. Generically, to span the continuous parameter space for the network-exotic-physics encounter, one has to deal with multiple templates. In such template-bank searches, the natural generalization of the SNR statistic is the maximum-signal-to-noise-ratio (SNR-max) statistic, defined as the maximum of the absolute values of SNRs determined from individual template matching. While individual SNR realizations are Gaussian distributed, SNR-max probability distribution is non-Gaussian. Moreover, as the individual template-bank SNRs are computed using the same network data streams, SNRs become correlated between the templates. Cross-template correlations have a sizable effect on the SNR-max probability and cumulative distributions, and on the threshold SNR-max values. Computing threshold SNR-max values for large template banks is computationally prohibitive and we develop analytic approaches to computing properties of the SNR-max statistic. This is done for cases when the template bank is nearly orthogonal (small cross-template correlations) and for banks with cross-template correlation coefficient distribution ``squeezed'' about the most probable cross-template correlation value. Since the cross-template correlation coefficients quantify the similarity of templates, increasing correlations tend to decrease SNR-max thresholds for specific values of false positive rates. Increasing the number of templates in the bank increases the SNR-max thresholds. Our derivations are carried out for networks that may exhibit colored noise and cross-node correlations. Specific applications are illustrated with a dark matter search with atomic clocks onboard satellites of a Global Positioning System and with a ``toy'' planar network with cyclic rotational symmetry.

DEFECT: discover and eradicate fool around node in emergency network using combinatorial techniques

Mining the frequent pattern deals with the finding patterns in large set of data, subsequences and substructures that occur in a database frequently. Likewise, We can use Frequent pattern mining for MANET nodes in order to identify the paths which are participated in frequent data transaction among the various Mobile adhoc network nodes. The network data stream is a long and continuous sequence of data sets transmitted over the network. The OCA (Online Combinatorial Approximation) algorithm is used in the data stream for mining online data. The processing time of OCA was much less and accuracy of its approximate result was quite high like other traditional mining methods. The Data Path Combinatorial Approximation (DPCA) algorithm deals with a frequent pathset mining over the MANET data flow. The pathset is generation of path from the set of paths on any node which are provided paths to various other nodes participating in the data transmission. The mining algorithm is based on Approximate Inclusion–Exclusion technique. Without continual path scanning, approximate counts are calculated for the pathsets. Skip and complete technique and group count technique were combined together and integrated into the DPCA algorithm to improve the MANET performance in terms of identifying fool around (misbehaving) nodes.

Modeling of attack detection system based on hybridization of binary classifiers

The study considers the development of methods for detecting anomalous network connections based on hybridization of computational intelligence methods. An analysis of approaches to detecting anomalies and abuses in computer networks. In the framework of this analysis, a classification of methods for detecting network attacks is proposed. The main results are reduced to the construction of multi-class models that increase the efficiency of the attack detection system, and can be used to build systems for classifying network parameters during the attack. A model of an artificial immune system based on an evolutionary approach, an algorithm for genetic-competitive learning of the Kohonen network and a method of hierarchical hybridization of binary classifiers with the addition to the detection of anomalous network connections have been developed. The architecture of the network distributed attack detection system has been developed. The architecture of the attack detection system is two-tier: the first level provides the primary analysis of individual packets and network connections using signature analysis, the second level processes the processing of aggregate network data streams using adaptive classifiers. A signature analysis was performed to study network performance based on the Aho-Korasik and Boyer-Moore algorithms and their improved analogues were implemented using OpenMP and CUDA technologies. The architecture is presented and the main points of operation of the network attack generator are shown. A system for generating network attacks has been developed. This system consists of two components: an asynchronous transparent proxy server for TCP sessions and a frontend interface for a network attack generator. The results of the experiments confirmed that the functional and non-functional requirements, as well as the requirements for computing intelligent systems, are met for the developed attack detection system.

Subchannel Allocation Based on Clustered Interference Alignment for Differentiated Data Streams in Dense Small Cell Networks

This paper investigates subchannel allocation based on clustered interference alignment in dense small cell networks when all small cell user equipments (SUEs) have differentiated requirements for data streams. By imposing the condition that each cluster has a size not exceeding the maximum value achieved when each SUE needs only one data stream, we maximize the number of SUEs with guaranteed requirements for data streams, which is NP-hard. Hence, we propose a two-phase efficient solution with much lower complexity and reduced feedback overhead. First, similarity clustering is performed by graph partition, and then, subchannel allocation is done through a coloring algorithm. Numerical results show that the proposed solution offers a performance better than the related schemes and close to the approximate optimal solution.

Online Topology Inference from Streaming Stationary Graph Signals with Partial Connectivity Information

We develop online graph learning algorithms from streaming network data. Our goal is to track the (possibly) time-varying network topology, and affect memory and computational savings by processing the data on-the-fly as they are acquired. The setup entails observations modeled as stationary graph signals generated by local diffusion dynamics on the unknown network. Moreover, we may have a priori information on the presence or absence of a few edges as in the link prediction problem. The stationarity assumption implies that the observations’ covariance matrix and the so-called graph shift operator (GSO—a matrix encoding the graph topology) commute under mild requirements. This motivates formulating the topology inference task as an inverse problem, whereby one searches for a sparse GSO that is structurally admissible and approximately commutes with the observations’ empirical covariance matrix. For streaming data, said covariance can be updated recursively, and we show online proximal gradient iterations can be brought to bear to efficiently track the time-varying solution of the inverse problem with quantifiable guarantees. Specifically, we derive conditions under which the GSO recovery cost is strongly convex and use this property to prove that the online algorithm converges to within a neighborhood of the optimal time-varying batch solution. Numerical tests illustrate the effectiveness of the proposed graph learning approach in adapting to streaming information and tracking changes in the sought dynamic network.

Research on information steganography based on network data stream

To protect the information from being intercepted by third parties during the network communication process, this paper proposes a new type of data steganography technology based on network data flow. Using the network protocol itself and the relationship between data packets in the entire network data stream to perform network data steganography, transfer hidden data, and perform secondary identity authentication. Different from the traditional steganography method, this method can encode the hidden data and send the interval value by embedding the data packet, thereby hiding and transmitting the hidden data. In this technology, the operation of hidden data does not affect the user's access request for real network data, and it can perform processes such as hidden data transfer and secondary authentication without the user being able to detect it. Through experimental verification and evaluation, our method improves the concealment of the steganographic channel, is not easy to attract attention and has no obvious statistical characteristics of the traffic, and can improve the concealment and robustness of the steganography technology based on network data streams.

Evaluation of HMM-Based Network Intrusion Detection System for Multiple Multi-Stage Attacks

With the explosive growth of network security threats, there is a dire need to build secure network systems. In this article, we address the challenges of modeling and detecting advanced network attacks. In particular, we investigate how interleaving multiple multi-stage can exacerbate the stealthiness of the attack and deceive network intrusion detection systems. We design a detection architecture based on a leading statistical machine learning technique, HMM. The proposed architecture deploys a set of HMM templates of recognized multi-stage attacks to detect and track the progress of stealthy attacks. Extensive simulation experiments are conducted to assess the performance of the proposed architecture for multiple multi-stage attack scenarios in the presence of imperfect partitioning of network data streams and false alerts with various rates.

Load Balancing Algorithms for Big Data Flow Classification Based on Heterogeneous Computing in Software Definition Networks

Distributed network architecture of heterogeneous computing faces with such problems as strict performance constraints of network control, unpredictable mapping relationship between computing data algorithms of different mobile terminals and inconsistency between computing algorithms and link control of data networks. In order to solve the above problems, we begin with software definition network architecture and load balancing algorithm for heterogeneous computing, and gradually improve the real-time and reliability of heterogeneous computing. On the one hand, the heterogeneous computing data of fog node and cloud computing system are distributed. The centralized service of software-defined network combines with distributed computing of mobile edge terminal and its subnet. On the other hand, we define the centralized information and distributed scheduler of the network. In addition, we deploy the optimal assignment of data sharing and heterogeneous computing tasks in real time with ellipse-partitioned area as the object. A series of algorithms for classifying and assigning heterogeneous computing data streams in software-defined networks are designed to achieve the optimal balance among load balancing, minimum classification of large data streams, minimum resource occupation and time constraints. Experimental comparison compared and evaluated the Load Balancing with big data stream (LBBS), Load Balancing with Heterogeneous Computing (LBHC) and the proposed LBBHD. Compared with the other two algorithms, the proposed algorithm improves workload skewness, throughput and load balancing error respectively about 2.1%, 1.96%, 2.9%, 2.2%; 5.57%. 2.51%.

Unsupervised Anomaly Detection for Network Data Streams in Industrial Control Systems

The development and integration of information technology and industrial control networks have expanded the magnitude of new data; detecting anomalies or discovering other valid information from them is of vital importance to the stable operation of industrial control systems. This paper proposes an incremental unsupervised anomaly detection method that can quickly analyze and process large-scale real-time data. Our evaluation on the Secure Water Treatment dataset shows that the method is converging to its offline counterpart for infinitely growing data streams.

An Anomaly Data Mining Method for Mass Sensor Networks Using Improved PSO Algorithm Based on Spark Parallel Framework

Accurate detection and capture of anomaly data in complex network data stream is an important part of ensuring network security. Traditional methods cannot adapt to the high dynamic changes of abnormal data characteristics in complex network. Thus, the detection accuracy is reduced. In this paper, a k-means parallel clustering algorithm is proposed. It is optimized by particle swarm optimization with dynamic adaptive inertia weight (dsPSOK-means). And it is used to mine the anomaly data for mass sensor networks. The inertia weight is dynamically adjusted through the fitness function, so that the dsPSO algorithm has the adaptive characteristics. Then, the output of the dsPSO algorithm is taken as the input of the k-means algorithm. Thus, the intelligence and self-adaptability of the k-means algorithm in selecting the initial center point is improved. Finally, with the help of Spark platform, the parallelization of dsPSOK-means clustering algorithm in the clustering environment is designed and implemented. It is shown by the experimental results that the traffic among nodes in the execution process can be effectively reduced by the dsPSOK-means algorithm. And the accuracy of abnormal data mining in complex network data flow is 5% higher than that of the comparison algorithm on average.

Session-Level Reliability Analysis for Multi-Service Communication in Autonomous Vehicular Fleets

The support for reliable communication in the increasingly large fleets of autonomous vehicles is one of the important challenges for emerging 5G systems. The presence of heterogeneous data streams of different priority between the connected vehicles (coordinated autonomous driving, platooning, passenger entertainment services, etc.) calls for new methods to estimate the performance characteristics of these systems. In this article, a novel mathematical framework is proposed to model the process of dynamic radio resource (re-)allocation across multiple competing data streams in autonomous vehicular fleets equipped with 5G cellular capabilities. The developed framework is subsequently applied to: (i) study the coexistence of multiple traffic types having different service requirements and (ii) quantify the impact of session prioritization schemes. Our study reveals that the prioritization scheme initially offloading high-rate entertainment sessions, named ESpreempt, in most of the setups achieves a 5-30% performance gain in comparison with the scheme initially offloading low-rate platooning sessions, named PSpreempt. It is also shown that higher variations in the traffic load of autonomous driving sessions have a distinctly negative impact on system-level performance. The outlined framework can be applied in a wide range of 5G vehicular scenarios, as well as extended to capture other categories of data streams in future wireless networks.

Rethinking sketching as sampling: A graph signal processing approach

Sampling of signals belonging to a low-dimensional subspace has well-documented merits for dimensionality reduction, limited memory storage, and online processing of streaming network data. When the subspace is known, these signals can be modeled as bandlimited graph signals. Most existing sampling methods are designed to minimize the error incurred when reconstructing the original signal from its samples. Oftentimes these parsimonious signals serve as inputs to computationally-intensive linear operators. Hence, interest shifts from reconstructing the signal itself towards approximating the output of the prescribed linear operator efficiently. In this context, we propose a novel sampling scheme that leverages graph signal processing, exploiting the low-dimensional (bandlimited) structure of the input as well as the transformation whose output we wish to approximate. We formulate problems to jointly optimize sample selection and a sketch of the target linear transformation, so when the latter is applied to the sampled input signal the result is close to the desired output. Similar sketching as sampling ideas are also shown effective in the context of linear inverse problems. Because these designs are carried out off line, the resulting sampling plus reduced-complexity processing pipeline is particularly useful for data that are acquired or processed in a sequential fashion, where the linear operator has to be applied fast and repeatedly to successive inputs or response signals. Numerical tests showing the effectiveness of the proposed algorithms include classification of handwritten digits from as few as 20 out of 784 pixels in the input images and selection of sensors from a network deployed to carry out a distributed parameter estimation task.