Network Flow Monitoring Research Articles

Leveraging catchment scale automated novel data collection infrastructure to advance urban hydrologic-hydraulic modeling

Lack of long-term hydrological observations in urban catchments presents a significant obstacle to advancing reliability and accuracy of urban flood models, as sensor-based depth, flow or velocity monitoring in stormwater drainage networks is costly and rarely available. Commonly available networked technologies e.g., traffic cameras and cell phones, collectively referred to as novel data sources, if properly leveraged, could provide a high-quality, cost-effective source of urban hydrological observations. This proof-of-concept study utilizing experimental data collection infrastructure in Flagstaff, Arizona used remotely sensed flood-cameras based water depth time series and single point data, and citizens’ contributed single point data for model parameterization. An approach to reduce the number of parameters while addressing spatial heterogeneity was applied before parameterization. The results demonstrate the reduced uncertainty and improved prediction accuracy from calibrated model even from a single event calibration, emphasizing the potential of novel data in advancing pluvial flood modeling in challenging urban contexts.

ANALYSIS OF TCP FLOOD ATTACK USING NETFLOW

Traffic analysis is a common question for most of the production systems in various segments of computer networks. Attacks, configuration mistakes, and other factors can cause network increased accessibility and as a result danger for data privacy. Analyzing network flow and their single packets can be helpful for anomalies detection. Well-known network equipment has predeveloped network flow monitoring software. “NetFlow” data collector software “Nfsen” is an open-source way to collect information from agents. Also “Nfsen” is designed for data sorting and dataset for instruction detection system preparation. Prepared data can be split into fragments for artificial intelligent learning and testing. As AI unit can be used multilayer perceptron developed in a python programming language. This paper focused on real-world traffic dataset collection and multilayer perceptron deployment for TCP flood traffic detection.

A Correlation and Clustering Algorithm for Time Series Monitoring of Network Flows

Over the last few decades the advent of machine learning has found applications in large number of domains. The ability to learn and identify patterns in data and make predictions on deviations from these patterns has found large scope in several fields. In computer networks, this problem can be applied to understanding the behavior of a network at any point of time and predicting when the behavior may change over time. This paper discusses an approach that uses a statistical machine learning algorithm for the time series behavior analysis of computer networks. The proposed algorithm makes use of unsupervised learning and statistical data analysis methods over the flows in the network. The novel aspect being explored is the analysis of the inter-dependencies between flows, in addition to monitoring anomalies of individual flows. The results presented, provide waveform representations of the correlation and clustering patterns of some sample flows based on packet sizes.

Security monitoring strategies for your OT infrastructure

Cyber security threats are an ever-increasing risk to industrial control systems (ICS). Therefore, by considering the impact of cyberattacks as part of improvements in operational technology (OT) infrastructure resiliency serves the ultimate goal of minimising production downtime. In order to achieve this, organisations should consider flexible and innovative security monitoring strategies as well as a security operations centre (SOC). Having security monitoring mechanisms and a SOC function benefits the OT infrastructure in multiple ways, including incident response management, threat and vulnerability management, asset inventory and management, network flow monitoring, security logs monitoring and correlation, as well as the core function of defence against cyberattacks. This paper demonstrates the best practices for security monitoring and protection used by different industry sectors and explains security monitoring tools and their special features, as well as how these monitoring tools are integrated within the SOC. The paper then illustrates the baseline design for building a SOC for OT infrastructure and what needs to be considered when implementing an OT SOC, including data onboarding from OT infrastructure, SOC use cases development, and other SOC functionalities such as vulnerability and risk management, malware detection and other features. The paper aims to benefit the OT security community by providing a general guidance, future vision and required information to build a functional security strategy for critical national infrastructure (CNI) by considering the security life cycle starting from monitoring, through detection, to response and reporting.

Bayesian dynamic modeling and monitoring of network flows

AbstractIn the context of a motivating study of dynamic network flow data on a large-scale e-commerce website, we develop Bayesian models for online/sequential analysis for monitoring and adapting to changes reflected in node–node traffic. For large-scale networks, we customize core Bayesian time series analysis methods using dynamic generalized linear models (DGLMs). These are integrated into the context of multivariate networks using the concept of decouple/recouple that was recently introduced in multivariate time series. This method enables flexible dynamic modeling of flows on large-scale networks and exploitation of partial parallelization of analysis while maintaining coherence with an over-arching multivariate dynamic flow model. This approach is anchored in a case study on Internet data, with flows of visitors to a commercial news website defining a long time series of node–node counts on over 56,000 node pairs. Central questions include characterizing inherent stochasticity in traffic patterns, understanding node–node interactions, adapting to dynamic changes in flows and allowing for sensitive monitoring to flag anomalies. The methodology of dynamic network DGLMs applies to many dynamic network flow studies.

An incremental clustering method based on the boundary profile.

Many important applications continuously generate data, such as financial transaction administration, satellite monitoring, network flow monitoring, and web information processing. The data mining results are always evolving with the newly generated data. Obviously, for the clustering task, it is better to incrementally update the new clustering results based on the old data rather than to recluster all of the data from scratch. The incremental clustering approach is an essential way to solve the problem of clustering with growing Big Data. This paper proposes a boundary-profile-based incremental clustering (BPIC) method to find arbitrarily shaped clusters with dynamically growing datasets. This method represents the existing clustering results with a collection of boundary profiles and discards the inner points of clusters rather than keep all data. It greatly saves both time and space storage costs. To identify the boundary profile, this paper presents a boundary-vector-based boundary point detection (BV-BPD) algorithm that summarizes the structure of the existing clusters. The BPIC method processes each new point in an online fashion and updates the clustering results in a batch mode. When a new point arrives, the BPIC method either immediately labels it or temporarily puts it into a bucket according to the relationship between the new data and the boundary profiles. A bucket is employed to distinguish the noise from the potential seeds of new clusters and alleviate the effects of data order. When the bucket is full, the BPIC method will cluster the data within it and update the clustering results. Thus, the BPIC method is insensitive to noise and the order of new data, which is critical for the robustness of the incremental clustering process. In the experiments, the performance of the boundary point detection algorithm BV-BPD is compared with the state-of-the-art method. The results show that the BV-BPD is better than the state-of-the-art method. Additionally, the performance of BPIC and other two incremental clustering methods are investigated in terms of clustering quality, time and space efficiency. The experimental results indicate that the BPIC method is able to get a qualified clustering result on a large dataset with higher time and space efficiency.

Flow monitoring in Software-Defined Networks: Finding the accuracy/performance tradeoffs

In OpenFlow-based Software-Defined Networks, obtaining flow-level measurements, similar to those provided by NetFlow/IPFIX, is challenging as it requires to install an entry per flow in the flow tables. This approach does not scale well as the number of entries in the flow tables is limited and small. Moreover, labeling the flows with the application that generates the traffic would greatly enrich these reports, as it would provide very valuable information for network performance and security among others. In this paper, we present a scalable flow monitoring solution fully compatible with current off-the-shelf OpenFlow switches. Measurements are maintained in the switches and are asynchronously sent to a SDN controller. Additionally, flows are classified using a combination of DPI and Machine Learning (ML) techniques with special focus on the identification of web and encrypted traffic. For the sake of scalability, we designed two different traffic sampling methods depending on the OpenFlow features available in the switches. We implemented our monitoring solution within OpenDaylight and evaluated it in a testbed with Open vSwitch, using also a number of DPI and ML tools to find the best tradeoff between accuracy and performance. Our experimental results using real-world traffic show that the measurement and classification systems are accurate and the cost to deploy them is significantly reduced.

CBR-based network performance management with multi-agent approach

Nowadays, with the growth of heterogeneity of network equipment and software, and involved a variety of advanced network technology, the complexity of computer network have rapidly increased. Especially demands for high-performance network service in advanced data-intensive scientific research are dramatically increased. In order to cope with the network performance issue, this paper proposes the end-to-end (ETE) network performance management framework based on case-based reasoning with the case library and multi-agent integrated with perfSONAR as well as large-scale network flow monitoring. It provides a sophisticated framework for both network operator and user to systemically identify ETE network performance issues that are detected, diagnosed and recovered. The real cases are modeled and implemented into the casebase in the real experimental environment, a national research network of Korea. To verify the proposed framework, validation of the cases in the casebase is demonstrated.

A Resource-Efficient Flow Monitoring System

Network flow monitoring provides critical information for many network applications, and several approaches have been proposed to address flow monitoring. However they either lack flexibility in adapting to ever-changing network traffic (e.g. NetFlow), or require intensive computing resources (e.g. ANF). In this paper, we propose a resource-efficient flow monitoring system (called REFMS), which consists of pre-sampling module, sample & hold module, and flow removal module. Our REFMS can improve sampling accuracy and effectively utilize measurement resource with multiple sampling algorithms.

Detecting a Security Disturbance in Multi Commodity Stochastic Networks

Here is a multi-criteria approach to identify and evaluate network-flow monitoring strategies in a stochastic environment. We propose an analytical approach for assessing flow disturbance, based on limited sampling of arc-flow information in multi commodity, or multiple origin-destination (O---D), networks. A disturbance is defined here as abnormal network traffic beyond routine fluctuations. The network is characterized by arcs that are available with certain probabilities, suggesting a busy packet-switching/circuit-switching network, or simply network with failing components. Given the huge state space of such stochastic networks, our first objective is to bound the expected flow for computational efficiency. This is accomplished by extending available single-commodity-network results to the multi commodity case. The second objective is to determine the best placement of flow monitors to obtain the most accurate estimates of traffic flow, as represented by O---D pair traffic volumes in order to detect a disturbance. We followed a multi-criteria approach in defining and evaluating all possible monitor-placement strategies satisfying monitor availability. To assess a traffic pattern, O---D volumes are estimated using the l p -metric for p = 1, 2 and ?, representing a decision-maker's risk-preference structure. The final objective is to define a disturbance metric providing confident assessments on the occurrence of a flow disturbance. In this regard, we use linear-regression to generate confidence intervals around the expected flow for each O---D pair. This is supplemented by the Fisher Method of Randomization, a non-parametric test, for small networks. Unlike the Fisher-test results, it appears the linear-regression "disturbance metric" is very sensitive to detecting "disturbance." Preliminary results suggest that among the three l p metrics in O---D estimation, it appears that the best is the l 2-metric, which provides the appropriate sensitivity and accuracy. Counter to intuition, it is also observed that the "disturbance" estimation error generally decreases as the number of sampled arcs decreases.