Flow-level Data Research Articles

AIS-NIDS: An intelligent and self-sustaining network intrusion detection system

The ever-evolving landscape of network security is continually molded by the dynamic evolution of attack vectors and the relentless emergence of new, highly sophisticated attacks. Attackers consistently employ increasingly advanced techniques, rendering their actions elusive and formidable. In response to this ever-growing threat, the demand for intelligent and autonomous security systems has reached paramount importance. In this paper, we introduce AIS-NIDS (An Intelligent and Self-Sustaining Network Intrusion Detection System), an innovative network intrusion detection system (NIDS) that delves into the realm of packet-level analysis. By doing so, AIS-NIDS is capable of identifying threats with intricate payload-level details, a level of granularity that traditional NIDS relying solely on flow-level data may overlook. The defining feature of AIS-NIDS is its dual functionality, driven by autonomous and intelligent learning. It not only autonomously distinguishes between benign and unknown attacks using machine learning models but also conducts incremental learning, adapting to new attack classes. In essence, AIS-NIDS bridges the gap between traditional NIDS and the next generation of intelligent systems, endowing the system with the capacity for independent decision-making and real-time adaptability in the face of evolving threats. Our extensive experiments stand as a testament to AIS-NIDS’ ability to efficiently manage and identify new attack classes, thus establishing it as a valuable asset in the reinforcement of network infrastructures. Through our experimentation, we have demonstrated the practical efficacy of the proposed approach by simulating a real-world scenario in which certain attack classes are unknown. AIS-NIDS not only effectively identified these unknown threats but also autonomously learned to recognize them as it encountered them, enhancing the system’s capabilities for future encounters with these threats.

Model and application of inversion data cleaning for flow monitoring stations in the middle route of the South-to-North Water Diversion Project

For a long-distance open-channel water transfer project, the correctness of the hydrological state study and hydrodynamic numerical simulation is impacted directly by the spatiotemporal and consistency of the flow monitoring data along the project. Reported here is an inverted data cleaning model for flow monitoring stations based on the principle of water dynamic balance and the longest sequence method of interval flow, targeting the abnormal phenomenon of unequal flow at monitoring stations in the dispatching operation and using the middle route of the South-to-North Water Diversion Project in China as the research object. As an example, a hydrodynamic model is built for verification using flow and water level data after model cleaning as the upper and lower boundaries, respectively, for the middle-route project from the Baihe River to Huangjin River. The findings indicate that the cleaning model enhances the accuracy of flow monitoring data and reduces both the mean absolute error of the water level in front of the gate and the root-mean-square error by 0.0757 and 0.0895 m, respectively. In terms of data spatial consistency and logic, the cleaned flow data are superior to the measured flow data.

Rear-End Crash Risk Analysis considering Drivers’ Visual Perception and Traffic Flow Uncertainty: A Hierarchical Hybrid Bayesian Network Approach

Rear-end crashes or crash risk is widely recognized as safety-critical state of vehicles under comprehensive conditions. This study investigated the association between traffic flow uncertainty, drivers’ visual perception, car-following behavior, roadway and vehicular characteristics, and rear-end crash risk variation and compared the crash risk variation prediction with and without specific flow-level data. Two datasets comprising 5055 individual vehicles in car-following state were collected through on-road experiments on two freeways in China. A hierarchical hybrid BN model approach was proposed to capture the association between drivers’ visual perception, traffic flow uncertainty, and rear-end crash risk variation. Results show that (1) the BN model with flow-level data outperformed the BN model without flow-level data and could predict 85.3% of the cases of crash risk decrease, with a false alarm rate of 21.4%; (2) the hierarchical hybrid BN models showed plausible spatial transferability in predicting crash risk variation; and (3) the incorporation of specific flow-level variables and data greatly benefited the successful identification of rear-end crash risk variations. The findings of this study suggest that rear-end crash risk is inherently associated with both individual driving behaviors and traffic flow uncertainty, and appropriate visual perceptual information could compensate for crash risk and improve safety.

Self-attentive deep learning method for online traffic classification and its interpretability

Traffic classification is one of the fundamental tasks in computer networking. This task aims to associate network traffic to a specific class according to the requirements (e.g., QoS provisioning). Online classification, which refers to the situations where flows need to be classified in real time, is an essential technique for this topic. In recent academic research, traffic classification methods based on machine learning (ML) or deep learning (DL) have been proposed. However, most of these methods take flow-level data as input, which requires observing the entire or large portion of a flow and violates the restrictions of online classification. Furthermore, the DL-based methods scarcely discuss the interpretability (e.g., which features are learned by DL, where is the discrimination power from). The lack of interpretability makes people question their reliability and may hinder their further applications. In this paper, we propose a self-attentive method (SAM) for traffic classification. We properly design a neural network whose input can be more fine-grained (i.e., packet-level). This neural network outputs classification results (∼2ms per packet) and consequently satisfies the requirements of online classification. Furthermore, a new technique called self-attention mechanism is utilized for interpretability exploration. By assigning attentive weights to different parts of the input, the self-attention mechanism allows us to understand how the DL model learns discriminative features from the input. According to experimental results, SAM outperforms the current state-of-the-art schemes, improving classification accuracy by ∼8% (protocol classification), ∼5% (application classification), and ∼13% (traffic type classification). The code is available at https://github.com/xgr19/SAM-for-Traffic-Classification.

Assessment of Rainfall, Streamflow and Reservoir Level Trends for Malewa River Catchment, Naivasha, Kenya

Appropriate water quality and quantity are critical problem greatly affecting significant proportion of the population particularly those living in semi-arid and arid areas. Rainfall characteristics, streamflow and reservoir levels are some of the key hydro-meteorological variables within a catchment in the determination of sufficiency of water quantity. Lake Naivasha reservoir level continues to be hit by anthropogenic stressors such as loss of wetlands, water abstraction and eutrophication. Also there has been a lot of fluctuation in flow discharge of Malewa River over the years where the tributaries (Turasha and Karati streams) usually dries up during dry season. This paper presents the results of a research to assess the time series trends of rainfall, streamflow and lake basin levels data in the catchment from 1980-2018 using Mann-Kendall trend test to assess water quantity status. Monthly stream flow, rainfall and reservoir level data in the catchment were obtained from Water Resources Management Authority (WRMA) and Kenya Meteorological Department (KMD) for the years 1980-2018. Non-parametric Mann-Kendall rank statistics test was used to examine the trend components of hydro-meteorological variables. The P-values were used to test whether the null hypotheses were statistically significant (α<0.05) at a 5% level of significance. Rainfall values had an increasing trend from 2011 onwards but not statistically significant while annual streamflow values had no trend as P-value showed weak evidences to rejected the null hypothesis. Reservoir level had an upward trend in years 1981, 2001, 2003, 2012 and 2018 while downward trends in 1980, 1982, 1984, 1986, 1987, 2000, 2004, 2008 and 2014. From 2010 onwards, in monthly data, there have been upward water levels in the lake as shown by the trend. These show that the hydro-meteorological indicators in the catchment are independently trending, therefore some other factors such as land cover changes or climate changes were adversely affecting the hydrological cycle. These findings are helpful for planning and management in water resource systems such as forecasting water distribution and mitigate flood and drought.

Trend analysis of river flow and groundwater level for Shipra river basin in India

The declining baseflow and drying up of river during the lean period are important issues in major rivers systems in India. In recent times, more focus has been given to formulate an action plan for river rejuvenation. The present study aims at the assessment of declining flow regime in Shipra river in the Madhya Pradesh state of India. Long-term daily runoff data recorded at the Ujjain gauge discharge site on Shipra river was analyzed to identify trends in dependable flows using a non-parametric Mann–Kendall test. The declining trend for dependable river flow at various probability levels was observed at Ujjain during the beginning of monsoon (month of July) and at the end of monsoon (months of September and October). However, a rising trend was observed during the mid-monsoon season (month of August). The trend analysis of pre-monsoon and post-monsoon groundwater level data indicated falling groundwater level trend in the entire Shipra basin. Significant declining trends in groundwater levels were observed in the middle reach area of the basin where important cities like Indore and Ujjain are located. Wigley and Jones’s index was used to judge the rising or declining flow regime of Shipra river. Average annual rainfall, as well as river runoff, has decreased substantially during the last two decades. The trend analysis of flow and groundwater level data of the Shipra river basin and information derived using Wigley and Jones index will be useful to formulate strategies for the river rejuvenation planning.

Towards Learning-Based, Content-Agnostic Detection of Social Bot Traffic

With the fast-growing popularity of online social networks (OSNs), the security and privacy of OSN ecosystems becomes essential for the public. Among threats OSNs face, malicious social bots have become the most common and detrimental. They are often employed to violate users’ privacy, distribute spam, and disturb the financial market, posing a compelling need for effective social bot detection solutions. Unlike traditional social bot detection approaches that have strict requirements on data sources (e.g., private payload information, social relationships, or activity histories), this article proposes a method called BotFlowMon that relies only on content-agnostic flow-level data as input to identify OSN bot traffic. BotFlowMon introduces several new algorithms and techniques to classify social bot traffic from real OSN user traffic, including aggregating network flow records to obtain OSN transaction data, fusing transaction data to extract features and visualize flows, and an innovative density-valley-based clustering algorithm to subdivide each transaction into individual actions. The evaluation shows BotFlowMon can identify the traffic from social bots with a 96.1 percent accuracy, which, based on the worst case study on a testing machine, only takes no more than 0.71 seconds on average after it sees the traffic.

Processes controlling river-mouth lagoon dynamics on high-energy mixed sand and gravel coasts

The river mouth lagoons of New Zealand's gravel-bed braided rivers are highly dynamic, responding to changes in waves and river flows at timescales of hours and days to years. Unlike other freshwater-dominated coastal wetlands and lagoons, they exist along open coasts that are typically retreating with cliffed hinterlands. These lagoons are common on New Zealand's high energy, coarse sediment coastlines and are known locally as ‘hāpua’. This paper employs field observations from the Hurunui hāpua, to investigate the main processes controlling the dynamics of hāpua lagoons. Based on two years of time-lapse imagery, plus concurrent wave, river flow and sea level data an improved conceptual model of lagoon dynamics was developed. The model describes how onshore bar migration of river sourced gravel and sand drives initial constriction/offsetting of the river mouth following floods. It also explains how gravel deposition in the lagoon reduces the threshold flow required to cause primary breach of the barrier opposite the point the river enters the lagoon. Three processes are identified which affect lagoon width: wave overwashing of the gravel barrier narrows the lagoon; migration of the highly dynamic lagoon outlet channel ‘resets’ the barrier position to seaward; and river floods occurring while the outlet channel is offset from the river flush sediment from the lagoon system, eroding its bed and banks. Both processes affecting lagoon widening rely on lagoon outlet channel migration, indicating that outlet channel dynamics are an important control on lagoon size.

Deep Learning Hierarchical Representation From Heterogeneous Flow-Level Communication Data

The success of a detection model depends heavily on feature engineering. Deep learning has been successfully applied in numerous research fields as a universal representation learning method. However, the heterogeneity of flow-level communication data obstructs the application of deep learning to communication representation learning, and research on this problem is still lacking. To cope with this problem, we propose a heterogeneous communication data-encoding approach to extract fixed-size encoding data to apply deep learning to heterogeneous communication data by preserving the spatiotemporal characteristics of the data. Then, we propose a feature extractor based on deep learning to automatically learn hierarchical and robust communication representations without expert knowledge. We show that the proposed approach can replicate and optimize the key steps of feature engineering well and learn hierarchical representations directly from heterogeneous communication data. Moreover, compared with features extracted with principal component analysis (PCA), manifold learning and manually crafted methods, the features extracted by deep learning are more robust and are characterized by their better adaptability to various classifiers and datasets. To the best of our knowledge, the initial work here is the first to apply deep learning techniques to heterogeneous flow-level data; consequently, the heterogeneous communication data processing method can provide technical means for the application of deep learning in other communication-related research fields.

MalAlert

In recent years, we witness the spreading of a significant variety of malware, which operate and propagate relying on network communications. Due to the staggering growth of traffic in the last years, detecting malicious software has become infeasible on a packet-by-packet basis. In this paper, we address this challenge by investigating malware behaviors and designing a method to detect them relying only on network flow-level data. In our analysis we identify malware types with regards to their impact on a network and the way they achieve their malicious purposes. Leveraging this knowledge, we propose a machine learning-based and privacy-preserving method to detect malware. We evaluate our results on two malware datasets (MalRec and CTU-13) containing traffic of over 65,000 malware samples, as well as one month of network traffic from the University of Oxford containing over 23 billion flows. We show that despite the coarse-grained information provided by network flows and the imbalance between legitimate and malicious traffic, MalAlert can distinguish between different types of malware with the F1 score of 90%.

Security Data Collection and Data Analytics in the Internet: A Survey

Attacks over the Internet are becoming more and more complex and sophisticated. How to detect security threats and measure the security of the Internet arises a significant research topic. For detecting the Internet attacks and measuring its security, collecting different categories of data and employing methods of data analytics are essential. However, the literature still lacks a thorough review on security-related data collection and analytics on the Internet. Therefore, it becomes a necessity to review the current state of the art in order to gain a deep insight on what categories of data should be collected and which methods should be used to detect the Internet attacks and to measure its security. In this paper, we survey existing studies about security-related data collection and analytics for the purpose of measuring the Internet security. We first divide the data related to network security measurement into four categories: 1) packet-level data; 2) flow-level data; 3) connection-level data; and 4) host-level data. For each category of data, we provide a specific classification and discuss its advantages and disadvantages with regard to the Internet security threat detection. We also propose several additional requirements for security-related data analytics in order to make the analytics flexible and scalable. Based on the usage of data categories and the types of data analytic methods, we review current detection methods for distributed denial of service flooding and worm attacks by applying the proposed requirements to evaluate their performance. Finally, based on the completed review, a list of open issues is outlined and future research directions are identified.

A Flexible Sketch-Based Network Traffic Monitoring Infrastructure

Sketch-based data streaming algorithms are used in many network traffic monitoring applications to obtain accurate estimates of traffic flow statistics. However, the current implementation of sketch-based methods in network monitoring is too application-specific. Their flexibility is limited as the hardware implementation of sketch data structure in the network device depends on the measurement tasks. The sketch counters only summarize flow statistics that are relevant to a specific measurement task and cannot be reused for different measurement tasks. In this paper, we propose FlexSketchMon, a system designed to provide the flexibility of using various sketch-based algorithms for traffic monitoring and measurement tasks. FlexSketchMon leverages on a novel data plane architecture that collects traffic flow statistics and provides arbitrary flow aggregations to the monitoring applications. The data plane design comprises a flow counter table and a flow key table for storing flow-level data. FlexSketchMon is implemented on the NetFPGA-SUME platform and is capable of processing network traffic at line rate in a worst-case scenario corresponding to a 64-byte minimum Ethernet frame size. The update of the flow counter table, which is the critical path in the proposed system, can achieve a throughput of 96 Gbps. The simulation results based on a real-world network traffic traces for three monitoring applications-estimation, superspreader detection, and heavy hitter detection-are presented to demonstrate the performance of FlexSketchMon. The results show that FlexSketchMon yields comparable and better measurement accuracy compared to previous approaches.

Temporal understanding of human mobility: A multi-time scale analysis.

The recent availability of digital traces generated by cellphone calls has significantly increased the scientific understanding of human mobility. Until now, however, based on low time resolution measurements, previous works have ignored to study human mobility under various time scales due to sparse and irregular calls, particularly in the era of mobile Internet. In this paper, we introduced Mobile Flow Records, flow-level data access records of online activity of smartphone users, to explore human mobility. Mobile Flow Records collect high-resolution information of large populations. By exploiting this kind of data, we show the models and statistics of human mobility at a large-scale (3,542,235 individuals) and finer-granularity (7.5min). Next, we investigated statistical variations and biases of mobility models caused by different time scales (from 7.5min to 32h), and found that the time scale does influence the mobility model, which indicates a deep coupling of human mobility and time. We further show that mobility behaviors like transportation modes contribute to the diversity of human mobility, by exploring several novel and refined features (e.g., motion speed, duration, and trajectory distance). Particularly, we point out that 2-hour sampling adopted in previous works is insufficient to study detailed motion behaviors. Our work not only offers a macroscopic and microscopic view of spatial-temporal human mobility, but also applies previously unavailable features, both of which are beneficial to the studies on phenomena driven by human mobility.

Groundwater Source Analysis in Fractured Rock Mass Based On Multi-Source Hydrogeology Information

The recharge source of groundwater is an important part of groundwater circulation. Controlled by geological structures, the distribution of groundwater in bedrock area is complicated. The study on the source of groundwater in bedrock has guiding significance for the evaluation of water source site and groundwater exploitation. By analyzing the model of water-rich zone based on remote sensing information, and according to the groundwater dynamics, major ion concentration, velocity and direction of groundwater flow, pumping and water level data of water source site, this article studies the source of groundwater recharge of Barakan Water Source Site(BWSS) in front of Kuruktag, northwest of China, and analyses the reasons for the abnormal change of the total water yield and the drawdown of the well. The results show that the groundwater of the water source site comes from the Aggregate Reclaiming Plant(ARP) about 6.5 kilometers northwest of BWSS. The ARP water pumping is the main reason of the abnormal change of the total water yield and the drawdown of the well of BWSS. Based on this conclusions, the methods to protect the water source site safety are put forward.

The creation, management, and use of data quality information for life cycle assessment.

Despite growing access to data, questions of "best fit" data and the appropriate use of results in supporting decision making still plague the life cycle assessment (LCA) community. This discussion paper addresses revisions to assessing data quality captured in a new US Environmental Protection Agency guidance document as well as additional recommendations on data quality creation, management, and use in LCA databases and studies. Existing data quality systems and approaches in LCA were reviewed and tested. The evaluations resulted in a revision to a commonly used pedigree matrix, for which flow and process level data quality indicators are described, more clarity for scoring criteria, and further guidance on interpretation are given. Increased training for practitioners on data quality application and its limits are recommended. A multi-faceted approach to data quality assessment utilizing the pedigree method alongside uncertainty analysis in result interpretation is recommended. A method of data quality score aggregation is proposed and recommendations for usage of data quality scores in existing data are made to enable improved use of data quality scores in LCA results interpretation. Roles for data generators, data repositories, and data users are described in LCA data quality management. Guidance is provided on using data with data quality scores from other systems alongside data with scores from the new system. The new pedigree matrix and recommended data quality aggregation procedure can now be implemented in openLCA software. Additional ways in which data quality assessment might be improved and expanded are described. Interoperability efforts in LCA data should focus on descriptors to enable user scoring of data quality rather than translation of existing scores. Developing and using data quality indicators for additional dimensions of LCA data, and automation of data quality scoring through metadata extraction and comparison to goal and scope are needed.

MGB-IPH model for hydrological and hydraulic simulation of large floodplain river systems coupled with open source GIS

Large-scale hydrological models are useful tools for water resources studies, however, river network flow routing is generally represented using simplified methods, which may lead to simulation errors in flat regions. We present recent improvements to the large-scale hydrological model MGB-IPH to improve its capability of simulating large river basins with extensive floodplains. We also describe the coupling of MGB-IPH to an open source GIS and a large set of developed pre-processing tools with a user-friendly interface for remote sensing data preparation and output visualization. The new features implemented are demonstrated applying the model to the whole Araguaia river basin (380,000 km2). Results are compared to the previous MGB-IPH routing method, observed flow and water level data and remote sensing imagery, showing improvement in the representation of floodplain inundation dynamics. The test case also shows that the proposed model software framework amplifies possibilities of large-scale simulation of ungauged basins.

Upper washita river experimental watersheds: reservoir, groundwater, and stream flow data.

Surface and groundwater quantity and quality data are essential in many hydrologic applications and to the development of hydrologic and water quality simulation models. We describe the hydrologic data available in the Little Washita River Experimental Watershed (LWREW) of the Southern Great Plains Research Watershed (SGPRW) and Fort Cobb Reservoir Experimental Watershed (FCREW), both located in southwest Oklahoma. Specifically, we describe the flood retarding structures and corresponding stage, discharge, seepage, and consumptive use data (), stream gauges, and groundwater wells and their corresponding stream flow (; LWREW ARS 522-526 stream gauges) and groundwater level data (SGPRW groundwater levels data; LWREW groundwater data; ; ), respectively. Data collection is a collaborative effort between federal and state agencies. Stage, discharge, seepage, and consumptive use data for the Fort Cobb Reservoir are available from the Bureau of Reclamation and cover a period of 1959 to present. There are 15 stream gauges in the LWREW and six in the FCREW with varying data records. There were 479 observation wells with data in the SGPRW and 80 in the LWREW, with the latest records collected in 1992. In addition, groundwater level data are available from five real-time monitoring wells and 34 historical wells within the FCREW. These data sets have been used for several research applications. Plans for detailed groundwater data collection are underway to calibrate and validate the linked Soil and Water Assessment Tool (SWAT)-MODFLOW model. Also, plans are underway to conduct reservoir bathymetric surveys to determine the current reservoir capacity as affected by land use/land cover and overland and stream channel soil erosion.

A Large-Scale Network Data Analysis via Sparse and Low Rank Reconstruction

With the rapid growth of data communications in size and complexity, the threat of malicious activities and computer crimes has increased accordingly as well. Thus, investigating efficient data processing techniques for network operation and management over large-scale network traffic is highly required. Some mathematical approaches on flow-level traffic data have been proposed due to the importance of analyzing the structure and situation of the network. Different from the state-of-the-art studies, we first propose a new decomposition model based on accelerated proximal gradient method for packet-level traffic data. In addition, we present the iterative scheme of the algorithm for network anomaly detection problem, which is termed as NAD-APG. Based on the approach, we carry out the intrusion detection for packet-level network traffic data no matter whether it is polluted by noise or not. Finally, we design a prototype system for network anomalies detection such as Probe and R2L attacks. The experiments have shown that our approach is effective in revealing the patterns of network traffic data and detecting attacks from large-scale network traffic. Moreover, the experiments have demonstrated the robustness of the algorithm as well even when the network traffic is polluted by the large volume anomalies and noise.

A software-based sensor for combined sewer overflows

A new methodology for online estimation of excess flow from combined sewer overflow (CSO) structures based on simulation models is presented. If sufficient flow and water level data from the sewer system is available, no rainfall data are needed to run the model. An inverse rainfall-runoff model was developed to simulate net rainfall based on flow and water level data. Excess flow at all CSO structures in a catchment can then be simulated with a rainfall-runoff model. The method is applied to a case study and results show that the inverse rainfall-runoff model can be used instead of missing rain gauges. Online operation is ensured by software providing an interface to the SCADA-system of the operator and controlling the model. A water quality model could be included to simulate also pollutant concentrations in the excess flow.

Robust Pipelined Memory System with Worst Case Performance Guarantee for Network Processing

Many network processing applications require wirespeed access to large data structures or a large amount of packet and flow-level data. Therefore, it is essential for the memory system of a router to be able to support both read and write accesses to such data at link speeds. As link speeds continue to increase, router designers are constantly grappling with the unfortunate trade-offs between the speed and cost of SRAM and DRAM. The capacity of SRAMs is woefully inadequate in many cases and it proves too costly to store large data structures entirely in SRAM, while DRAM is viewed as too slow for providing wirespeed updates at such high speed. In this paper, we analyze a robust pipelined memory architecture that can emulate an ideal SRAM by guaranteeing with very high probability that the output sequence produced by the pipelined memory architecture is the same as the one produced by an ideal SRAM under the same sequence of memory read and write operations, except time shifted by a fixed pipeline delay of \Delta. Given a fixed pipeline delay abstraction, no interrupt mechanism is required to indicate when read data are ready or a write operation has completed, which greatly simplifies the use of the proposed solution. The design is based on the interleaving of DRAM banks together with the use of a reservation table that serves in part as a data cache. In contrast to prior interleaved memory solutions, our design is robust under all memory access patterns, including adversarial ones, which we demonstrate through a rigorous worst case theoretical analysis using a combination of convex ordering and large deviation theory.