IP Flow Information Export Research Articles

Combining Stochastic and Deterministic Modeling of IPFIX Records to Infer Connected IoT Devices in Residential ISP Networks

Residential Internet service providers (ISPs) today have limited device-level visibility into subscriber houses, primarily due to the network address translation (NAT) technology. The continuous growth of “unmanaged” consumer Internet of Things (IoT) devices combined with the rise of work-from-home makes home networks attractive targets to sophisticated cyber attackers. Volumetric attacks sourced from a distributed set of vulnerable IoT devices can impact ISPs by deteriorating the performance of their network, or even making them liable for being a carrier of malicious traffic. This article explains how ISPs can employ IP Flow Information eXport (IPFIX), a flow-level telemetry protocol available on their network, to infer connected IoT devices and ensure their cyber health without making changes to home networks. Our contributions are threefold: 1) we analyze more than nine million IPFIX records of 26 IoT devices collected from a residential testbed over three months and identify 28 flow features pertinent to their network activity that characterize the network behavior of IoT devices—we release our IPFIX records as open data to the public; 2) we train a multiclass classifier on stochastic attributes of IPFIX flows to infer the presence of certain IoT device types in a home network with an average accuracy of 96%. On top of the machine learning (ML) model, we develop a trust metric to track network activity of detected devices over time; and 3) finally, we develop deterministic models (DTs) of specific and shared cloud services consumed by IoTs, yielding an average accuracy of 92%. We show a combination of stochastic and DTs mitigates false positives in 75% of incidents at the expense of an average 7% reduction in true positives.

Why are My Flows Different? A Tutorial on Flow Exporters

Network flows build the basis of modern network data analysis by aggregating properties of network packets with common characteristics. A consistent and unambiguous definition of the network flow concept is an indispensable prerequisite and starting point for reproducible network research. However, in today’s practice, the flow output of distinct flow exporters, which is software to generate flows from observed network packets, varies substantially on identical network packet stream input. In this paper we present an in-depth comparison of different flow exporters and show how their outputs differ significantly. We argue that this substantially impairs reproducibility for traffic analysis research. We first present the detailed flow definition of the IP Flow Information eXport (IPFIX) standard including explanations and examples, analyze design and implementation of existing flow exporters, and explore the reasons why many projects and publications chose to implement their own flow exporters. Based on this analysis we highlight the main challenges in the flow exporting process and present a detailed tutorial on how to design and implement a flow exporter such that it yields consistent, reproducible output. Based on the tutorial’s theoretical analysis and lessons learned we present design and main concepts of a versatile, flexible, and open source flow exporting solution called go-flows that generates deterministic, reproducible network flows. Finally, we present a flow-by-flow comparison of the analyzed flow exporters’ output, explore the differences in terms of their generated flows, compare flow exporter performance, and conclude with guidelines on parameters that play a crucial role in improving the reproducibility of exported flows.

Analysis of network traffic features for anomaly detection

Anomaly detection in communication networks provides the basis for the uncovering of novel attacks, misconfigurations and network failures. Resource constraints for data storage, transmission and processing make it beneficial to restrict input data to features that are (a) highly relevant for the detection task and (b) easily derivable from network observations without expensive operations. Removing strong correlated, redundant and irrelevant features also improves the detection quality for many algorithms that are based on learning techniques. In this paper we address the feature selection problem for network traffic based anomaly detection. We propose a multi-stage feature selection method using filters and stepwise regression wrappers. Our analysis is based on 41 widely-adopted traffic features that are presented in several commonly used traffic data sets. With our combined feature selection method we could reduce the original feature vectors from 41 to only 16 features. We tested our results with five fundamentally different classifiers, observing no significant reduction of the detection performance. In order to quantify the practical benefits of our results, we analyzed the costs for generating individual features from standard IP Flow Information Export records, available at many routers. We show that we can eliminate 13 very costly features and thus reducing the computational effort for on-line feature generation from live traffic observations at network nodes.

TinyIPFIX: An efficient application protocol for data exchange in cyber physical systems

Wireless sensor networks (WSNs) as a central part of cyber-physical systems are gaining commercial momentum in many areas, including building monitoring and intelligent home automation. Users wish to successively deploy hardware from different vendors. Interoperability is taken for granted by the customers who want to avoid the need for exhaustive configuration and set-up. Therefore, the need for an interoperable and efficient application layer protocol for machine-to-machine communication in and across the boundaries of WSNs arises. We address these issues with our implementation of TinyIPFIX, an adaption of the IP Flow Information Export (IPFIX) protocol. Throughout the paper we show how to leverage TinyIPFIX in the context of an office scenario and we discuss how the protocol may be applied to other significant WSN deployments presented in literature over the past few years. This article additionally shows how to improve the functionality of TinyIPFIX by adding both syntactic and semantic aggregation functionality to the established system. Finally, we evaluate the performance of TinyIPFIX in a large test bed with over 40 motes running TinyOS and analyze TinyIPFIX’s system performance in comparison with previous approaches.

An introduction to IP flow information export (IPFIX)

The IP Flow Information Export protocol (IPFIX) is an IETF Proposed Standard for the export of information about network flows in IP networks. It is the logical successor to Cisco Net- Flow version 9, upon which it is based. The key innovations of IPFIX are the flexible definition of a network flow and the runtime description of record formats through templates based on a well defined, extensible information model. These allow IPFIX to export flow information from present as well as future networks, and make it applicable to network management beyond the network and transport layers. In this article, we describe the protocol, from its motivation and history through its design and implementation, and explore its deployment within a network monitoring research project.