IntStream: Towards Flexible, Expressive, and Scalable Network Telemetry

Abstract

Due to the complexity of the network structure and the high growth of the transmission speed, the measurement and management of the network are facing serious challenges. The traditional bottom-up network telemetry methods are no longer applicable to complex network scenarios. To bridge this gap, we propose IntStream, a flexible, expressive and scalable network telemetry framework to allow network operators to measure and analyze network through passive stream processing and active probing. However, there are three key challenges to building an intent-based telemetry system: (1) The diversity of network data sources. (2) The complexity of the measurement tasks. (3) The low overhead requirements of the telemetry system. IntStream introduces a lightweight component to extract and parse data from various data sources and divides the data stream processing into local and global stages. IntStream provides a set of rich expressive primitives to support operators to write telemetry tasks based on intent. By performing part of the telemetry task on the local stage, the transmission overhead of intermediate data can be effectively reduced. The evaluation results conducted on a large campus network show that IntStream can support a wide range of telemetry tasks while reducing the intermediate data transmission overhead by 99.31% on average.