Stream Processing Techniques for Network Management

Abstract

For network operators, understanding the types and volumes of traffic carried on the Internet is fundamental to maintaining its stability, reliability, security, and performance. Having efficient and comprehensive network monitoring systems is the key to achieving this understanding. The process of network monitoring varies in complexity from simple long term collection of link utilization statistics to complicated ad-hoc upper-layer protocol analysis for detecting network intrusions, tuning network performance, and debugging protocols. Existing network monitoring tools suffer from critical shortcomings and can no longer fully address network monitoring and debugging needs. To address these problems, we have created Gigascope—a fast and flexible stream database for network monitoring. Gigascope was designed around two key aspects. First, Gigascope has a highly flexible SQL-like query language, GSQL, for its interface. Using a database query language provides us with great flexibility and allows Gigascope to be quickly adapted for new problems—only the high-level query need be changed. Second, Gigascope was designed using the overriding principle of reducing data as early as possible to allow high-speed monitoring. Gigascope queries are automatically broken up into hierarchical components. Low-level components can run on the network interface card itself, reducing data before it reaches the main system bus. High-level query components may run either in kernel or user space and can be used to extract application layer information from the network.