Heavy Hitter Detection Research Articles

MpScope: Enabling multi-pipeline monitoring inside a switch

The core of programmable switches is the flexible data plane, composed of multiple programmable pipelines in existing programmable switches. These pipelines are isolated from each other and cannot share state and data. However, most of network monitoring systems ignore this condition and implicitly assume that the switch has only a single pipeline. This results in an inaccurate measurement and high communication overhead with the practical switch. To tackle this problem, we propose MpScope, a general multi-pipeline monitoring framework, which centers around the control plane, supporting accurate and efficient network monitoring. Specifically, MpScope combines the switch’s data plane and control plane to achieve comprehensive network monitoring of the whole switch scope. The control plane aggregates the statistical results from multiple pipelines and tunes the monitoring module residing in the different pipelines in the data plane dynamically. The data plane is responsible for real-time traffic measurement and statistic reports. Its behaviors can be adjusted periodically with the instructions from the control plane. Two typical monitoring applications, i.e., heavy hitter detection and distinct counting, are developed with MpScope to validate the effectiveness of multi-pipeline monitoring. Experiments show that MpScope significantly reduces communication overhead compared to the static threshold scheme while maintaining high detection accuracy over time.

Unbiased Real-Time Traffic Sketching

In network measurement, sliding window measurement has the advantage of providing recent and timely measurement results. Recently, sketches have become the most popular method of conducting flow-level network measurements due to their favorable trade-off between small memory overhead and high measurement accuracy. However, it remains a challenge that no current sketches are able to support unbiased estimation toward flow size measurement, which can improve the performance of tasks including network diagnoses, delay measurement and heavy hitter detection. In this paper, we propose the first work that achieves unbiased flow size measurement in sliding windows, namely <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Unbiased Cleaning sketch (UC sketch)</b> . The key technique of the UC sketch is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Unbiased Cleaning</i> which can remove outdated keys from the sliding windows in a balanced way. Besides, we significantly reduce the variance of flow size by two optimization techniques, namely <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Linear Scaling</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Column Randomizing</i> . To prove the result, we conduct rigorous mathematical analysis and reasonable experiments. All related source codes are open-sourced at Github anonymously.

Local Differentially Private Heavy Hitter Detection in Data Streams with Bounded Memory

Top-k frequent items detection is a fundamental task in data stream mining. Many promising solutions are proposed to improve memory efficiency while still maintaining high accuracy for detecting the Top-k items. Despite the memory efficiency concern, the users could suffer from privacy loss if participating in the task without proper protection, since their contributed local data streams may continually leak sensitive individual information. However, most existing works solely focus on addressing either the memory-efficiency problem or the privacy concerns but seldom jointly, which cannot achieve a satisfactory tradeoff between memory efficiency, privacy protection, and detection accuracy. In this paper, we present a novel framework HG-LDP to achieve accurate Top-k item detection at bounded memory expense, while providing rigorous local differential privacy (LDP) protection. Specifically, we identify two key challenges naturally arising in the task, which reveal that directly applying existing LDP techniques will lead to an inferior "accuracy-privacy-memory efficiency" tradeoff. Therefore, we instantiate three advanced schemes under the framework by designing novel LDP randomization methods, which address the hurdles caused by the large size of the item domain and by the limited space of the memory. We conduct comprehensive experiments on both synthetic and real-world datasets to show that the proposed advanced schemes achieve a superior "accuracy-privacy-memory efficiency" tradeoff, saving 2300× memory over baseline methods when the item domain size is 41,270. Our code is anonymously open-sourced via the link.

MVPipe: Enabling Lightweight Updates and Fast Convergence in Hierarchical Heavy Hitter Detection

Finding hierarchical heavy hitters (HHHs) (i.e., hierarchical aggregates with exceptionally huge amounts of traffic) is critical to network management, yet it is often challenged by the requirements of fast packet processing, real-time and accurate detection, as well as resource efficiency. Existing HHH detection schemes either incur expensive packet updates for multiple aggregation levels in the IP address hierarchy, or need to process sufficient packets to converge to the required detection accuracy. We present, an invertible sketch that achieves both lightweight updates and fast convergence in HHH detection. builds on the skewness property of IP traffic to process packets via a pipeline of majority voting executions, such that most packets can be updated for only one or few aggregation levels in the IP address hierarchy. We show how can be feasibly deployed in P4-based programmable switches subject to limited switch resources. We also theoretically analyze the accuracy and coverage properties of . Evaluation with real-world Internet traces shows that achieves high accuracy, high throughput, and fast convergence compared to six state-of-the-art HHH detection schemes. It also incurs low resource overhead in the Tofino switch deployment.

Tree sketch: An accurate and memory-efficient sketch for network-wide measurement

Traffic measurement provides essential information for bandwidth management and Quality of Service (QoS), which is an enabler of network status discovery. However, with the explosive growth of network traffic, traditional network measurement methods face challenges in terms of memory, computation, accuracy, especially in the high-speed networks. Network devices have no enough CPU or memory resources for collecting, counting and analyzing data streams, which makes measurement results inaccurate and unreliable. To address the problem, we design an accurate and memory-efficient Tree sketch. It consists of three different structures for heavy flow detection, Distributed Denial-of-Service (DDoS) attack detection, super-spreaders detection, respectively. The proposed heavy cuckoo algorithm stores the flows in categories according to their flow sizes reducing hash collisions in the sketch. Besides, Tree sketch employs the Single Instruction Multiple Data (SIMD) instructions to speed up packet processing. Experimental results show that F1 score of Tree sketch is 0.997 with 30 KB memory usage in heavy hitter detection, which is 0.019–0.137 higher than the state-of-the-art sketch algorithms. It is an efficient tool of traffic measurement in the backbone network and provides a good trade-off among accuracy, speed and memory usage.

Efficient and Accurate Flow Record Collection With HashFlow

Traditional tools like NetFlow face great challenges as both the speed and the complexity of the network traffic increase. To keep the pace up, we propose HashFlow for more efficient and accurate collection of flow records. HashFlow keeps large flows in its main flow table and uses an ancillary table to summarize the other flows when the main table is full. With our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">flow collision resolution</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">flow record promotion</i> schemes, a flow in the ancillary table is promoted back to the main flow table with a guaranteed probability when it becomes large enough. These operations can be performed highly efficiently, so HashFlow can keep up with ultra-high traffic speed. We implement HashFlow in a Tofino switch, and using traces from different operational networks, we compare its performance against some state-of-the-art flow measurement algorithms. Our experiments show that, for various types of traffic analysis applications, HashFlow consistently demonstrates clearly better performance than its competitors. For example, the performance of HashFlow in flow size estimation, flow size distribution estimation and heavy hitter detection is up to 21, 60 and 35 percent better than those of the best competitors respectively, and these merits of HashFlow come with almost no degradation of throughput.

Low Complexity Sequential Search With Size-Dependent Measurement Noise

This paper considers a target localization problem where at any given time an agent can choose a region to query for the presence of the target in that region. The measurement noise is assumed to be increasing with the size of the query region the agent chooses. Motivated by practical applications such as initial beam alignment in array processing, heavy hitter detection in networking, and visual search in robotics, we consider practically important complexity constraints/metrics: time complexity, computational and memory complexity, and the complexity of possible query sets in terms of geometry and cardinality. Two novel search strategy, dyaPM and hiePM, are proposed. Pertinent to the practicality of our solutions, dyaPM and hiePM are of a connected query geometry (i.e. query set is always a connected set) implemented with low computational and memory complexity. Additionally, hiePM has a hierarchical structure and, hence, a further reduction in the cardinality of possible query sets, making hiePM practically suitable for applications such as beamforming in array processing where memory limitations favors a small and predefined query sets. Through a unified analysis with Extrinsic Jensen Shannon (EJS) Divergence, dyaPM is shown to be asymptotically optimal in search time complexity (asymptotic in both resolution (rate) and error (reliability)). On the other hand, hiePM is shown to be near-optimal in rate. In addition, both hiePM and dyaPM are shown to outperform prior work in the non-asymptotic regime.

Joint Routing and Sketch Configuration in Software-Defined Networking

Traffic measurement is very important for various applications, such as traffic engineering and attack detection, in software-defined networks. Due to limited resources ( e.g. , computing, memory) on SDN switches, sketches have been widely used for efficient traffic measurement. Meanwhile, multiple independent sketches are required for different application requirements, such as heavy hitter detection and flow size distribution estimation. To avoid the measurement redundancy, we configure which sketch(es) will measure flow on each switch. If traffic measurement is performed on each switch without considering network routing, due to traffic dynamics, it will cause massive measurement overhead on a switch, which may exceed the switch’s computing capacity. In this paper, we study the joint optimization of flow routing and sketch configuration in SDNs. We formulate this problem as an integer linear programming and prove its NP-hardness. To solve this problem, we propose a rounding-based offline algorithm and a primal-dual-based online algorithm for different application scenarios. To deal with bursty traffic, we also design an adaptive sampling mechanism for high-fidelity traffic measurement. We formally analyze the approximation performance or competitive ratio of the proposed algorithms. The extensive simulation results show the high efficiency of our proposed algorithms. For example, the online algorithm can improve the network throughput 30% compared with the state-of-the-art.

On Frequency Estimation and Detection of Heavy Hitters in Data Streams

A stream can be thought of as a very large set of data, sometimes even infinite, which arrives sequentially and must be processed without the possibility of being stored. In fact, the memory available to the algorithm is limited and it is not possible to store the whole stream of data which is instead scanned upon arrival and summarized through a succinct data structure in order to maintain only the information of interest. Two of the main tasks related to data stream processing are frequency estimation and heavy hitter detection. The frequency estimation problem requires estimating the frequency of each item, that is the number of times or the weight with which each appears in the stream, while heavy hitter detection means the detection of all those items with a frequency higher than a fixed threshold. In this work we design and analyze ACMSS, an algorithm for frequency estimation and heavy hitter detection, and compare it against the state of the art ASketch algorithm. We show that, given the same budgeted amount of memory, for the task of frequency estimation our algorithm outperforms ASketch with regard to accuracy. Furthermore, we show that, under the assumptions stated by its authors, ASketch may not be able to report all of the heavy hitters whilst ACMSS will provide with high probability the full list of heavy hitters.

Designing Heavy-Hitter Detection Algorithms for Programmable Switches

Programmable network switches promise flexibility and high throughput, enabling applications such as load balancing and traffic engineering. Network measurement is a fundamental building block for such applications, including tasks such as the identification of heavy hitters (largest flows) or the detection of traffic changes. However, high-throughput packet processing architectures place certain limitations on the programming model, such as restricted branching, limited capability for memory access, and a limited number of processing stages. These limitations restrict the types of measurement algorithms that can run on programmable switches. In this paper, we focus on the Reconfigurable Match Tables (RMT) programmable high-throughput switch architecture, and carefully examine its constraints on designing measurement algorithms. We demonstrate our findings while solving the heavy hitter problem. We introduce PRECISION, an algorithm that uses Partial Recirculation to find top flows on a programmable switch. By recirculating a small fraction of packets, PRECISION simplifies the access to stateful memory to conform with RMT limitations and achieves higher accuracy than previous heavy hitter detection algorithms that avoid recirculation. We also evaluate each of the adaptations made by PRECISION and analyze its effect on the measurement accuracy. Finally, we suggest two algorithms for the hierarchical heavy hitters detection problem in which the goal is identifying the subnets that send excessive traffic and are potentially malicious. To the best of our knowledge, our work is the first to do so on RMT switches.

Reduction of Table Flow Occupancy and Packet Loss Detection in SDN Switch

Software Defined Networking (SDN) facilitating network management and monitoring experiences some inevitable problems which affects the performance in terms of delay, on revisiting the limited flow table size problem, a major roadblock to individual flow management of all flows in SDN. We explicitly defined the objective and cost functions that we are interested in, and presented the optimum (i.e., best theoretically possible) solution for flow promotion to achieve it. Then we used the optimum solution as an inspiration to propose a new flow promotion that predicts TCP flow termination (expedited eviction) and incubates non-TCP flows (delayed installation). We provided simulation results that compared our new proposed delayed installation and expedited eviction approach with the default Open Flow approach, the optimum approach, and size-based and rate-based heavy hitter detection using fuzzy model. The results demonstrate that not only our approach provides significant savings in comparison to the default Open Flow approach, it can also perform better than heavy hitter detectors when we use small idle timeouts.

SDN Soft Computing Application for Detecting Heavy Hitters

To avoid distributed denial-of-service (DDoS) attacks or real-time transmission control protocol (TCP) incast in the software-defined networking (SDN) environment, the HashPipe algorithm was developed following the space-saving approach. Unfortunately, HashPipe implemented in the behavioral model (bmv2) cannot be directly executed at a real programming protocol-independent packet processor (P4) switch due to P4 pipeline limitation. Based on the Banzai machine model, this paper shows how to smartly utilize the Banzai atoms to develop HashPipe as a soft computing application in a real P4 switch. Then we propose an enhanced HashPipe algorithm that significantly improves the accuracy of the original HashPipe. The proposed heavy hitter detection is executed at the line-rate of the Tofino P4 switch with the highest process rate in the world.

A Distributed Hierarchical Heavy Hitter Detection Method in Software-Defined Networking

Software-defined networking (SDN) enables fast innovation in networks. The management and measurement features that we require can be easily implemented in SDN. However, when simplicity is introduced, SDN also faces some problems, of which the most challenging one is the scalability problem. As the foundation of the networking management, scalability of the traffic measurement is also important. Recently, many works have focused on TCAM-based measurement, which is considered to be scalable and efficient enough. In this paper, we propose a distributed hierarchical heavy hitter (HHH) detection method, which is also a TCAM-based method. Unlike previous works, this method focuses on optimizing the detection speed by dynamically controlling the resource allocated to each measurement task. We propose an efficient solution to solve the optimization problem. The simulation with network-wide tasks and diverse traffic has shown that this method can improve the detection speed when compared with other resource allocation methods, and it can work better under strict resource limitation.

Delayed Installation and Expedited Eviction: An Alternative Approach to Reduce Flow Table Occupancy in SDN Switches

Limited flow table size in switches is a major concern for SDN applications. The common approach to overcome this problem is to identify elephant flows and solely focus on them. However, there is no gold standard to assess the effectiveness of such greedy solutions. In this paper, we formally define this problem by choosing a cost function ( hit ratio ) and an objective function to optimize the ( average table occupancy ) and present the optimum solution (i.e., theoretical gold standard) for it. We model the problem as a knapsack problem, analyze how its solution minimizes the table occupancy, and the similarities to and differences from the default idle timeout mechanism used in OpenFlow. We also present a new approach to minimize flow table occupancy based on the insight gained from the knapsack model analysis. Our solution expedites rule evictions by forecasting the TCP flow termination from RST/FIN packets and delays rule installation by incubating non-TCP flows. It reduces average flow table occupancy between 16%–62% in various networks with less than 1.5% reduction in hit ratio. Using three real-world packet traces, we compare the performance of our solution with the theoretically optimum solution, the static idle timeout approach used in current OpenFlow systems, and heavy hitter detection approaches that are commonly used to solve this problem. We provide in-depth analysis of when and where our approach outperforms other solutions, while discussing why it might be better to use rate-based heavy hitter detection in some scenarios.

Sketch Acceleration on FPGA and its Applications in Network Anomaly Detection

Sketch, a highly accurate data stream summarization technique, has gained much interest in the research community in recent years. Because of its sub-linear memory complexity, Sketch-based techniques consume significantly less memory than the traditional per-item-state techniques for processing high throughput data streams. One of the major applications of Sketch is in network anomaly detection, which is critical for network management and security in both Internet and data centers. In these applications, throughput is a key performance metric. Due to the low memory complexity, Sketch-based techniques can be supported by the fast onchip storage of the state-of-the-art computing platforms to achieve high throughput. In this work, we first propose a generic architecture on FPGA to accelerate Sketch and adopt it to 2 widely used Sketches: Count-min Sketch and K-ary Sketch. We propose online Sketchbased algorithms for 2 key network anomaly detection tasks: heavy hitter detection and heavy change detection. We adopt the proposed generic architecture for Sketch to accelerate these online algorithms. The post place-and-route results on a state-of-the-art FPGA show that our generic architecture can accelerate both Count-min Sketch and K-ary Sketch to over 150Gbps, demonstrating significant throughput performance improvements compared with other Sketch acceleration techniques. Our architectures for online anomaly detection tasks sustain 100-150 Gbps throughput for various system configurations.

A learning-based measurement framework for traffic matrix inference in software defined networks

In this paper, we propose an intelligent framework for Traffic Matrix (TM) inference in Software Defined Networks (SDN) where the Ternary Content Addressable Memory (TCAM) entries of switches are partitioned into two parts to: 1) effectively aggregate part of incoming flows for aggregate measurements, and 2) de-aggregate and directly measure the most informative flows for per-flow measurements. These measurements are then processed to effectively estimate the size of network flows. Under hard resource constraints of limited TCAM sizes, we show how to design the optimal and efficient-compressed flow aggregation matrices. We propose an optimal Multi-Armed Bandit (MAB) based algorithm to adaptively measure the most rewarding flows. We evaluate the performance of our framework using real traffic traces from different network environments and by considering two main applications: TM estimation and Heavy Hitter (HH) detection. Moreover, we have implemented a prototype of our framework in Mininet to demonstrate its effectiveness.

SDPA: Toward a Stateful Data Plane in Software-Defined Networking

As the prevailing technique of software-defined networking (SDN), open flow introduces significant programmability, granularity, and flexibility for many network applications to effectively manage and process network flows. However, open flow only provides a simple “match-action” paradigm and lacks the functionality of stateful forwarding for the SDN data plane, which limits its ability to support advanced network applications. Heavily relying on SDN controllers for all state maintenance incurs both scalability and performance issues. In this paper, we propose a novel stateful data plane architecture (SDPA) for the SDN data plane. A co-processing unit, forwarding processor (FP), is designed for SDN switches to manage state information through new instructions and state tables. We design and implement an extended open flow protocol to support the communication between the controller and FP. To demonstrate the practicality and feasibility of our approach, we implement both software and hardware prototypes of SDPA switches, and develop a sample network function chain with stateful firewall, domain name system (DNS) reflection defense, and heavy hitter detection applications in one SDPA-based switch. Experimental results show that the SDPA architecture can effectively improve the forwarding efficiency with manageable processing overhead for those applications that need stateful forwarding in SDN-based networks.

High Throughput Sketch Based Online Heavy Hitter Detection on FPGA

In the context of networking, a heavy hitter is an entity in a data stream whose amount of activity (such as bandwidth consumption or number of connections) is higher than a given threshold. Detecting heavy hitters is a critical task for network management and security in the Internet and data centers. Data streams in modern network usually contain millions of entities, such as traffic flows or IP domains. It is challenging to detect heavy hitters at a high throughput while supporting such a large number of entities. I this work, we propose a high throughput online heavy hitter detector based on the Count-min sketch algorithm on FPGA. We propose a high throughput hash computation architecture, optimize the Count-min sketch for hardwarebased heavy hitter detection and use forwarding to deal with data hazards. The post place-and-route results of our architecture on a state-of-the-art FPGA shows high throughput and scalability. Our architecture achieves a throughput of 114 Gbps while supporting a typical 1 M concurrent entities. It sustains 100+ Gbps throughput while supporting various number of concurrent entities, stream sizes and accuracy requirements. Our implementation demonstrates improved performance compared with other sketch acceleration techniques on various platforms using similar sketch configurations.

Optimal Nested Test Plan for Combinatorial Quantitative Group Testing

We consider the quantitative group testing problem where the objective is to identify defective items in a given population based on results of tests performed on subsets of the population. Under the quantitative group testing model, the result of each test reveals the number of defective items in the tested group. The minimum number of tests achievable by nested test plans was established by Aigner and Schughart in 1985 within a minimax framework. The optimal nested test plan offering this performance, however, was not obtained. In this paper, we establish the optimal nested test plan in a closed form. This optimal nested test plan is also order optimal among all test plans as the population size approaches infinity. Using heavy-hitter detection as a case study, we show via simulation examples orders of magnitude improvement of the group testing approach over two prevailing sampling-based approaches in detection accuracy and counter consumption. Other applications include anomaly detection and wideband spectrum sensing in cognitive radio systems.