Heavy Hitters Research Articles

MpScope: Enabling multi-pipeline monitoring inside a switch

The core of programmable switches is the flexible data plane, composed of multiple programmable pipelines in existing programmable switches. These pipelines are isolated from each other and cannot share state and data. However, most of network monitoring systems ignore this condition and implicitly assume that the switch has only a single pipeline. This results in an inaccurate measurement and high communication overhead with the practical switch. To tackle this problem, we propose MpScope, a general multi-pipeline monitoring framework, which centers around the control plane, supporting accurate and efficient network monitoring. Specifically, MpScope combines the switch’s data plane and control plane to achieve comprehensive network monitoring of the whole switch scope. The control plane aggregates the statistical results from multiple pipelines and tunes the monitoring module residing in the different pipelines in the data plane dynamically. The data plane is responsible for real-time traffic measurement and statistic reports. Its behaviors can be adjusted periodically with the instructions from the control plane. Two typical monitoring applications, i.e., heavy hitter detection and distinct counting, are developed with MpScope to validate the effectiveness of multi-pipeline monitoring. Experiments show that MpScope significantly reduces communication overhead compared to the static threshold scheme while maintaining high detection accuracy over time.

Better Differentially Private Approximate Histograms and Heavy Hitters using the Misra-Gries Sketch

Better Differentially Private Approximate Histograms and Heavy Hitters using the Misra-Gries Sketch

Technical Perspective on 'Better Differentially Private Approximate Histograms and Heavy Hitters using the Misra-Gries Sketch'

Technical Perspective on 'Better Differentially Private Approximate Histograms and Heavy Hitters using the Misra-Gries Sketch'

Conditional heavy hitter monitoring and application of heterogeneous graph streams based on sketches

Graph streams offer valuable insights into real network evolution in the era of big data. Heavy hitters and conditional heavy hitters focus on identifying entities with significant network traffic, revealing network data characteristics. In this paper, we present two methods, BLSketch and MHLSketch, for identifying heavy hitters and conditional heavy hitters in heterogeneous graph streams. BLSketch uses the bidirectional mapper, Bimap, for efficient query response times while retaining candidate information. MHLSketch uses min heap to preserve a minimal set of candidates to minimize storage space, excelling in space efficiency. The two methods can be incorporated with any sketch based graph summarization method, showing excellent adaptability. Experimental results on four real-world heterogeneous datasets of varying scales with the largest dataset containing millions of edges demonstrate our methods significantly improve the quality of query answering, outperforming state-of-the-art methods, with precision close to 1 and average relative error close to 0. We further combine our methods with density clustering to introduce the SDC model, which efficiently identifies hot regions and key users in Location-Based Social Networks (LBSN). In experiments, SDC outperforms traditional clustering by reducing up to 93.1% of cold Points of Interest (POIs) and shows strong potential for practical applications.

Unbiased Real-Time Traffic Sketching

In network measurement, sliding window measurement has the advantage of providing recent and timely measurement results. Recently, sketches have become the most popular method of conducting flow-level network measurements due to their favorable trade-off between small memory overhead and high measurement accuracy. However, it remains a challenge that no current sketches are able to support unbiased estimation toward flow size measurement, which can improve the performance of tasks including network diagnoses, delay measurement and heavy hitter detection. In this paper, we propose the first work that achieves unbiased flow size measurement in sliding windows, namely <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Unbiased Cleaning sketch (UC sketch)</b> . The key technique of the UC sketch is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Unbiased Cleaning</i> which can remove outdated keys from the sliding windows in a balanced way. Besides, we significantly reduce the variance of flow size by two optimization techniques, namely <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Linear Scaling</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Column Randomizing</i> . To prove the result, we conduct rigorous mathematical analysis and reasonable experiments. All related source codes are open-sourced at Github anonymously.

Who are the heavy hitters? A citation analysis of the most impactful research in the Journal of Experimental Criminology over the past two decades

Who are the heavy hitters? A citation analysis of the most impactful research in the Journal of Experimental Criminology over the past two decades

Local Differentially Private Heavy Hitter Detection in Data Streams with Bounded Memory

Top-k frequent items detection is a fundamental task in data stream mining. Many promising solutions are proposed to improve memory efficiency while still maintaining high accuracy for detecting the Top-k items. Despite the memory efficiency concern, the users could suffer from privacy loss if participating in the task without proper protection, since their contributed local data streams may continually leak sensitive individual information. However, most existing works solely focus on addressing either the memory-efficiency problem or the privacy concerns but seldom jointly, which cannot achieve a satisfactory tradeoff between memory efficiency, privacy protection, and detection accuracy. In this paper, we present a novel framework HG-LDP to achieve accurate Top-k item detection at bounded memory expense, while providing rigorous local differential privacy (LDP) protection. Specifically, we identify two key challenges naturally arising in the task, which reveal that directly applying existing LDP techniques will lead to an inferior "accuracy-privacy-memory efficiency" tradeoff. Therefore, we instantiate three advanced schemes under the framework by designing novel LDP randomization methods, which address the hurdles caused by the large size of the item domain and by the limited space of the memory. We conduct comprehensive experiments on both synthetic and real-world datasets to show that the proposed advanced schemes achieve a superior "accuracy-privacy-memory efficiency" tradeoff, saving 2300× memory over baseline methods when the item domain size is 41,270. Our code is anonymously open-sourced via the link.

ComPipe: A Novel Flow Placement and Measurement Algorithm for Programmable Composite Pipelines

Programmable networks comprise heterogeneous network devices based on both hardware and software. Hardware devices provide superior bandwidth and low latency but encounter challenges in managing large table entries. Conversely, software devices offer abundant flow tables but have a limited forwarding capacity. To overcome this limitation, some commercial switches offer implementations that combine both hardware and software devices. In this context, this paper presents the Composite Pipeline (ComPipe), an algorithm for high-performance and high-precision flow placement and measurement. ComPipe utilizes a multi-level hashing algorithm for the real-time identification of heavy hitters, incorporates a unique flow eviction strategy, and is implemented on commercial programmable hardware. For non-heavy flows, ComPipe employs sketch structures to accomplish a high-performance flow synopsis within limited memory constraints. This design allows to replace flow rules entirely in the data plane, ensuring the timely detection and offloading of heavy-hitter flows, and offering a unified interface to the controller. The ComPipe prototype has been implemented in both testbed and simulation environments. The results indicate that ComPipe is an effective solution for dynamic flow placement in programmable networks, distinguished by its low cost, high performance, and high accuracy.

Resource Critical Flow Monitoring in Software-Defined Networks

Flow monitoring is widely applied in software-defined networks (SDNs) for monitoring network performance. Especially, detecting heavy hitters can prevent the Distributed Denial of Service (DDoS) attack. However, many existing approaches fall into one of two undesirable extremes: (i) inefficient collection where only accuracy is concerned in the method; (ii) sacrifice of accuracy due to fast detection. One practical problem with this is that it does not have the flexibility to adjust the monitoring strategy to the monitoring needs, making it difficult to meet different applications. To alleviate this problem, we propose our design of a novel flow monitoring framework that keeps the balance between accuracy and efficiency. It provides customized monitoring services for applications, where network resources can be saved, and the error rate can also be confined. In this paper, we present cReFeR, a three-step “compression Report-Feedback-Report” framework to monitor SDNs. The IP and the value compressor are specially designed to reduce the volume of flow statistics collection. This framework thus can achieve accuracy-ensured and resource-saving flow monitoring in SDNs. Theoretical analysis and simulated evaluation have proved the effectiveness of our solution. cReFeR keeps the error rate under <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3\%$</tex-math> </inline-formula> and reduces the amount of monitoring data more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$40\%$</tex-math> </inline-formula> , which guarantees high efficiency compared with existing methods.

An Accurate and Invertible Sketch for Super Spread Detection

Super spread detection has been widely applied in network management, recommender systems, and cyberspace security. It is more complicated than heavy hitter owing to the requirement of duplicate removal. Accurately detecting a super spread in real-time with small memory demands remains a nontrivial yet challenging issue. The previous work either had low accuracy or incurred heavy memory overhead and could not provide a precise cardinality estimation. This paper designed an invertible sketch for super spread detection with small memory demands and high accuracy. It introduces a power-weakening increment strategy that creates an environment encouraging sufficient competition at the early stages of discriminating a super spread and amplifying the comparative dominance to maintain accuracy. Extensive experiments have been performed based on actual Internet traffic traces and recommender system datasets. The trace-driven evaluation demonstrates that our sketch actualizes higher accuracy in super spread detection than state-of-the-art sketches. The super spread cardinality estimation error is 2.6–19.6 times lower than that of the previous algorithms.

Heavy Hitter Identification Over Large-Domain Set-Valued Data With Local Differential Privacy

Heavy Hitter Identification Over Large-Domain Set-Valued Data With Local Differential Privacy

Design-of-Experiment (DoE) based history matching for probabilistic integrity analysis—A case study of the FE-experiment at Mont Terri

We present an application of design-of-experiment (DoE) based history matching as an approach to reduce and investigate parameter uncertainties in finite-element models for repositories of high-level radioactive waste. We combine experimental data from the FE-experiment at the Mont Terri underground research laboratory in Switzerland with thermo-hydro-mechanical modeling using the open-source package OpenGeoSys. Uncertainties were reduced by an initial parameter screening to find heavy hitters and an experiment-matching procedure using Monte-Carlo sampling on a Gaussian proxy model to fit the error between modeling response and the experiment. Furthermore, we performed a global sensitivity analysis based on the proxy model, demonstrating the spatial impact of parameter sensitivities. Very good agreement between the experimental data and the model was found for the temperature response, whereas the pressure match hints at a significant remaining gap in the physical models and/or structure. This gap could not be filled within the scope of our contribution and needs further investigation.

MVPipe: Enabling Lightweight Updates and Fast Convergence in Hierarchical Heavy Hitter Detection

Finding hierarchical heavy hitters (HHHs) (i.e., hierarchical aggregates with exceptionally huge amounts of traffic) is critical to network management, yet it is often challenged by the requirements of fast packet processing, real-time and accurate detection, as well as resource efficiency. Existing HHH detection schemes either incur expensive packet updates for multiple aggregation levels in the IP address hierarchy, or need to process sufficient packets to converge to the required detection accuracy. We present, an invertible sketch that achieves both lightweight updates and fast convergence in HHH detection. builds on the skewness property of IP traffic to process packets via a pipeline of majority voting executions, such that most packets can be updated for only one or few aggregation levels in the IP address hierarchy. We show how can be feasibly deployed in P4-based programmable switches subject to limited switch resources. We also theoretically analyze the accuracy and coverage properties of . Evaluation with real-world Internet traces shows that achieves high accuracy, high throughput, and fast convergence compared to six state-of-the-art HHH detection schemes. It also incurs low resource overhead in the Tofino switch deployment.

Finding recently persistent flows in high-speed packet streams based on cuckoo filter

In high-speed networks, flow-level traffic measurement is an essential tool to understand how network bandwidth is consumed and support the detection of anomalous traffic. While many prior work focuses on tracking the frequent flows (a.k.a. heavy hitters), in this paper, we put more focus on tracking the persistent flows, which jointly consider the frequency, duration and regularity of the packet arrival events of a flow. Although this more generalized metric called persistence has been defined before, it is still unknown how to use limited memory on data plane to monitor the top-k persistent flows in a recent time interval. In this paper, we propose an algorithm named PFD-DW built upon cuckoo filter (an improved version of hash table with better memory efficiency), to monitor the top-k persistent flows under the time-decaying window. It can be regarded as a variant of cuckoo filter, which transforms each bucket into a bucket-level min-heap. Its advantage is that, when the table is full and a packet of a new flow arrives, it can select the least persistent flow along the cuckoo kicking path as the victim of flow replacement. We deliberately avoid the scanning of the entire table to keep the high time efficiency. Based on real-world network traffic traces, we evaluate the performance of our DAKP-CF, a degraded version of PFD-DW that only considers each flow’s packet arrival frequency. The results show that it outperforms the existing algorithms for the top-k frequent flow identification task. It provides nearly 98% identification accuracy with roughly 50% less memory cost. We also evaluate our PFD-DW algorithm for the more generalized task of identifying the recently persistent flows. It provides 93% identification rate of top-3000 persistent flows using only 576KB memory, while attaining 1.5% persistence estimation error. Its memory cost is reduced by 97% than another proposed solution PFD-SW under sliding time window. We have also developed a prototype of PFD-DW, based on the Fd.io VPP software switch accelerated by Intel DPDK.

Universal and Accurate Sketch for Estimating Heavy Hitters and Moments in Data Streams

In computer networks, traffic measurement is a module in a network probe to measure flow-level statistics from an IP packet stream, which are the basis for network performance monitoring and malicious activity detection. This module extracts the flow IDs from incoming IP packets, classifies packets into flows, and counts the number of packets (or bytes) for each flow. It is a great challenge to measure the per-flow statistics for a high-speed network device, using only the size-limited SRAM on its line cards. Therefore, many algorithms using sublinear memory have been proposed, such as CountMin and CountSketch. However, most of previous algorithms are designed for specific measurement tasks. To obtain multiple types of statistics, people have to deploy multiple sketches, which demands more resources of a network device. It is useful to design a universal sketch that can track not only the top- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$</tex-math> </inline-formula> largest individual flows (called heavy hitters) but also the overall traffic distribution statistics (called moments). Prior work named UnivMon successfully tackled this ambitious quest. However, it incurs large and variable per-packet processing overhead, which may result in a significant throughput bottleneck in high-rate packet stream, given that each packet requires 33 hashes and 32 memory accesses on average and many times of that in the worst case. To address this performance issue, we fundamentally redesign the solution architecture from hierarchical sampling to new progressive sampling and from CountSketch to new GenericCM, which ensure that per-packet overhead is a small constant (5 hashes and 8 memory accesses in the worst case), making it more suitable for online operations, especially for hardware pipeline implementation. This new design also makes effort to reduce memory footprint or equivalently improve measurement accuracy under the same memory. Our experiments show that our solution reduces measurement error by roughly 98.1% for second-order moment and by 91.5% for entropy, when given the same 0.2MB memory as UnivMon.

HH-IPG: Leveraging Inter-Packet Gap Metrics in P4 Hardware for Heavy Hitter Detection

The research community has recently proposed several solutions based on modern programmable switches to detect entirely in the data plane the flows exceeding pre-determined thra eshold in a time window, i.e., Heavy Hitters (HH). This is commonly achieved by dividing the network stream into fixed time slots and identifying each separately without considering the traffic trends from previous intervals. In this work, we show that using specified time windows can lead to high inaccuracies. We make a case for rethinking how switches analyze the incoming packets and propose to leverage per-flow Inter Packet Gap (IPG) analytics instead of using flow counters for HH detection. We propose an algorithm and present a P4 pipeline design using this new metric in mind. We implement our solution on P4 hardware and experimentally evaluate it against real traffic traces. We show that our results are more accurate than related work by up to 20% while reducing the control channel overhead by up to two orders of magnitude. Finally, we showcase a QoS-oriented application of the proposed dataplane-only IPG-based HH detection in a mobile network scenario.

Cuckoo Counter: Adaptive Structure of Counters for Accurate Frequency and Top-k Estimation

Frequency estimation and top-k flows identification are fundamental problems in network traffic measurement. Sketch, as a basic probabilistic data structure, has been extensively investigated and used in different management applications. However, few of them is suitable for both estimating frequency and finding top-k flows due to the unbalanced distribution of real-world network streams. By introducing a pre-filtering stage to isolate elephant and mice flows, the recently proposed Augmented Sketch (ASketch) significantly improves accuracy for both tasks. However, it suffers from serious performance degradation because of frequent flow exchanges. In this paper, we propose Cuckoo Counter (CC), an adaptive structure that consists of several buckets organized in a specific way. The size of the entry in each bucket is carefully designed to match the actual distribution of streams. During processing, CC hashes a flow to buckets and uses the idea of cuckoo hashing to relocate the flow if an overflow or collision happens, which contributes to fully utilizing memory. Therefore, the replacement strategy helps CC precisely record elephant flows and cover more mice flows, and also guarantees the throughput. Extensive experimental results show that CC has the highest (Freq.) accuracy, excellent (Heavy hitter / change) accuracy, highest (Top-k) precision, and competitive throughput compared to the state-of-the-art. Specifically, CC improves the throughput and accuracy by around 1 and 2 orders of magnitude respectively compared to the well-known ASketch.

Sustainable Livelihoods a Foundation for Rural Development Leads to Sustainability

The primary objective of this study is to evaluate and analyse the significance of the Sustainable Livelihoods Approach (SLA) and its application to rural development projects and policies. A literature review is conducted, with the primary focus being on the primary components of the SLA. The researchers noted that the SLA effectively interrogates the livelihoods of the poor and the various mechanisms in the approach. These mechanisms include all forms of capital, the vulnerability aspect, livelihood strategies, and outcomes, as well as the different laws and regulations governing the access and use of resources. The research also noted that the SLA effectively addresses the poor’s vulnerabilities. Scholarly heavy hitters like Chambers and Scoones (1992) have pointed out that this method is all-encompassing and hierarchical. The methodology has demonstrated in a theoretical sense that it is beneficial in understanding the livelihoods problem of the less fortunate in rural communities. It is strongly suggested that the government and its development partners adopt and incorporate the SLA into their policies to ensure sustainable livelihoods leading to sustainable development.

Tricking the Hashing Trick: A Tight Lower Bound on the Robustness of CountSketch to Adaptive Inputs

CountSketch and Feature Hashing (the ``hashing trick'') are popular randomized dimensionality reduction methods that support recovery of l2 -heavy hitters and approximate inner products. When the inputs are not adaptive (do not depend on prior outputs), classic estimators applied to a sketch of size O(l / epsilon) are accurate for a number of queries that is exponential in l. When inputs are adaptive, however, an adversarial input can be constructed after O(l) queries with the classic estimator and the best known robust estimator only supports ~O(l^2) queries. In this work we show that this quadratic dependence is in a sense inherent: We design an attack that after O(l^2) queries produces an adversarial input vector whose sketch is highly biased. Our attack uses ``natural'' non-adaptive inputs (only the final adversarial input is chosen adaptively) and universally applies with any correct estimator, including one that is unknown to the attacker. In that, we expose inherent vulnerability of this fundamental method.

Network Monitoring on Multi-Pipe Switches

Programmable switches have been widely used to design network monitoring solutions that operate in the fast data-plane level, e.g., detecting heavy hitters, super-spreaders, computing flow size distributions and their entropy. Existing works assume packets access the same memory region in a switch. However, high-speed ASIC switches deploy multiple packet processing pipes, each equipped with its own independent memory. In this work, we first quantify the accuracy degradation due to splitting a monitoring data structure across multiple pipes (e.g., up to 3000x worse flow-size estimation average error). We then present PipeCache, a system that adapts existing data-plane mechanisms to multi-pipe switches by storing monitoring information for a traffic class into a single pipe. PipeCache stores monitoring information into a cache and piggybacks this information onto existing data packets to the correct pipe. Our implementation shows a 2-20x memory reduction to achieve an accuracy similar to single-pipe deployments.