Packet Traces Research Articles

A Fast Regular Expressions Matching Algorithm for NIDS

In this paper, we propose a new algorithm to accelerate the searching speed in network intrusion detection system (NIDS) and we implement our algorithm in Snort, a popular open-source intrusion detection system. The algorithm is based on the fact that normal data stream rarely matches any virus signature and different packets need to check different keys. The algorithm does not need preprocessing and can check multiple characters in parallel. Experimental results show that our implementation is faster than original NFA/DFA based algorithms to deal with the same real packet traces while consuming an order of magnitude less memory.

Mean–variance relationship of the number of flows in traffic aggregation and its application to traffic management

We consider the mean–variance relationship of the number of flows in traffic aggregation, where flows are divided into several groups randomly, based on a predefined flow aggregation index, such as source IP address. We first derive a quadratic relationship between the mean and the variance of the number of flows belonging to a randomly chosen traffic aggregation group. Note here that the result is applicable to sampled flows obtained through packet sampling. We then show that our analytically derived mean–variance relationship fits well those in actual packet trace data sets. Next, we present two applications of the mean–variance relationship to traffic management. One is an application to detecting network anomalies through monitoring a time series of traffic. Using the mean–variance relationship, we determine the traffic aggregation level in traffic monitoring so that it meets two predefined requirements on false positive and false negative ratios simultaneously. The other is an application to load balancing among network equipments that require per-flow management. We utilize the mean–variance relationship for estimating the processing capability required in each network equipment.

Intrusion Detection System for High Volume and High Velocity Packet Streams: A Clustering Approach

 Abstract—The success of any Intrusion Detection System lies in its ability to quickly adapt to new threats in near real time and further prevent new attacks. This implies extremely efficient machine learning algorithms in the backend, which in turn may use clustering algorithms capable of distinguishing between normal and anomalous network traffic. This work is a first step towards proposing such an IDS, which is built on clustering-based machine learning. The authors evaluate different clustering algorithms using a network packet trace and provide results, which help in evaluating these algorithms. The work-in-progress section of the paper visualizes the IDS which can be used in an environment where the traffic volumes are very high, enterprise boundaries are blurred, and the likelihood of malicious attacks is extremely high.

Node-based Sampling P2P Bot Detection

The concept of using node-based sampling for the treatment of packet capture mechanism based on Libpcap of network-based detecting Peer-to-Peer botnet process was tested, and its effect on the time window of feature extracting and sampling time interval was explored. Node-based sampling treatment resulted in significant increase in the detection performance due to node profile of the novel behaviors to the detected computer in Peer-to-Peer bot detection, and the degradation of false positive. At relatively right time window (e.g., about 180s), precision was completely maximized, while the false positive decreased by 10% to 15%. The detection rate can be significantly increased due to the false positive degradation. A new performance index called Comprehensive Evaluation Index is proposed for more clearly represent the effectiveness. Sampling can reduce morn than 60% input raw packet traces and achieve a high detection rate (about 99%) and a low false positive rates (0-2%). DOI: http://dx.doi.org/10.11591/telkomnika.v10i5.1272

On the Network Characteristics of the Google’s Suggest Service

This paper investigates application- and transport-level characteristics of Google's interactive Suggest service by analyzing a passively captured packet trace from a campus network. In particular, we study the number of HTTP GET requests involved in user search queries, the inter-request times, the number of HTTP GET requests per TCP connection, the number of keyword suggestions that Google's Suggest service to users, and how often users utilize these suggestions in enhancing their search queries. Our findings indicate, for example, that nearly 40% of Google search queries involve five or more HTTP GET requests. For 36% of these requests, Google returns no suggestions, and 57% of the time users do not utilize returned suggestions. Furthermore, we find that some HTTP characteristics such as inter-request generation time for interactive search application are different from that of traditional Web applications. These results confirm the findings of other studies that examined interactive applications and reported that such applications are more aggressive than traditional Web applications.

Differential piracy

In all seriousness, Differential Privacy is a new technique and set of tools for managing responses to statistical queries over secured data, in such a way that the user cannot reconstruct more precise identification of principles in the dataset beyond a formally well-specified bound. This means that personally sensitive data such as Internet packet traces or social network measurements can be shared between researchers without invading personal privacy, and that assurances can be made with accuracy. With less seriousness, I would like to talk about Differential Piracy , but not without purpose. For sure, while there are legitimate reasons for upstanding citizens to live without fear of eternal surveillance, there is also a segment of society that gets away with things they shouldn't, under a cloak. Perhaps that is the (modest) price we have to pay for a modicum less paranoia in this brave new world. So, there has been a lot of work recently on Piracy Preserving Queries and Differential Piracy. These two related technologies exploit new ideas in statistical security. Rather than security through obscurity, the idea is to offer privacy through lack of differentiation (no, not inability to perform basic calculus, more the inability to distinguish between large numbers of very similar things).

A Markov-Based Packet Dropout Model for UAV Wireless Communications

In this paper, we study the problem of modeling packet dropout for unmanned aerial vehicle (UAV) wireless communications. A Markov model is proposed, which incorporates the effects of Ricean fading. Unlike the classic Markov channel models, the proposed model is a two-state hidden Markov model with each state being associated with a time-varying packet error rate. The model is able to capture the non-stationary packet dropout characteristics of wireless channels. Intuitively, we use the time-varying packet error rate associated with the channels to describe the non-stationary nature of the packet dropouts, and the two-state Markov model to capture the correlation of the packet dropouts. A closed-form solution is provided for estimating the model parameters from network packet traces. Computer simulations and analysis are carried out to demonstrate the performance and effectiveness of the proposed model.

Automated Inference System for End-To-End Diagnosis of Network Performance Issues in Client-Terminal Devices

Traditional network diagnosis methods of Client-Terminal Device (CTD) problems tend to be laborintensive, time consuming, and contribute to increased customer dissatisfaction. In this paper, we propose an automated solution for rapidly diagnose the root causes of network performance issues in CTD. Based on a new intelligent inference technique, we create the Intelligent Automated Client Diagnostic (IACD) system, which only relies on collection of Transmission Control Protocol (TCP) packet traces. Using soft-margin Support Vector Machine (SVM) classifiers, the system (i) distinguishes link problems from client problems and (ii) identifies characteristics unique to the specific fault to report the root cause. The modular design of the system enables support for new access link and fault types. Experimental evaluation demonstrated the capability of the IACD system to distinguish between faulty and healthy links and to diagnose the client faults with 98% accuracy. The system can perform fault diagnosis independent of the user's specific TCP implementation, enabling diagnosis of diverse range of client devices

A Qualitative Risk Assessment Framework for Sharing Computer Network Data

The availability of computer network data is critically important for network operations, the development of next-generation systems, and creation of the evidence-based policies that drive them. Collection and sharing of this network data, which may range from detailed packet traces to aggregated security alerts, is effectively the only way to concretely understand the complex structure and function of real-world networks. The information gleaned from such data sharing efforts is key to enabling innovation and resolving formidable issues in electronic crime forensics, infrastructure security, Internet governance, and intellectual property protection. The data provider's decision to share network data is ultimately anchored in trust. The nature and extent of that trust is a confluence of the capacity to satisfy relevant legal requirements, the value placed on potential benefits, and confidence that the data recipient will not increase the provider's risk of disclosing sensitive information. What is most notable about the current state of practice is that most would-be data providers have a relatively weak understanding of these individual issues and their interplay, particularly in the context of computer network data. Risk is fueled by uncertainty about the application and interpretation of legal restrictions and obligations (e.g., regulations and privacy laws) related to network data disclosure, and exacerbated by unfamiliarity with disclosure control methods. At present, most efforts related to network data sharing focus on defining common data formats and exchange procedures for information interoperability. However, little work has been done to address the need for generalizable and scalable guidance that helps data providers understand and reason about these data sharing issues, thereby enabling risk-sensitive data disclosures that consider both legal constraints and utility needs. We propose a reference risk assessment framework comprised of three phases that are designed to be generalizable across a wide range of data sharing scenarios. The first phase establishes a disclosure control continuum along two primary axes: the intended utility objectives and the disclosure restrictions imposed by legal, contractual, and ethical concerns. In the second phase, we describe operational, technical, and policy-oriented disclosure control methods, along with how they may be applied to adjust the balance between utility and risk mitigation. Finally, the third phase assesses the chosen controls with respect to the stated utility objectives and disclosure restrictions in the context of a qualitatively defined threat model. Significantly, our approach is unique among similar data sharing efforts (e.g., health data) because there is no generally accepted quantitative approach for assessing the risks associated with network data sharing, nor a practicable framework for addressing the growing number of related threats. Instead, we focus on educating data providers toward more effective, balanced, and pragmatic decisions that build a level trust and understanding necessary for productive network data sharing.

Low-storage capture and loss recovery selective replay of real flows

Capturing and replaying real flows are important for testing network security products. However, capturing real flows demands a high storage cost and runs a risk of capture loss, which makes the replay inaccurate. Replaying real flows should be accurate and stateful to adapt to the reaction of the device under test. It should also efficiently reproduce a defect and help developers identify the flows triggering defects. Therefore, this work first presents the (N, M, P) capture scheme which begins with, for each connection, capturing at most N bytes of application payload and then at most M bytes of application payload for at most each of the subsequent P packets in the same connection. This scheme reduces 87 percent of storage cost while retaining 99.74 percent of original events. This work develops a tool named SocketReplay with the mechanisms of loss recovery, stateful replay, and selective replay. Loss recovery tracks TCP sequence numbers to identify capture loss and recovers incomplete flows with dummy data. Stateful replay maintains the states in the TCP/IP stack to replay real flows. Selective replay incrementally selects flows to replay. The results show that SocketReplay can accurately and efficiently reproduce product events and significantly decrease the volume of replayed packet traces.

Libtrace

This paper introduces libtrace, an open-source software library for reading and writing network packet traces. Libtrace offers performance and usability enhancements compared to other libraries that are currently used. We describe the main features of libtrace and demonstrate how the libtrace programming API enables users to easily develop portable trace analysis tools without needing to consider the details of the capture format, file compression or intermediate protocol headers. We compare the performance of libtrace against other trace processing libraries to show that libtrace offers the best compromise between development effort and program run time. As a result, we conclude that libtrace is a valuable contribution to the passive measurement community that will aid the development of better and more reliable trace analysis and network monitoring tools.

IP Traceback for Flooding Attacks on Internet Threat Monitors (ITM ) Using Honeypots

The Internet Threat Monitoring (ITM) is an efficient monitoring system used globally to measure, detect, characterize and track threats such as denial of service (DoS) and distributed Denial of Service (DDoS) attacks and worms. . To block the monitoring system in the internet the attackers are targeted the ITM system. In this paper we address the flooding attack of DDoS against ITM monitors to exhaust the network resources, such as bandwidth, computing power, or operating system data structures by sending the malicious traffic. We propose an information-theoretic frame work that models the flooding attacks using Botnet on ITM. One possible way to counter DDoS attacks is to trace the attack sources and punish the perpetrators. we propose a novel traceback method for DDoS using Honeypots. IP tracing through honeypot is a single packet tracing method and is more efficient than commonly used packet marking techniques.

PcapIndex

Long-term historical analysis of captured network traffic is a topic of great interest in network monitoring and network security. A critical requirement is the support for fast discovery of packets that satisfy certain criteria within large-scale packet repositories. This work presents the first indexing scheme for network packet traces based on compressed bitmap indexing principles. Our approach supports very fast insertion rates and results in compact index sizes. The proposed indexing methodology builds upon libpcap, the de-facto reference library for accessing packet-trace repositories. Our solution is therefore backward compatible with any solution that uses the original library. We experience impressive speedups on packet-trace search operations: our experiments suggest that the index-enabled libpcap may reduce the packet retrieval time by more than 1100 times.

LIVE STREAMING USING PEER DIVISION MULTIPLEXING

A Number of commercial peer-to-peer (P2P) systems for live streaming have been introduced in recent years. The behaviour of the popular systems has been extensively studied in several measurement papers. However, these studies have to rely on a “black-box” approach, where packet traces are collected from a single or a limited number of measurement points, to infer various properties of the traffic on the control and data planes. Although, such studies are useful to compared different systems from the end user’s perspective. It is difficult to intuitively understand the observed properties without fully reverseengineering the underlying systems. In this paper, we describe the network architecture of Zattoo, one of the largest production, live streaming providers, in Europe, at the time of writing, and present a large-scale measurement study of zattoo, using data collected by the provider. To highlight we found that even, when the zattoo system was heavily loaded with as high as 20000 concurrent users on a single overlay, the median channel join delay remained less than 2-5 s, and that, for a majority of users, the streamed signal lags over-the-air broadcast signal by more than 3 s.

Knowledge-independent traffic monitoring: Unsupervised detection of network attacks

The philosophy of traffic monitoring for detection of network attacks is based on an acquired knowledge perspective: current techniques detect either the well-known attacks on which they are programmed to alert, or those anomalous events that deviate from a known normal operation profile or behavior. In this article we discuss the limitations of current knowledge-based strategy to detect network attacks in an increasingly complex and ever evolving Internet. In a diametrically opposite perspective, we place the emphasis on the development of unsupervised detection methods, capable of detecting network attacks in a changing environment without any previous knowledge of either the characteristics of the attack or the baseline traffic behavior. Based on the observation that a large fraction of network attacks are contained in a small fraction of traffic flows, we demonstrate how to combine simple clustering techniques to accurately identify and characterize malicious flows. To show the feasibility of such a knowledge-independent approach, we develop a robust multiclustering-based detection algorithm, and evaluate its ability to detect and characterize network attacks without any previous knowledge, using packet traces from two real operational networks.

Combining Single and Packet-Ray Tracing for Arbitrary Ray Distributions on the Intel MIC Architecture

Wide-SIMD hardware is power and area efficient, but it is challenging to efficiently map ray tracing algorithms to such hardware especially when the rays are incoherent. The two most commonly used schemes are either packet tracing, or relying on a separate traversal stack for each SIMD lane. Both work great for coherent rays, but suffer when rays are incoherent: The former experiences a dramatic loss of SIMD utilization once rays diverge; the latter requires a large local storage, and generates multiple incoherent streams of memory accesses that present challenges for the memory system. In this paper, we introduce a single-ray tracing scheme for incoherent rays that uses just one traversal stack on 16-wide SIMD hardware. It uses a bounding-volume hierarchy with a branching factor of four as the acceleration structure, exploits four-wide SIMD in each box and primitive intersection test, and uses 16-wide SIMD by always performing four such node or primitive tests in parallel. We then extend this scheme to a hybrid tracing scheme that automatically adapts to varying ray coherence by starting out with a 16-wide packet scheme and switching to the new single-ray scheme as soon as rays diverge. We show that on the Intel Many Integrated Core architecture this hybrid scheme consistently, and over a wide range of scenes and ray distributions, outperforms both packet and single-ray tracing.

An Adaptive Approach to Improve the Accuracy of Packet Pre-Filtering

The current day networks are under deliberate, continuous and premeditated attacks such as Hacker attacks, DoS attacks, IP Address Spoofing, Phishing, Sniffer attacks etc. The Network Intrusion Detection Systems (NIDS) proved to be reliable in parrying most of the issues and challenges faced by the corporate network security systems. But, the NID systems fall short in providing a completely fool-proof network security environment. False negatives and false positives proved to be considerable bottle necks in securing the networks from the attacks. This paper deals with the introduction of a software approach for the packet pre-filtering to ease security threats and the introduction of Network Behavior Analysis to enhance the security of the network. The Network Behavior Analysis helps the system to ease the burdens to the network and security of the network by the false positives. The NIDS compares all the incoming packets with the pre-defined rules or signatures to find suspicious patterns. The pre-filtering approach used in this paper is a result of the observation that very rarely an incoming packet matches the signatures or the IDS rules. During the pre-filtering step, a small portion of the packet is compared against the predefined signatures for any suspicious patterns and the initial pre-filtering match is considered for a full match. For time efficiency, this strategy is compared to more optimistic schemes that allow reassignment of flows between threads, and evaluated using several network packet traces.

A fuzzy pattern-based filtering algorithm for botnet detection

Botnet has become a popular technique for deploying Internet crimes. Although signature-based bot detection techniques are accurate, they could be useless when bot variants are encountered. Therefore, behavior-based detection techniques become attractive due to their ability to detect bot variants and even unknown bots. In this paper, we propose a behavior-based botnet detection system based on fuzzy pattern recognition techniques. We intend to identify bot-relevant domain names and IP addresses by inspecting network traces. If domain names and IP addresses used by botnets can be identified, the information can be further used to prevent protected hosts from becoming one member of a botnet. To work with fuzzy pattern recognition techniques, we design several membership functions based on frequently observed bots’ behavior including: (1) generate failed DNS queries; (2) have similar DNS query intervals; (3) generate failed network connections; and (4) have similar payload sizes for network connections. Membership functions can be easily altered, removed, or added to enhance the capability of the proposed system. In addition, to improve the overall system performance, we develop a traffic reduction algorithm to reduce the amount of network traffic required to be inspected by the proposed system. Performance evaluation results based on real traces show that the proposed system can reduce more than 70% input raw packet traces and achieve a high detection rate (about 95%) and a low false positive rates (0–3.08%). Furthermore, the proposed FPRF algorithm is resource-efficient and can identify inactive botnets to indicate potential vulnerable hosts.

Spectral Models for Bitrate Measurement from Packet Sampled Traffic

In network measurement systems, packet sampling techniques are usually adopted to reduce the overall amount of data to collect and process. Being based on a subset of packets, they introduce estimation errors that have to be properly counteracted by using a fine tuning of the sampling strategy and sophisticated inversion methods. This problem has been deeply investigated in the literature with particular attention to the statistical properties of packet sampling and to the recovery of the original network measurements. Herein, we propose a novel approach to predict the energy of the sampling error in the real time estimation of traffic bitrate, based on spectral analysis in the frequency domain. We start by demonstrating that the error introduced by packet sampling can be modeled as an aliasing effect in the frequency domain. Then, we derive closed-form expressions for the Signal-to-Noise Ratio (SNR) to predict the distortion of traffic bitrate estimates over time. The accuracy of the proposed SNR metric is validated by means of real packet traces. Furthermore, a comparison with respect to an analogous SNR expression derived using classic stochastic tools is proposed, showing that the frequency domain approach grants for a higher accuracy when traffic rate measurements are carried out at fine time granularity.

Measurement Study of IPv6 Users on Private BT

The Internet continually evolves in scope and complexity, with the wide deployment of more capacity backbone links and IPv6, the emergence of streaming applications, and even the rapid changing nature of the same applications. These changes lead us to question how is the evolution trend as well as present situation of IPv6 users on the most popular Internet applications (such as Bittorrent). To tackle this issue, we conducted this measurement study on a private BitTorrent System, which is deployed over a campus network serving more than 25,000 users with three external links connecting to Telecom, CERNET, and CERNET2 (IPv6). With packet traces collected from external links and the 10-month log files from the private tracker, we first provide amount and components of IPv6 traffic, and compared with those of IPv4 traffic, then present in depth measurement and analysis from users group evolution, each group’s status, and level of activity.