Kernel Tracing Research Articles

AI Cyber Defense and eBPF

This paper will delve into the integration of Artificial Intelligence (AI) and extended Berkeley Packet Filter (eBPF) technology to enhance cyber defense capability. The most important aspect of AI is detection of threats where it employs sophisticated algorithms in the analysis of large data sets to identify any form of pattern which signals a threat in cyberspace. It enhances behavioral analysis in the monitoring of user and system behavior to identify suspicious activities. AI also helps to instantly react against the threats by automatically taking actions like isolating an infected machine or blocking the suspicious network traffic in order to minimize the response time and consequent damage. AI can predict the attacks based on historical data and current trends and design proactive defense strategies. On the other hand. eBPF technology complements AI by providing programmable kernel tracing, real-time monitoring, low overhead, and security enhancements. Attaching eBPF programs to kernel hooks provides insights into network traffic, system events, and application behavior, intrusion detection, performance monitoring, and troubleshooting. Its implementation is still optimal with less performance impact on the system. The system retains its robust security stance due to the deployment of eBPF, enforcement of kernel-level security policies, and detection of malicious activities. This synergy between AI and eBPF would mean smarter cybersecurity solutions that can change adeptly with each emerging threat and help raise defenses for the organization.

Combining Distributed and Kernel Tracing for Performance Analysis of Cloud Applications

Distributed tracing allows tracking user requests that span across multiple services and machines in a distributed application. However, typical cloud applications rely on abstraction layers that can hide the root cause of latency happening between processes or in the kernel. Because of its focus on high-level events, existing methodologies in applying distributed tracing can be limited when trying to detect complex contentions and relate them back to the originating requests. Cross-level analyses that include kernel-level events are necessary to debug problems as prevalent as mutex or disk contention, however cross-level analysis and associating events in the kernel and distributed tracing data is complex and can add a lot of overhead. This paper describes a new solution for combining distributed tracing with low-level software tracing in order to find the latency root cause better. We explain how we achieve a hybrid trace collection to capture and synchronize both kernel and distributed request events. Then, we present our design and implementation for a critical path analysis. We show that our analysis describes precisely how each request spends its time and what stands in its critical path while limiting overhead.

Performance analysis of distributed storage clusters based on kernel and userspace traces

SummaryDistributed storage systems are commonly used in modern computing. They are highly scalable and offer data replication and fault tolerance. The complexity of those systems makes them difficult to debug using traditional tools. The existing tools are able to evaluate the overall performance of such systems but they do not provide enough information to find the root cause of performance issues. In this article, we propose a tracing‐based performance analysis framework for storage clusters. We use a tracing strategy that reduces the tracing overhead in production systems. The traces collected from the different storage nodes are correlated and used to generate a data model that represents the cluster. Userspace tracing is used to gather data from the storage daemons, while Kernel tracing is used to provide detailed information about operating system internals such as disk queues, network queues and process scheduling. Efficient data structures are used to store the model and to generate metrics and graphical views. Our tool is used in different real world scenarios and is able to investigate interesting performance problems including I/O latencies, data replication and storage nodes failures.

Multilevel analysis of the java virtual machine based on kernel and userspace traces

Performance analysis of Java applications requires a deep understanding of the Java virtual machine and the system on which it is running. An unexpected latency can be caused by a bug in the source code, a misconfiguration or an external factor like CPU or disk contention. Existing tools have difficulties finding the root cause of some latencies because they do not efficiently collect performance data from the different layers of the system. In this paper, we propose a multilevel analysis framework that uses Kernel and userspace tracing to help developers understand and evaluate the performance of their applications. Kernel tracing is used to gather information about thread scheduling, system calls, I/O operations, etc. and userspace tracing is used to monitor the internal components of the JVM such as the garbage collectors and the JIT compilers. By bridging the gap between kernel and userspace traces, our tool provides full visibility to developers and helps them diagnose difficult performance issues. We show the usefulness of our approach by using it to detect problems in different Java applications.

Critical Path Analysis through Hierarchical Distributed Virtualized Environments Using Host Kernel Tracing

Critical Path Analysis through Hierarchical Distributed Virtualized Environments Using Host Kernel Tracing

Event-based application modeling for analysis of asymmetric multicore-based mobile systems

Abstract In this paper, we suggest a generalized application model that can be utilized to support system-level execution behavior analysis of up-to-date and future mobile systems, and propose a new method (kernel event-based application modeling; KAM.2) that extracts the Android application models that are independent of processor configuration and system-management policies. To completely describe the dynamic behavior of a target application, KAM.2 analyzes the kernel events that are obtained when it is running on the system, instead of statically analyzing its source code. This analysis enables extraction of models of real-world applications, source code of which is generally not available. We validated the effectiveness of KAM.2 on a commercial smartphone with real-world applications including user-interactive ones. The runtime overhead incurred by kernel tracing was 0.29%. In terms of core utilization, KAM.2 exhibited 2.58% of mean absolute errors on average and 0.93 of Pearson correlation coefficients on average. These results demonstrate that KAM.2 extracts reliable and realistic Android application models. Also, it was found that the application model extracted using KAM.2 is independent of processor configuration and system-management policies. Therefore, KAM.2 enables practical and flexible analysis of the dynamic behavior of recent and future mobile systems.

Linux page fault analysis in android systems

In modern smartphones, system performances are tightly related to a variety of underlying subsystems. In particular, internal storage, along the years, has become crucial because it is extensively used to access content relevant to the system and, finally, to the end user. To understand its role in a commercial Android smartphone and to evaluate its effects on the User experience, within the context of a real usage, we analyzed Linux page fault handling, a critical mechanism that puts pressure on storage devices and may cause system inefficiencies. A kernel tracing technique has been conceived for real-time measurement of Android applications and services on commercial smartphones. The experimental results presented in this work are derived from the use of this kernel tracing on a 64-bit Android smartphone, equipped with a ufs storage subsystem. The main subject of the study is major page fault handling, a kernel mechanism behind many end-user actions recognized at industry level as source of possible performance deterioration in a smartphone. The analysis shows that major page fault handling is dominated by read accesses to ufs (between 30% and 40% of the total time), and that the related storage traffic is significantly affected by the ReadAhead mechanism, which is not always efficient.

Recovering disk storage metrics from low‐level trace events

SummaryBlock devices such as magnetic disks are nonvolatile data storage devices that transfer data in fixed‐size chunks. They are the main nonvolatile memory that holds the file system, and they are also used in virtual memory mechanisms such swapping and page fault handling. Investigating storage performance issues requires a full insight into the operating system internals. Kernel tracing offers an efficient mechanism to gather information about the storage subsystem at runtime. Still, the tracing output is often huge and difficult to analyze manually.In this paper, we introduce a framework to compute meaningful storage performance metrics from low‐level trace events generated by LTTng. A stateful approach is used to model the state of the storage subsystem. Efficient data structures and algorithms are proposed to offer a reasonable response time, allowing the user to navigate throughout the trace and to retrieve metrics from any time range. The framework includes a visualization system that provides different graphical views that represent the collected information in a convenient way. These views are synchronized together, forming a comprehensive perspective that makes storage performance investigation a much more comfortable task. Different use cases are presented to show the usefulness of the framework in real‐world applications.

Optimization of System’s Performance with Kernel Tracing by Cohort Intelligence

Optimization of System’s Performance with Kernel Tracing by Cohort Intelligence

Fine-grained multilayer virtualized systems analysis

With the consolidation of computer services in large cloud-based data centers, almost all applications and even application development execute in virtualized systems (VS's), sometimes nested. Whether it is inside a container, a virtual machine (VM) running on a physical host, or in a nested virtual machine, every process eventually runs on a physical CPU. Consequently, multiple virtualized systems might unknowingly compete with each other for physical resources. In this paper we study the interactions between all the VS's running on a physical machine. We introduce an analysis based on kernel tracing that erases the bounds between VS's and their host, to display a multilayer system as a single layer. As a result, it becomes possible to know exactly which process is currently running on a physical CPU, even if it is launched inside multiple layers of containers, themselves enclosed into two layers of VMs. To use this analysis, we developed in Trace Compass a view that displays a time line for each host CPU, showing across time which process is running. Moreover, the full hierarchy of the VS's is retrieved from the analysis and is displayed in the view. By using a system of dynamic and permanent filters, we added the possibility to highlight in this view either traced VMs, virtual CPUs, specific processes and containers. This last feature, combined with our view, allows to thoroughly apprehend the execution flow on the physical host, although it may involve multiple nested virtualized systems.

Enhanced Userspace and In-Kernel Trace Filtering for Production Systems

Trace tools like LTTng have a very low impact on the traced software as compared with traditional debuggers. However, for long runs, in resource constrained and high throughput environments, such as embedded network switching nodes and production servers, the collective tracing impact on the target software adds up considerably. The overhead is not just in terms of execution time but also in terms of the huge amount of data to be stored, processed and analyzed offline. This paper presents a novel way of dealing with such huge trace data generation by introducing a Just-In-Time (JIT) filter based tracing system, for sieving through the flood of high frequency events, and recording only those that are relevant, when a specific condition is met. With a tiny filtering cost, the user can filter out most events and focus only on the events of interest. We show that in certain scenarios, the JIT compiled filters prove to be three times more effective than similar interpreted filters. We also show that with the increasing number of filter predicates and context variables, the benefits of JIT compilation increase with some JIT compiled filters being even three times faster than their interpreted counterparts. We further present a new architecture, using our filtering system, which can enable co-operative tracing between kernel and process tracing VMs (virtual machines) that share data efficiently. We demonstrate its use through a tracing scenario where the user can dynamically specify syscall latency through the userspace tracing VM whose effect is reflected in tracing decisions made by the kernel tracing VM. We compare the data access performance on our shared memory system and show an almost 100 times improvement over traditional data sharing for co-operative tracing.

Problem Detection in Real-Time Systems by Trace Analysis

This paper focuses on the analysis of execution traces for real-time systems. Kernel tracing can provide useful information, without having to instrument the applications studied. However, the generated traces are often very large. The challenge is to retrieve only relevant data in order to find quickly complex or erratic real-time problems. We propose a new approach to help finding those problems. First, we provide a way to define the execution model of real-time tasks with the optional suggestions of a pattern discovery algorithm. Then, we show the resulting real-time jobs in a Comparison View, to highlight those that are problematic. Once some jobs that present irregularities are selected, different analyses are executed on the corresponding trace segments instead of the whole trace. This allows saving huge amount of time and execute more complex analyses. Our main contribution is to combine the critical path analysis with the scheduling information to detect scheduling problems. The efficiency of the proposed method is demonstrated with two test cases, where problems that were difficult to identify were found in a few minutes.

Linux Low-Latency Tracing for Multicore Hard Real-Time Systems

Real-time systems have always been difficult to monitor and debug because of the timing constraints which rule out any tool significantly impacting the system latency and performance. Tracing is often the most reliable tool available for studying real-time systems. The real-time behavior of Linux systems has improved recently and it is possible to have latencies in the low microsecond range. Therefore, tracers must ensure that their overhead is within that range and predictable and scales well to multiple cores. The LTTng 2.0 tools have been optimized for multicore performance, scalability, and flexibility. We used and extended the real-time verification tool rteval to study the impact of LTTng on the maximum latency on hard real-time applications. We introduced a new real-time analysis tool to establish the baseline of real-time system performance and then to measure the impact added by tracing the kernel and userspace (UST) with LTTng. We then identified latency problems and accordingly modified LTTng-UST and the procedure to isolate the shielded real-time cores from the RCU interprocess synchronization routines. This work resulted in extended tools to measure the real-time properties of multicore Linux systems, a characterization of the impact of LTTng kernel and UST tracing tools, and improvements to LTTng.

Validity of the Rytov Approximation in the Form of Finite-Frequency Sensitivity Kernels

The first-order (or linear) Rytov or Born approximation is the foundation for formulation of wave-equation tomography and waveform inversion, so the validity of the Rytov/Born approximation can substantially affect the applicability of these theories. However, discussions and research reported in literature on this topic are insufficient or limited. In this paper we introduce five variables in scattering theory to help us discuss conditions under which the Rytov approximation, in the form of the finite frequency sensitivity kernels (RFFSK), the basis of waveform inversion and tomography, is valid. The five variables are propagation length L, heterogeneity scale a, wavenumber k, anisotropy ratio ξ, and perturbation strength ɛ. Combined with theoretical analysis and numerical experiments, we conclude that varying the conditions used to establish the Rytov approximation can lead to uninterpretable or undesired results. This conclusion has two consequences. First, one cannot rigorously apply the linear Rytov approximation to all theoretical or practical cases without discussing its validity. Second, the nonlinear Rytov approximation is essential if the linear Rytov approximation is not valid. Different from previous literature, only phase (or travel time) terms for the whole wavefield are discussed. The time shifts of two specific events between the background and observed wavefields measured by cross-correlation will serve as a reference for evaluation of whether the time shifts predicted by the FFSKs are reasonably acceptable. Significantly, the reference “cross-correlation” should be regarded as reliable only if the condition “two specific similar signals” is satisfied. We cannot expect it to provide a reasonable result if this condition is not met. This paper reports its reliability and experimental limitations. Using cross-correlation (CC) samples as the X axis and sensitivity kernel (SK) or ray tracing (RT) samples as the Y axis, a chart of cross validation among the three is established. According to experimental analysis for a random and deterministic medium, the RFFSK works well for diffraction and geometric optics enclosed by the boundaries ka > 1 and (Λ 1, Φ < 2π), where Λ is a diffraction variable and Φ is a variable used to measure scattering strength. Both Λ and Φ depend on the five variables mentioned above. However, a large perturbation will limit the applicability of the Rytov approximation and narrow the range in which it is valid, because larger ɛ results in higher-amplitude fluctuations and larger back-scattering. Therefore, large ɛ is important in making SK/RT predictions discrepant from correct CC measurements. We also confirm that the RFFSK theory has wider range of application than ray theory. All the experiments point out the importance, both practical and theoretical, of respecting the conditions used to establish the Rytov approximation.

Fine-grained preemption analysis for latency investigation across virtual machines

This paper studies the preemption between programs running in different virtual machines on the same computer. One of the current monitoring methods consist of updating the average steal time through collaboration with the hypervisor. However, the average is insufficient to diagnose abnormal latencies in time-sensitive applications. Moreover, the added latency is not directly visible from the virtual machine point of view. The main challenge is to recover the cause of preemption of a task running in a virtual machine, whether it is a task on the host computer or in another virtual machine. We propose a new method to study thread preemption crossing virtual machines boundaries using kernel tracing. The host computer and each monitored virtual machine are traced simultaneously. We developed an efficient and portable trace synchronization method, which is required to account for time offset and drift that occur within each virtual machine. We then devised an algorithm to recover the root cause of preemption between threads at every level. The algorithm successfully detected interactions between multiple competing threads in distinct virtual machines on a multi-core machine.

Lockless multi-core high-throughput buffering scheme for kernel tracing

Studying execution of concurrent real-time online systems, to identify far-reaching and hard to reproduce latency and performance problems, requires a mechanism able to cope with voluminous information extracted from execution traces. Furthermore, the workload must not be disturbed by tracing, thereby causing the problematic behavior to become unreproducible. In order to satisfy this low-disturbance constraint, we created the LTTng kernel tracer. It is designed to enable safe and race-free attachment of probes virtually anywhere in the operating system, including sites executed in non-maskable interrupt context. In addition to being reentrant with respect to all kernel execution contexts, LTTng offers good performance and scalability, mainly due to its use of per-CPU data structures, local atomic operations as main buffer synchronization primitive, and RCU (Read-Copy Update) mechanism to control tracing. Given that kernel infrastructure used by the tracer could lead to infinite recursion if traced, and typically requires non-atomic synchronization, this paper proposes an asynchronous mechanism to inform the kernel that a buffer is ready to read. This ensures that tracing sites do not require any kernel primitive, and therefore protects from infinite recursion. This paper presents the core of LTTng's buffering algorithms and measures its performance.

Synchronization for fast and reentrant operating system kernel tracing

AbstractTo effectively trace an operating system, a performance monitoring and debugging infrastructure needs the ability to trace various execution contexts. These contexts range from kernel running as a thread to Non‐Maskable Interrupt (NMI) contexts. Given that any part of the kernel infrastructure used by a kernel tracer could lead to infinite recursion if traced, and because most kernel primitives require synchronization unsuitable for some execution contexts, all interactions of the tracing code with the existing kernel infrastructure must be considered in order to correctly inter‐operate with the existing operating system kernel. This paper presents a new low overhead tracing mechanism and motivates the choice of synchronization sequences suitable for operating system kernel tracing, namely local atomic instructions as main buffer synchronization primitive and the Read–Copy Update (RCU) mechanism to control tracing. It also proposes a wait‐free algorithm extending the time‐base needed by the tracer to 64‐bit on architectures that lack hardware 64‐bit time‐base support. Copyright © 2010 John Wiley &amp; Sons, Ltd.

File system usage in Windows NT 4.0

We have performed a study of the usage of the Windows NT File System through long-term kernel tracing. Our goal was to provide a new data point with respect to the 1985 and 1991 trace-based File System studies, to investigate the usage details of the Windows NT file system architecture, and to study the overall statistical behavior of the usage data.In this paper we report on these issues through a detailed comparison with the older traces, through details on the operational characteristics and through a usage analysis of the file system and cache manager. Next to architectural insights we provide evidence for the pervasive presence of heavy-tail distribution characteristics in all aspect of file system usage. Extreme variances are found in session inter-arrival time, session holding times, read/write frequencies, read/write buffer sizes, etc., which is of importance to system engineering, tuning and benchmarking.