An Active Events Model for Systems Monitoring

Abstract

We present an interaction model enabling data-source probes and action-based gauges to communicate using an intelligent event model known as ActEvents. ActEvents build on conventional event concepts by associating structural and semantic information with raw data, thereby allowing recipients to be able to dynamically understand the content of new kinds of events. Two submodels of ActEvents are proposed: SmartEvents, which are XML-structured events containing references to their syntactic and semantic models, and Gaugents, which are heavier but more flexible intelligent mobile software agents. This model is presented in light of DARPA’s DASADA program, where ActEvents are used in a larger-scale subsystem, called KX, which supports continual validation of distributed, componentbased systems. ActEvents are emitted by probes in this architecture, and propagated to gauges, where “measurements” of the raw data associated with probes are made, thereby continually determining updated target-system properties. ActEvents are also proposed as solutions for a number of other applications, including a distributed collaborative virtual environment (CVE) known as CHIME. Introduction and Motivation DARPA’s DASADA program has focused on standards for distributed systems to ease assembly and maintenance of systems that are composed of components “from anywhere” (e.g., COTS, GOTS, open source, etc.). This program has focused on four areas: architecture description languages to describe the composed system, probes to gather information about the current system configuration and state, gauges to interpret this information, and adaptation engines that can reconfigure the system as necessary. This paper focuses on the interaction between probes and gauges, and proposes a standard for data interchange between them. The control interfaces for both probes and gauges have been developed extensively, and standards have been proposed by others. However, the format and transmission mechanism for data collected from probes is underdeveloped. We examine the problem and suggest possible models and architectures, along with a description of our implementation and experience using it. Probes, Gauges and Events A probe is defined as “an individual sensor attached, either statically or dynamically, to a running program” [1]. Probes emit events that describe some aspect of a program’s execution, either at a specific point in time or over some duration. Probes usually: • are integrated into or wrapped onto the application itself; • communicate with the application via an API; or • look at indirect measures such as operating system or network resource usage. The proposed control interface for probes consists of the following methods: Deploy, Install, Activate (and their inverses), Query-Sensed and Generate-Sensed to enumerate the events that a probe can send, and the Sensed method to publish an event. The newer Focus interface allows additional probes to be activated for detailed examination of a problem. The DASADA standard assumes that probe data will be emitted in the form of Siena events. For the purposes of this paper, we define an event as “a collection of data produced by a system component, and of interest to zero or more other system components.” Note that this description makes no assertions about formatting, routing, or transport. The University of Colorado at Boulder’s Siena event system [2] enables Internet-scale content-based event delivery. Siena models events as an unordered, flat collection of attribute-value pairs. Gauges [3] are defined as “software entities that gather, aggregate compute, analyze, disseminate and/or visualize measurement information about software systems.” Gauges support a simple configuration interface. The proposed gauge standard includes the concept of a “Gauge Reporting Bus,” which is specifically for communicating gauge reports to consumers (who might, e.g., authorize repairs). Consumers supply callbacks to the reporting bus, which are called when an event of interest occurs. Probe-Gauge Interaction Probes use system-specific techniques to extract data from the target system. Gauges use the Gauge Reporting Bus interface to report to higher-level components. While the respective APIs for Probes and Gauges are clearly specified, there is no proposed standard for formatting probe data and sending it to the appropriate gauges. Since one cannot assume that probes and gauges will be located on the same machine, some form of networked interprocess communication (IPC) is necessary. Since the machines may be of heterogeneous type, the format for probe data should be as portable as possible. While we do not address the issue in this paper, we also note that the standard interface for controlling probes, although presumably intended to be an event-based interface, is in fact specified as an RPCstyle function API. The Problem There are three aspects of the probe-gauge relationship that make the problem of connection difficult: the dynamic nature of individual probes, the dynamic topology of the various components, and the heterogeneous nature of the systems involved. Individual probes may be frequently added and removed from the system. Probes may be heterogeneously sourced, with possibly different semantics for similar-looking data; simply labeling the type of data elements within the event, as in traditional attribute/value pairs, is insufficient. Instead, the semantic information required for proper interpretation of the probe data must be associated with the event. Probes and gauges will be activated and deactivated, and may migrate from machine to machine. Some of these components (especially probes) may be running on constrained devices, and requiring every component to maintain a complete network topology is not feasible. Further, since the main tasks for most probes are straightforward, requiring all of them to add the data and logic necessary to manage bidirectional RPC with gauges in a changing environment would increase their complexity considerably. Detailed knowledge of event routing and dispatch should ideally be removed from most probes and gauges. While more advanced systems such as CORBA can help with component discovery, probes will typically have many consumers for a single event, which is not handled efficiently under the CORBA model nor under analogous RPC extensions. The systems involved may be completely heterogeneous with different byte-ordering, operating systems, architectures, etc. Message formatting should be completely architecture independent, and leverage industry standards to the degree possible.