Computer System Evaluation Research Articles

Numerical inversion of multidimensional Laplace transforms by the Laguerre method

Numerical transform inversion can be useful to solve stochastic models arising in the performance evaluation of telecommunications and computer systems. We contribute to this technique in this paper by extending our recently developed variant of the Laguerre method for numerically inverting Laplace transforms to multidimensional Laplace transforms. An important application of multidimensional inversion is to calculate time-dependent performance measures of stochastic systems. Key features of our new algorithm are: (1) an efficient FFT-based extension of our previously developed variant of the Fourierseries method to calculate the coefficients of the multidimensional Laguerre generating function, and (2) systematic methods for scaling to accelerate convergence of infinite series, using Wynn's ϵ-algorithm and exploiting geometric decay rates of Laguerre coefficients. These features greatly speed up the algorithm while controlling errors. We illustrate the effectiveness of our algorithm through numerical examples. For many problems, hundreds of function evaluations can be computed in just a few seconds.

An interpolation approximation for the GI/G/1 queue based on multipoint Padé approximation

The performance evaluation of many complex manufacturing, communication and computer systems has been made possible by modeling them as queueing systems. Many approximations used in queueing theory have been drawn from the behavior of queues in light and heavy traffic conditions. In this paper, we propose a new approximation technique, which combines the light and heavy traffic characteristics. This interpolation approximation is based on the theory of multipoint Pade approximation which is applied at two points: light and heavy traffic. We show how this can be applied for estimating the waiting time moments of the GI/G/1 queue. The light traffic derivatives of any order can be evaluated using the MacLaurin series analysis procedure. The heavy traffic limits of the GI/G/1 queue are well known in the literature. Our technique generalizes the previously developed interpolation approximations and can be used to approximate any order of the waiting time moments. Through numerical examples, we show that the moments of the steady state waiting time can be estimated with extremely high accuracy under all ranges of traffic intensities using low orders of the approximant. We also present a framework for the development of simple analytical approximation formulas.

Mathematics, technology, and trust: formal verification, computer security, and the U.S. military

A distinctive concern in the US military for computer security dates from the emergence of time-sharing systems in the 1960s. This paper traces the subsequent development of the idea of a "security kernel" and of the mathematical modeling of security, focusing in particular on the paradigmatic Bell-LaPadula model. The paper examines the connections between computer security and formal, deductive verification of the properties of computer systems. It goes on to discuss differences between the cultures of communications security and computer security, the bureaucratic turf war over security, and the emergence and impact of the Department of Defense's Trusted Computer System Evaluation Criteria (the so-called Orange Book), which effectively took its final form in 1983. The paper ends by outlining the fragmentation of computer security since the Orange Book was written.

Structured evaluation of computer systems

Evaluating computers and other systems is difficult for a couple of reasons. First, the goal of evaluation is typically ill-defined: customers, sometimes even designers, either don't know or can't specify exactly what result they expect. Often, they don't specify the architectural variants to consider, and often the metrics and workload they expect you to use are ill-defined. Second, they rarely clarify which kind of model and evaluation method best suit the evaluation problem. These problems have consequences. For one thing, the decision-maker may not trust the evaluation. For another, poor planning means the evaluation cannot be reproduced if any of the parameters are changed slightly. Finally, the evaluation documentation is usually inadequate, and so some time after the evaluation you might ask yourself, how did I come to that conclusion? An approach developed at Siemens makes decisions explicit and the process reproducible.

Study on Fuzzy Queueing Theory and Its Application to Performance Evaluation of Computer Systems

Study on Fuzzy Queueing Theory and Its Application to Performance Evaluation of Computer Systems

The UltraSAN modeling environment

Model-based evaluation of computer systems and networks is an increasingly important activity. For modeling to be used effectively, software environments are needed that ease model specification, construction, and solution. Easy to use, graphical methods for model specification that support solution of families of models with differing parameter values are also needed. Since no model solution technique is ideal for all situations, multiple analysis and simulation-based solution techniques should be supported. This paper describes UltraSAN, one such software environment. The design of UltraSAN reflects its two main purposes: to facilitate the evaluation of realistic computer systems and networks, and to provide a test-bed for investigating new modeling techniques. In UltraSAN, models are specified using stochastic activity networks, a stochastic variant of Petri nets, using a graphical X-Window based interface that supports large-scale model specification, construction, and solution. Models may be parameterized to reduce the effort required to solve families of models, and a variety of analysis and simulation-based solution techniques are supported. The package has a modular organization that makes it easy to add new construction and solution techniques as they become available. In addition to describing the features, capabilities, and organization of UltraSAN, the paper illustrates the use of the package in the solution for the unreliability of a fault-tolerant multiprocessor using two solution techniques.

Measuring cache and TLB performance and their effect on benchmark runtimes

In previous research, we have developed and presented a model for measuring machines and analyzing programs, and for accurately predicting the running time of any analyzed program on any measured machine. That work is extended here by: (1) developing a high level program to measure the design and performance of the cache and TLB units; (2) using those measurements, along with published miss ratio data, to improve the accuracy of our runtime predictions; (3) using our analysis tools and measurements to study and compare the design of several machines, with particular reference to their cache and TLB performance. As part of this work, we describe the design and performance of the cache and TLB for ten machines. The work presented, in this paper extends a powerful technique for the evaluation and analysis of both computer systems and their workloads; this methodology is valuable both to computer users and computer system designers.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Rational approximants for some performance analysis problems

Performance index functions with integer parameters are important in the evaluation and analysis of computer systems, communication networks and general distributed and parallel processing systems. These functions are often difficult to evaluate for large integers. On the other hand, most of these performance functions have nice properties such as monotonicity, convexity, analyticity, as well as obtainable asymptotic behavior. These properties make it possible to use rational approximants to extrapolate the performance functions. In this paper we present the rational approximants for several example systems. Our numerical results suggest that this approach could lead to a promising tool for many applications.

Fair distribution of concerns in design and evaluation of fault-tolerant distributed computer systems

In analysing the fault tolerance capabilities of distributed computer system designs, the functionally or physically replaceable components have usually been modelled as one of the two extreme types with respect to their failure symptoms (faulty output behaviour): the fail-silent unit (FSU) model at the simplest end, and the malicious unit (MaU) model, also called the Byzantine unit model, at the other end. The basic weaknesses of these models for use in practical system design and evaluation are pointed out. The FSU model is justifiable for handling most simple parts of a system, but not so for other complex parts. The MaU model is misleading in that it tends to draw attention to events of negligible occurrence probabilities while taking attention away from events of higher occurrence probabilities. It is also pointed out that the state-of-the-art in analytic modelling and evaluation of fault-tolerant distributed computer systems has a vast weakly characterized region in the domain of conceivable component models enclosed by the two extreme models. The main constructive proposition made with respect to advancing the state-of-the-art is to establish scientific procedures for fair distribution of concerns over possible occurrences of anomalous events during system design and validation. A direction for obtaining such a fair modelling procedure relying on extensive probabilistic reasoning is proposed. The principle may have broad applicability, extending much beyond the area of fault-tolerant computing.

Objective evaluation of human and computational stereoscopic visual systems

Several computational models have been proposed to account for the abilities of human observers in solving the stereo correspondence problem. Such models are often evaluated by counting the number of correct matches that the model makes when presented with a test stimulus, but this measure is difficult to compare with conventional psychophysical measures of human stereo performance. This paper develops an absolute comparison of human and model stereo correspondence by using the concept of statistical efficiency, through which the performance of both humans and models is compared against that of an ideal observer. Some important assumptions need to be examined before the efficiencies of human and model can be compared because these assumptions affect the statistical performance of models. One assumption involves the use of an epipolar constraint to help constrain the matching problem but it is unclear what the resolution (or “height”) of an epipolar line should be. Experimental estimates were obtained of the vertical region over which dot position may be varied without disruption of human stereoacuity. These estimates constrained the effective “height” of the zone above and below an epipolar line. Two models of stereopsis were then compared against human performance: the first (due to Pollard, Mayhew & Frisby, Perception, 14, 449–470, 1985) uses local constraints to solve the correspondence problem; the second is a simple area-based correlation model. For nearly all the stimuli tested, both models show strong similarities with human observers. For one particular stimulus, the efficiency of the correlation model is different from that of humans and of the PMF algorithm. This implies that a more complex process than simple correlation may be required to account for human stereo processing and indicates other limitations of correlation models.

Generating non-standard random distributions for discrete event simulation systems

This paper introduces a novel method for generating non-standard random distributions. These distributions are required when the system cannot be modelled accurately using conventional probabilistic distributions. In the proposed method, the desired random distribution is built by splitting the density function into smaller segments which are individually approximated with simple polynomial functions. Then, the inverse transformation method is used to form the final random distribution function. This paper presents also new criteria for expediting the selection of the correct segment. The complexity of the proposed process for segment selection is O(log M), where M is the number of segments. An example of the application of the proposed method to simulation of disk access patterns for performance evaluation of computer systems is provided.

A High Assurance Window System Prototype1

This paper describes the architecture of a prototype multilevel secure windowing system based on the X Window System. The prototype, known as TX, is designed to meet the class B3 architectural requirements of the Trusted Computer System Evaluation Criteria (TCSEC). The architecture and prototype described here demonstrate that high assurance windowing technology is feasible.The TX architecture is based on the encapsulation of untrusted functionality, such as that contained in an ordinary X server, using a relatively small amount of trusted applications code. The untrusted functionality is then polyinstantiated or replicated once for each active sensitivity level. This leads to a combination of high assurance and complex functionality while reducing the evaluation effort to a tractable level. The architecture of TX is described, and its information flow and visible labeling security policies are discussed. The trade-offs that were made to maintain assurance while achieving other software engineering goals are considered. TX is compared with several other trusted windowing systems.

A unified framework for the performability evaluation of fault-tolerant computer systems

The problem of evaluating the performability density and distribution of degradable computer systems is considered. A generalized model of performability is considered, wherein the dynamics of configuration modes are modeled as a nonhomogeneous Markov process, and the performance rate in each configuration mode can be time dependent. The key to the development of a unifying mathematical framework is the introduction of two related performability processes: the forward performability process over the interval (0,t), and the performability-to-go process over the interval (t,T), where T is the mission time. Stochastic differential equations techniques show that the joint density of the forward performability and configuration states satisfies a linear, hyperbolic partial differential equation (PDE) with time-dependent coefficients that runs forward in time, while the performability-to-go process satisfies an adjoint PDE running reverse in time. A numerical method for solving the PDEs is presented and is illustrated with examples. >

Development trends in methods for efficiency evaluation of ES-based computer systems

This paper concerns some methods for efficiency evaluation of IBM-compatible universal ES computers, as being improved during their short life-time. The multi-level structure of computer efficiency is described, and the factors influencing its quantification are pointed out. The measured results are given on the capacity of individual computers with different loads.

System and real performance evaluation of computer

The purpose of this paper is to review some methods for efficiency evaluation of universal computer systems. This paper is continued of [1] and concerns the measurements and analytical modeling for performance evaluation on system and real levels.

An evaluation of HP-UX (UNIX) for database protection using the European ITSEC

This paper describes an evaluation of the effectiveness of two versions of Hewlett Packard's UNIX operating system, HP-UX, to protect a sensitive database set up under the ORACLE Data Base Management System, using Version 1.0 of the European Information Technology Security Evaluation Criteria (ITSEC) as the basis for the evaluation. An outline of these criteria is given, along with mention of the US Trusted Computer System Evaluation Criteria (TCSEC) for comparison purposes. The research project reported in this paper was aimed at determining the level of effort required to perform such an evaluation and also the relevance of the ITSEC to commercial environments where information systems security, incorporating authenticity, integrity and privacy, are of paramount concern. Two successive releases of the HP-UX operating system were found to be markedly different when evaluated from the ITSEC point of view, with the latter version being quite effective in maintaining security parameters. It was also concluded that evaluations are time consuming but can have positive implications for commercial systems developers.

SIQUEUE-PET

Developing models for performance evaluation of computer systems, logistic systems etc. often is a complex task. The effort can be considerably reduced if appropriate software tools are available. In many cases queueing network models are suitable for solving the problem to a sufficient degree of accuracy. SIQUEUE-PET provides an environment for construction, evaluation and result representation of such models. The user is assisted through a graphical interface for model construction as well as for result representation. The availability of a support for object management provides further alleviation in the modelling activities. This contribution gives a brief overview of the main features of SIQUEUE-PET. From the viewpoint of modelling style, the availability of aggregation techniques and the capability of processing hierarchically structured models is to be emphasized. An example is included for illustrative purposes.

Complexity Measures for the Analysis of Specifications of (Reliability Related) Computer Systems

Measurement has become recognized as a useful way to plan, control and evaluate software projects. Especially the early stages of the software development process, such as requirement analysis and design, are vital for the successful implementation of computer systems. High order Petri nets are introduced as a powerful formalism for the specification and analysis of concurrent systems. The paper describes the application of several complexity measures for Petri nets and their use to find overly complex structures. The assessment and evaluation of computer systems for reliability is illustrated with an exemplary computer controlled pay phone that was specified and analyzed with the Specs Petri net tool Problems resulting from software/hardware interaction and different, however interacting, states of the system and their formalization with Petri nets are discussed. This technique supports the early analysis of sufficiently refined specifications that can be used as a system prototype for customer presentations and reliability assessment.

A framework for human factors evaluation

Abstract Successful human factors evaluation of interactive computer systems has tended to rely heavily on the experience of the practitioner, who has had little explicit support on which to draw. This paper concerns support for evaluation in the form of a framework for describing and guiding the general activity. The paper starts with a critique of current approaches to evaluation, and particularly of evaluation within the ‘design for usability’ approach. Following a definition of evaluation, a framework is proposed that attempts to clarify what can be done towards which goals and how it can be done. This highlights and discusses notions of system performance, of assessment statements, and of assessment methods. The paper concludes with a discussion of the implications of the framework for evaluation practice.

Computer-Assisted Adult Medical Diagnosis

Three decades after the conceptual foundation was laid for computer-aided diagnosis, some of its potential has been realized. Systems based on probabilistic reasoning have been developed and applied within limited domains (e.g., acute abdominal pain) and for the general diagnosis of systemic disorders. Less progress has been made in the development and application of diagnostic systems based on "artificial intelligence", reflecting theoretical limits to this application of computers to medicine and the enormity of the task. The presently available probabilistic systems have recently been joined by a new microcomputer-based system, MEDITEL Computer-Assisted Diagnosis, Adult System. The performance of this system was evaluated with both clinical-pathologic conference cases and consecutive admissions with undiagnosed illnesses. The correct diagnosis appeared on the list generated by the system in 80 to 90% of the cases. Experience with this and other systems illustrates current issues in the evaluation of computer systems for aid in diagnosis and of computer-based medical "expert" systems in general. These issues include physician acceptance of these systems and the ethical, legal, and regulatory aspects of computer system application. We conclude that, in appropriately selected cases, the accuracy and efficiency of physician diagnosis can be enhanced with computer assistance, and the risk of overlooking the correct diagnosis can be reduced.