Analysis Of Simulation Output Data Research Articles

On credibility of simulation studies of telecommunication networks

In telecommunication networks, as in many other areas of science and engineering, the proliferation of computers as research tools has resulted in the adoption of computer simulation as the most commonly used paradigm of scientific investigations. This, together with a plethora of existing simulation languages and packages, has created a popular opinion that simulation is mainly an exercise in computer programming. In new computing environments, programming can be minimized, or even fully replaced, by the manipulation of icons (representing prebuilt programming objects containing basic functional blocks of simulated systems) on a computer monitor. One can say that we have witnessed another success of modern science and technology: the emergence of wonderful and powerful tools for exploring and predicting the behavior of such complex stochastic dynamic systems as telecommunication networks. But this enthusiasm is not shared by all researchers in this area. An opinion is spreading that one cannot rely on the majority of the published results on performance evaluation studies of telecommunication networks based on stochastic simulation, since they lack credibility. Indeed, the spread of this phenomenon is so wide that one can speak about a deep crisis of credibility. In this article this claim is supported by the results of a survey of over 2200 publications on telecommunication networks in proceedings of IEEE INFOCOM and such journals as IEEE Transactions on Communications, IEEE/ACM Transactions on Networking, and Performance Evaluation Journal. The discussion focuses on two important necessary conditions of a credible simulation study: use of appropriate pseudo-random generators of independent uniformly distributed numbers, and appropriate analysis of simulation output data. Having considered their perils and pitfalls, we formulate guidelines that, if observed, could help to ensure a basic level of credibility of simulation studies of telecommunication networks.

Automating parallel simulation using parallel time streams

This paper describes a package for parallel steady-state stochastic simulation that was designed to overcome problems caused by long simulation times experienced in our ongoing research in performance evaluation of high-speed and integrated-services communication networks, while maintaining basic statistical rigors of proper analysis of simulation output data. The package, named AKAROA, accepts ordinary (nonparallel) simulation programs, and alll further stages of stochastic simulation should be transparent for users. The package employs a new method of sequential estimation for the Multiple-Replications-in-Parallel scenario. All basic functions, including the transformation of originally nonparallel simulators into ones suitable for parallel execution, control of the precision of estimates, and stopping of parallel simulation processes when the required precision of the overall steady-state estimates is achieved, are automated. The package can be used on multiprocessor systems and/or heterogeneous computer networks, involving an arbitrary number of processors. The design issues, architecture, and implementation of AKAROA, as well as the results of its preliminary performance studies are presented

Quantitative stochastic simulation of telecommunication networks in DESC+ +

DESC+ + (Discrete Event Simulation package using C+ +) has been developed as an object-oriented tool for quantitative simulation studies of communication protocols and architectures of telecommunication networks. The main issue of such performance studies is to secure proper statistical accuracy of the final simulation results. In DESC+ + this problem has been solved by automating analysis of simulation output data, aimed at stopping simulation when the estimates reach the required level of precision. The package consists of various object classes. While some of them are responsible for on-line output data analysis, others allow simulation programs of telecommunication networks to be easily developed by re-using existing (sub)models. We discuss main design and implementational issues of DESC+ +, and illustrate them by an example.

Analyzing Simulation Output Using the Bootstrap Method

There are many situations in which parametric statistical techniques are less than ideal for evaluating a simulated system. For most simulation output, one must rely heavily on the central limit theorein in order to apply parametric statistical techniques. The bootstrap statistic is a nonparametric sample-resample technique that makes no distributional assumptions and may be used for estimation and hypothesis testing. The authors propose the bootstrap as a valuable tool for the analysis of simulation output data since it can be used in situations in which either the distribution is not known or normal approximations are inappropriate. Furthermore, since bootstrapping is itself a simulation technique it is inherently satisfying as a tool for the analysis of simulation output data. Illustrations are presented.

Evaluating alternative system configurations using simulation: A nonparametric approach

The real utility of simulation lies in comparing different alternatives that might represent competing system designs. Conventional statistical techniques are not directly applicable to the analysis of simulation output data in the evaluation of competing alternatives since the usual assumptions of normality and common variance are difficult to justify in simulation experiments. This paper revisits a known nonparametric test whose application has recently become feasible due to considerable increases in computing power:randomization tests assess the significance of the observed value of the test statistic by evaluating different permutations of the data. The procedure only requires invariance of the data under all permutations.

Steady-state simulation of queueing processes: survey of problems and solutions

For years computer-based stochastic simulation has been a commonly used tool in the performance evaluation of various systems. Unfortunately, the results of simulation studies quite often have little credibility, since they are presented without regard to their random nature and the need for proper statistical analysis of simulation output data. This paper discusses the main factors that can affect the accuracy of stochastic simulations designed to give insight into the steady-state behavior of queuing processes. The problems of correctly starting and stopping such simulation experiments to obtain the required statistical accuracy of the results are addressed. In this survey of possible solutions, the emphasis is put on possible applications in the sequential analysis of output data, which adaptively decides about continuing a simulation experiment until the required accuracy of results is reached. A suitable solution for deciding upon the starting point of a steady-state analysis and two techniques for obtaining the final simulation results to a required level of accuracy are presented, together with pseudocode implementations.

Expert post-processor for simulation output analysis

This paper discusses the conceptual framework and capabilities of an expert Post-Processor to aid in the analysis of simulation output data. It is proposed to handle both terminating and steady state systems and for analyzing a single system or for statistically comparing two or more systems. The various modules in it are designed to handle determination of the initial truncation point, sample size or run length, variance reduction techniques and also for performing sensitivity analysis of the system. The framework presented here is still in the conceptual stages and is designed for inclusion in the Intelligent Simulation Generator (ISG), work on which is currently being done. This is designed for use in conjunction with any simulator, irrespective of the language it uses.

Feature Article—Statistical Analysis of Simulation Output Data

We present a state-of-the-art survey of statistical analyses for simulation output data of a single simulated system. The various statistical problems associated with output data analyses such as start-up bias and determination of estimator accuracy are described in detail. We then discuss the best available techniques for addressing these problems, as well as topics for future research. The paper concludes with a discussion of how developments in simulation languages, computer graphics, and computer execution speed may affect the future of output analyses.

A simulation study with a combined network and production systems model of pilot behavior on an ILS-approach

A simulation study was conducted for determining information flow requirements of a pilot-cockpit interface during an ILS-approach. Study steps for modeling, simulating, and analyzing system processes are discussed. Starting with the problem definition which includes relevant flight processes, the network of pilot tasks, and performance measures, the conceptual model of processes is mathematically described by means of algebraic and difference equations as well as production systems. To transform the model into a simulation program, the simulation language SLAM is used. SLAM elements utilized for modeling flight processes and pilot tasks as well as the procedure of model validation are discussed. The analysis of simulation output data, i.e. state trajectories and task timelines, results in the determination of important information requirements useful in early stages of a design project.

Construction and Application of a Combined Network and Production Systems Model of Pilot Behaviour on an ILS-Approach

A simulation study was conducted for determining information flow requirements of a pilot-cockpit interface during an ILS-approach. The steps of the study for modeling, simulating, and analyzing system processes are discussed. Starting with the definition of the problem which includes the relevant flight processes, the network of pilot tasks, and the performance measures, the conceptual model of processes is mathematically described by means of algebraic and difference equations as well as production systems. To transform the model into a simulation program, the simulation language SLAM is used. SLAM elements utilized for modeling flight processes and pilot activities as well as the procedure of model validation are discussed. The analysis of simulation output data results in the determination of important information requirements useful in a design project such as range of variables to be indicated or affected, frequency and sequence of variables used by the pilot, pilot tasks, duration of task variables, and the average utilization rate of cockpit display and control components.