Optimistic Parallel Discrete Event Simulation Research Articles

Performance Analysis of Speculative Parallel Adaptive Local Timestepping for Conservation Laws

Stable simulation of conservation laws, such as those used to model fluid dynamics and plasma physics applications, requires the satisfaction of the so-called Courant-Friedrichs-Lewy condition. By allowing regions of the mesh to advance with different timesteps that locally satisfy this stability constraint, significant work reduction can be attained when compared to a time integration scheme using a single timestep size. However, parallelizing this algorithm presents considerable difficulty. Since the stability condition depends on the state of the system, dependencies become dynamic and potentially non-local. In this article, we present an adaptive local timestepping algorithm using an optimistic (Timewarp-based) parallel discrete event simulation. We introduce waiting heuristics to limit misspeculation and a semi-static load balancing scheme to eliminate load imbalance as parts of the mesh require finer or coarser timesteps. Last, we outline an interface for separating the physics of the specific conservation law from the temporal integration allowing for productive adoption of our proposed algorithm. We present a misspeculation study for three conservation laws, demonstrating both the productivity of the local timestepping API, for which 74% of the lines of code are reused across different conservation laws, and the robustness of the waiting heuristics—at most 1.5% of element updates are rolled back. Our performance studies demonstrate up to a 2.8× speedup versus a baseline unoptimized local timestepping approach, a 4x improvement in per-node throughput compared to an MPI parallelization of synchronous timestepping, and scalability up to 3,072 cores on NERSC’s Cori Haswell partition.

Synchronization of Processes in Parallel Discrete Event Simulation

The method of parallel discrete event simulation (PDES) is one of promising methods for effective use of supercomputer systems. In this paper, the results of the analysis of synchronization models for processor elements are presented when calculations are performed by the PDES method. An analogy between the change in the local time profile of processor elements and the increase of the sample surface during molecular epitaxy is used for the classification of synchronization models. Two important classes of such models are the models of conservative PDES and optimistic PDES. The conservative model belongs to the universality class of the Kardar–Parisi–Zhang (KPZ) equation, and the optimistic model belongs to the universality class of directed percolation (DP) problems. We present the results of the analysis of synchronization models in the case when the graph of message exchange between processor elements belongs to the class of small-world networks.

Optimistic Modeling and Simulation of Complex Hardware Platforms and Embedded Systems on Many-Core HPC Clusters

The large size and complexity of the modern digital hardware impose great challenges to design and validation. Hardware Description Languages (HDLs) and System-Level Description Languages (SLDLs) rely on sequential discrete event semantics. Parallel Discrete Event Simulation (PDES) has recently gained extensive attention for parallelizing these languages due to the ever-increasing complexity of embedded and cyber physical systems. However, PDES application has not yet reached acceptable maturity and pervasiveness for accelerating computer architecture problems. This is due to inherent complexity of hardware components that require using different advanced PDES techniques. In this paper, we look at the main problem from a radically different angle. First, we suppose there is only a single universal discrete event model of computation for simulation purely defined by distributed optimistic PDES, i.e., logical-process-based event scheduling worldview. Second, we construct a new parallel system-level simulation language called OSML for ESL. Third, we propose a unified Cloud-based CAD tool called Troodon to automatically parallelize existing hardware languages atop OSML. To the best of our knowledge, OSML is the first work on optimistic synchronization applied to hardware-specific SLDLs and hardware models at different levels of abstraction in ESL, which contain complex data structures by proposing a hybrid checkpointing scheme.

Transparently Mixing Undo Logs and Software Reversibility for State Recovery in Optimistic PDES

The Time Warp synchronization protocol for Parallel Discrete Event Simulation (PDES) is universally considered a viable solution to exploit the intrinsic simulation model parallelism and to provide model execution speedup. Yet it leads the PDES system to execute events in an order that may generate causal inconsistencies that need to be recovered via rollback , which requires restoration of a previous (consistent) simulation state whenever a causality violation is detected. The rollback operation is so critical for the performance of a Time Warp system that it has been extensively studied in the literature for decades to find approaches suitable to optimize it. The proposed solutions can be roughly classified as based on either checkpointing or reverse computing . In this article, we explore the practical design and implementation of a fully new approach based on the runtime generation of so-called undo code blocks , which are blocks of instructions implementing the reverse memory side effects generated by the forward execution of the events. However, this is not done by recomputing the original values to be restored, as instead it occurs in reverse computing schemes. Hence, the philosophy undo code blocks rely on is similar in spirit to that of undo-logs (as a form of checkpointing). Nevertheless, they are not data logs (as instead checkpoints are); rather, they are logs of instructions. Our proposal is fully transparent, thanks to the reliance on static software instrumentation (targeting the x86 architecture and Linux systems). Also, as we show, it can be combined with classical checkpointing to further improve the runtime behavior of the state recoverability support as a function of the workload. We also present experimental results related to our implementation, which is released as free software and fully integrated into the open source ROOT-Sim package. Experimental data support the viability and effectiveness of our proposal.

Model-based Dynamic Control of Speculative Forays in Parallel Computation

In simulations running in parallel, the processors would have to synchronize with other processors to maintain correct global order of computations. This can be done either by blocking computation until correct order is guaranteed, or by speculatively proceeding with the best guess (based on local information) and later correcting errors if/as necessary. Since the gainful lengths of speculative forays depend on the dynamics of the application software and hardware at runtime, an online control system is necessary to dynamically choose and/or switch between the blocking and speculative strategies. In this paper, we formulate the reversible speculative computing in large-scale parallel computing as a dynamic linear feedback control (optimization) system model and evaluate its performance in terms of time and cost savings as compared to the traditional (forward) computing. We illustrate with an exact analogy in the form of vehicular travel under dynamic, delayed route information. The objective is to assist in making the optimal decision on what computational approach is to be chosen, by predicting the amount of time and cost savings (or losing) under different environments represented by different parameters and probability distribution functions. We consider the cases of Gaussian, exponential and log-normal distribution functions. The control system is intended for incorporating into speculative parallel applications such as optimistic parallel discrete event simulations to decide at runtime when and to what extent speculative execution can be performed gainfully.

Model for the evolution of the time profile in optimistic parallel discrete event simulations

We investigate synchronisation aspects of an optimistic algorithm for parallel discrete event simulations (PDES). We present a model for the time evolution in optimistic PDES. This model evaluates the local virtual time profile of the processing elements. We argue that the evolution of the time profile is reminiscent of the surface profile in the directed percolation problem and in unrestricted surface growth. We present results of the simulation of the model and emphasise predictive features of our approach.

Autonomic State Management for Optimistic Simulation Platforms

We present the design and implementation of an autonomic state manager (ASM) tailored for integration within optimistic parallel discrete event simulation (PDES) environments based on the C programming language and the executable and linkable format (ELF), and developed for execution on ×86_64 architectures. With ASM, the state of any logical process (LP), namely the individual (concurrent) simulation unit being part of the simulation model, is allowed to be scattered on dynamically allocated memory chunks managed via standard API (e.g., malloc/free). Also, the application programmer is not required to provide any serialization/ deserialization module in order to take a checkpoint of the LP state, or to restore it in case a causality error occurs during the optimistic run, or to provide indications on which portions of the state are updated by event processing, so to allow incremental checkpointing. All these tasks are handled by ASM in a fully transparent manner via (A) runtime identification (with chunk-level granularity) of the memory map associated with the LP state, and (B) runtime tracking of the memory updates occurring within chunks belonging to the dynamic memory map. The co-existence of the incremental and non-incremental log/restore modes is achieved via dual versions of the same application code, transparently generated by ASM via compile/link time facilities. Also, the dynamic selection of the best suited log/ restore mode is actuated by ASM on the basis of an innovative modeling/optimization approach which takes into account stability of each operating mode with respect to variations of the model/environmental execution parameters.

Load sharing for optimistic parallel simulations on multi core machines

Parallel Discrete Event Simulation (PDES) is based on the partitioning of the simulation model into distinct Logical Processes (LPs), each one modeling a portion of the entire system, which are allowed to execute simulation events concurrently. This allows exploiting parallel computing architectures to speedup model execution, and to make very large models tractable. In this article we cope with the optimistic approach to PDES, where LPs are allowed to concurrently process their events in a speculative fashion, and rollback/ recovery techniques are used to guarantee state consistency in case of causality violations along the speculative execution path. Particularly, we present an innovative load sharing approach targeted at optimizing resource usage for fruitful simulation work when running an optimistic PDES environment on top of multi-processor/multi-core machines. Beyond providing the load sharing model, we also define a load sharing oriented architectural scheme, based on a symmetric multi-threaded organization of the simulation platform. Finally, we present a real implementation of the load sharing architecture within the open source ROme OpTimistic Simulator (ROOT-Sim) package. Experimental data for an assessment of both viability and effectiveness of our proposal are presented as well.

A Dynamic Partitioning Algorithm Based on Approximate Local Search for Optimistic Parallel Discrete Event Simulation

With the rise of simulation platforms which support efficiently migration,the dynamic partitioning mechanism can significantly improve performance of the PDES.How to estimate and optimize the communication structure becomes an essential research topic for dynamic partitioning.Currently,there are several dynamic partitioning algorithms.Since they consider computation and communication load balancing isolated,these algorithms may suffer under complex application background.This paper formalizes the dynamic partitioning problem statement which combine both computation and communication load balancing,and proposes an algorithm based on approximate local search,which obtains a solution with value no smaller than(1-1/|N|)(1-e)of the optimal solution value in polynomial time complexity.The experiments show that the algorithm has high performance and low overhead.

Multiprogrammed non-blocking checkpoints in support of optimistic simulation on myrinet clusters

CCL (checkpointing and communication library) is a software layer in support of optimistic parallel discrete event simulation (PDES) on myrinet-based COTS clusters. Beyond classical low latency message delivery functionalities, this library implements CPU offloaded, non-blocking (asynchronous) checkpointing functionalities based on data transfer capabilities provided by a programmable DMA engine on board of myrinet network cards. These functionalities are unique since optimistic simulation systems conventionally rely on checkpointing implemented as a synchronous, CPU-based data copy. Releases of CCL up to v2.4 only support monoprogrammed non-blocking checkpoints. This forces re-synchronization between CPU and DMA activities, which is a potential source of overhead, each time a new checkpoint request must be issued at the simulation application level while the last issued one is still being carried out by the DMA engine. In this paper we present a redesigned release of CCL (v3.0) that, exploiting hardware capabilities of more advanced myrinet clusters, supports multiprogrammed non-blocking checkpoints. The multiprogrammed approach allows higher degree of concurrency between checkpointing and other simulation specific operations carried out by the CPU, with benefits on performance. We also report the results of the experimental evaluation of those benefits for the case of a Personal Communication System (PCS) simulation application, selected as a real world test-bed.

Optimistic Simulations of Physical Systems Using Reverse Computation

Efficient computer simulation of complex physical phenomena has long been challenging due to their multiphysics and multiscale nature. In contrast to traditional time-stepped execution methods, the authors describe an approach using optimistic parallel discrete event simulation (PDES) and reverse computation techniques to execute plasma physics codes. They show that reverse computation-based optimistic parallel execution can significantly reduce the execution time of an example plasma simulation without requiring a significant amount of additional memory compared to conservative execution techniques. The authors describe an application-level reverse computation technique that is efficient and suitable for complex scientific simulations.

Cancellation strategy in rollback mechanism

Rollback is a synchronization mechanism in optimistic parallel discrete event simulation (PDES). A basic and popular realization of rollback is based on cancellation of error executed messages. In this paper, several cancellation strategies are introduced, and a new method proposed. Performance comparison is made based on experimental results.

Nonblocking checkpointing for optimistic parallel simulation: description and an implementation

Describes a nonblocking checkpointing mode in support of optimistic parallel discrete event simulation. This mode allows real concurrency in the execution of state saving and other simulation specific operations (e.g, event list update, event execution) with the aim of removing the cost of recording state information from the completion time of the parallel simulation application. We present an implementation of a C library supporting nonblocking checkpointing on a myrinet based cluster, which demonstrates the practical viability of this checkpointing mode on standard off-the-shelf hardware. By the results of an empirical study on classical parameterized synthetic benchmarks, we show that, except for the case of minimal state granularity applications, nonblocking checkpointing allows improvement of the speed of the parallel execution, as compared to commonly adopted, optimized checkpointing methods based on the classical blocking mode. A performance study for the case of a personal communication system (PCS) simulation is additionally reported to point out the benefits from nonblocking checkpointing for a real world application.

Design and Implementation of DynamicLoad Balancing Algorithms for Rollback Reductionin Optimistic PDES

In an optimistic parallel simulation, logical processes (Ips) proceed with their computation without any constraints. However, if the computing requirements of different lps are not balanced or if the processors are not homogeneous, some lps may lag behind in simulation time while others surge forward. In other words, if the simulation clocks of different lps are not progressing at the same rate, cascading rollbacks may occur nullifying the potential benefit of an optimistic parallel discrete event simulation (PDES). Hence it is necessary to balance the computational load on different lps in such a way that their local simulation clocks advance almost at the same rate. In this paper, we propose two algorithms for dynamic load balancing which reduce the number of rollbacks in an optimistic PDES system. Our first algorithm is based on the load transfer mechanism between lps; while the second algorithm, based on the principle of evolutionary strategy, migrates logical processes between several pairs of physical processors. We have implemented both of these algorithms on a cluster of heterogeneous workstations and studied their performance. The experimental results show that the algorithm based on the load transfer is effective when the grain size is greater than 10 milliseconds. The algorithm based on the process migration yields good performance only for grain sizes of 20 milliseconds or larger. In both of these cases the speed up ranges mostly between and 2 using four processors.

A probabilistic event scheduling policy for optimistic parallel discrete event simulation

A probabilistic event scheduling policy for optimistic parallel discrete event simulation

Parameterized Time Warp (PTW): An Integrated Adaptive Solution to Optimistic PDES

Time Warp is an optimistic synchronization protocol used for parallel discrete event simulation. While Time Warp has the potential to reduce the execution time of large simulations, it has been plagued by a variety of problems, namely: 1. Instability due to thrashing effects caused by echoing and cascading rollbacks. 2. Memory bottlenecks due to state saving and excessive optimism. 3. Inefficient scheduling algorithms for scheduling Time Warp processes on each processing node. These problems have inhibited the widespread use of Time Warp as a general purpose synchronization algorithm. The general trend of researchers attempting to solve these problems has been to statically limit the optimism of Time Warp. Unfortunately, these attempts have achieved only limited success. This is because a static set of parameters may perform well for one simulation but not for another. This paper attacks the problem using adaptive mechanisms to control optimism, using an index of performance called useful work. This research presents solutions for the above mentioned problems, by: 1. Stabilizing Time Warp using adaptive bounded time windows. 2. Reducing memory usage and overall execution time by using an adaptive mechanism to vary the checkpoint interval. 3. Scheduling Time Warp processes with the useful work parameter to favor more productive processes. Using this new performance index called Useful Work, several modifications to Time Warp are implemented to stabilize and improve Time Warp. Thus, this new improved Time Warp synchronization mechanism termed Parameterized Time Warp provides an integrated adaptive solution to optimistic Parallel Discrete Event Simulation. Empirical work showing that PTW outperforms an equivalent Time Warp simulation executing under similar partitioning and load conditions is also presented.

PORTS: a parallel, optimistic, real-time simulator

This paper describes issues concerning the design of an optimistic parallel discrete event simulation system that executes in environments that impose real-time constraints on the simulator's execution. Two key problems must be addressed by such a system. First the timing characteristics of the parallel simulator must be sufficiently predictable to allow one to guarantee that real-time deadlines for completing simulation computations will be met. Second, the optimistic computation must be able to interact with its surrounding environment with as little latency as possible, necessitating rapid commitment of I/O operations. To address the first question, we show that optimistic simulators that never send incorrect messages (sometimes called “aggressive-no-risk” simulators) provide sufficient predictability to allow traditional schedulability analysis techniques commonly used in real-time systems to be applied. We show that incremental state saving techniques introduce sufficient unpredictability that they are not well-suited for real-time environments. We observe that the traditional “lowest timestamp first” scheduling policy used in many optimistic parallel simulation systems is an optimal (in the real-time sense) scheduling algorithm when event timestamps and real-time deadlines are the same. Finally, to address the question for rapid commitment of I/O operations, we utilize a continuous GVT computation scheme for shared-memory multiprocessors where a new value of GVT is computed after processing each event in the simulation. These ideas are incorporated in a parallel, optimistic, real-time simulation system called PORTS. Initial performance measurements of the shared-memory based PORTS system executing on a Kendall Square Research multiprocessor are presented. Initial performance results are encouraging, demonstrating that PORTS achieves performance approaching that of a conventional Time Warp system for the benchmark programs that were tested.

A New Process to Processor Assignment Criterion for Reducing Rollbacks in Optimistic Simulation

A new criterion for assigning logical processes to processors during optimistic parallel discrete event simulation runs is presented. Instead of assigning closely coupled logical processes to the same processor, the logical processes experiencing rollback caused by a common source (another logical process) are assigned to the same processor. This approach significantly reduces the number of rollbacks. An algorithm to generate the assignments based on this new criterion is also presented.

Global synchronization for optimistic parallel discrete event simulation

A number of optimistic synchronization schemes for parallel simulation rely upon a global synchronization. The problem is to determine when every processor has completed all its work, and there are no messages in transit in the system that will cause more work. Most previous solutions to the problem have used distributed termination algorithms, which are inherently serial; other parallel mechanisms may be inefficient. In this paper we describe an efficient parallel algorithm derived from a common “barrier” synchronization algorithm used in parallel processing. The algorithm's principal attraction is speed, and generality—it is designed to be used in contexts more general than parallel discrete-event simulation. To establish our claim to speed, we compare our algorithm's performance with the standard barrier algorithm, and find that its additional costs are not excessive. Our experiments are conducted using up to 256 processors on the Intel Touchstone Delta.