Computational Steering Research Articles

JuMonC: A RESTful tool for enabling monitoring and control of simulations at scale

As systems and simulations grow in size and complexity, it is challenging to maintain efficient use of resources and avoid failures. In this scenario, monitoring becomes even more important and mandatory. This paper describes and discusses the benefits of the advanced monitoring and control tool JuMonC, which runs under user control alongside HPC simulations and provides valuable metrics via REST-API. In addition, plugin extensibility allows JuMonC to go a step further and provide computational steering of the simulation itself. To demonstrate the benefits and usability of JuMonC for large-scale simulations, two use cases are described employing nekRS and ICON on JURECA-DC, a supercomputer located at the Jülich Supercomputing Centre (JSC). Furthermore, a large-scale use case with nekRS on JSC’s flagship system JUWELS Booster is described. Finally, the interplay between JuMonC and LLview (a standard monitoring tool for HPC systems) is presented using a simple and secure JuMonC-LLview plugin, which collects performance metrics and enables their analysis in LLview. Overall, the portability and usefulness of JuMonC, together with its low performance impact, make it an important application for both current and future generations of exascale HPC systems.

Adaptive Synchronization and Pacing Control for Visual Interactive Simulation

Parallel and distributed computing enable the execution of large and complex simulations. Yet, the usual separation of (headless) simulation execution and (subsequent, offline) output analysis often renders the simulation endeavor long and inefficient. Recently, Visual Interactive Simulation (VIS) tools and methods that address this end-to-end efficiency are gaining relevance, offering in-situ visualization, real-time debugging, and computational steering. Here, the typically distributed computing nature of the simulation execution poses synchronization challenges between the headless simulation engine and the user-facing frontend required for Visual Interactive Simulation. To the best of our knowledge, state-of-the-art synchronization approaches fall short due to their rigidity and inability to adapt to real-time user-centric changes. This paper introduces a novel adaptive algorithm to dynamically adjust the simulation’s pacing through a buffer-based framework, informed by predictive workload analysis. Our extensive experimental evaluation across diverse synthetic scenarios illustrates our method’s effectiveness in enhancing runtime efficiency and synchronicity, significantly reducing end-to-end time while minimizing user interaction delays, thereby addressing key limitations of existing synchronization strategies.

Green Data Analytics of Supercomputing from Massive Sensor Networks: Does Workload Distribution Matter?

Energy costs represent a significant share of the total cost of ownership in high-performance computing systems. Using a unique data set collected by massive sensor networks in a petascale national supercomputing center, we first present an explanatory model to identify key factors affecting energy consumption in supercomputing. Our analytic results show that workload distribution among the nodes has significant effects and could effectively be leveraged to improve energy efficiency. We then establish the high model performance using in-sample and out-of-sample analyses and develop prescriptive models for energy-optimal runtime workload management. We present four dynamic resource management methodologies (packing, load balancing, threshold-based switching, and energy optimization), model their application at two levels (within-rack and cross-rack resource allocation), and explore runtime resource redistribution policies for jobs under the computational steering and comparatively evaluate strategies that use computational steering with those that do not. Our experimental results lead to a threshold strategy that yields near-optimal energy efficiency under all workload conditions. We further calibrate the energy-optimal resource allocations over the full range of workloads and present a bi-criteria evaluation to consider energy consumption and job performance tradeoffs. We conclude with implementation guidelines and policy insights into energy-efficient computing resource management in large supercomputing centers.

Steering-by-example for Progressive Visual Analytics

Progressive visual analytics allows users to interact with early, partial results of long-running computations on large datasets. In this context, computational steering is often brought up as a means to prioritize the progressive computation. This is meant to focus computational resources on data subspaces of interest so as to ensure their computation is completed before all others. Yet, current approaches to select a region of the view space and then to prioritize its corresponding data subspace either require a one-to-one mapping between view and data space, or they need to establish and maintain computationally costly index structures to trace complex mappings between view and data space. We present steering-by-example, a novel interactive steering approach for progressive visual analytics, which allows prioritizing data subspaces for the progression by generating a relaxed query from a set of selected data items. Our approach works independently of the particular visualization technique and without additional index structures. First benchmark results show that steering-by-example considerably improves Precision and Recall for prioritizing unprocessed data for a selected view region, clearly outperforming random uniform sampling.

EdgeCFD: a parallel residual-based variational multiscale code for multiphysics

High fidelity multiphysics simulations are ubiquitous in science and engineering but still face many challenges to run efficiently in today's supercomputers. This work reports advanced technologies present in EdgeCFD, a hybrid parallel variational multiscale multiphysics finite element software capable of running industrial problems involving turbulent incompressible, compressible, free-surface flows, advection-diffusion of multiple scalar fields, fluid-object interaction using an Arbitrary Lagrangian-Eulerian formulation, and, more recently, used to supporting uncertainty quantification. We also show the strategies to support the solution of large scale problems, including in-situ visualisation and how EdgeCFD handles computational steering. Industrial applications in the oil and gas industry are shown to demonstrate EdgeCFD capabilities. We present the analysis of vortex-induced vibration in cylindrical risers with appendages to reduce drag and lift, and the simulation of successive discharges of turbidity currents, a critical process for understanding the formation of oil reservoirs.

In situ analysis and visualization of massively parallel simulations of transitional and turbulent flows

The increase of computational resources with the generalization of massively parallel supercomputers benefits to various fields of physics among which turbulence and fluid mechanics, making it possible to increase time and space accuracy and gain further knowledge in fundamental mechanisms. Parametric studies, high fidelity statistics, high resolutions, can be realized. However, this access poses many problems in terms of data management, analysis and visualization. Traditional workflow, consisting of writing raw data on disks and performing post-processing to extract physical quantities of interest, considerably slows down the analysis, if not becomes impossible, because of data transfer, storage and re-accessibility issues. This is particularly difficult when it comes to visualization. Usage has to be revisited to maintain consistency with the accuracy of the computation step and in this context, in situ processing is a promising approach. We developed an in situ analysis and visualization strategy with an hybrid method for transitional and turbulent flow analysis with a pseudo-spectral solver. It is shown to have a low impact on computational time with a reasonable increase of resource usage, while enriching data exploration. Large time sequences have been analyzed. This could not have been achieved with the traditional workflow. Moreover, computational steering has been performed with real-time adjustment of the simulations, thereby getting closer to a numerical experiment process.

Application Health Monitoring for Extreme-scale Resiliency using Cooperative Fault Management.

Resiliency is and will be a critical factor in determining scientific productivity on current and exascale supercomputers, and beyond. Applications oblivious to and incapable of handling transient soft and hard errors could waste supercomputing resources or, worse, yield misleading scientific insights. We introduce a novel application-driven silent error detection and recovery strategy based on application health monitoring. Our methodology uses application output that follows known patterns as indicators of an application's health, and knowledge that violation of these patterns could be indication of faults. Information from system monitors that report hardware and software health status is used to corroborate faults. Collectively, this information is used by a fault coordinator agent to take preventive and corrective measures by applying computational steering to an application between checkpoints. This cooperative fault management system uses the Fault Tolerance Backplane as a communication channel. The benefits of this framework are demonstrated with two real application case studies, molecular dynamics and quantum chemistry simulations, on scalable clusters with simulated memory and I/O corruptions. The developed approach is general and can be easily applied to other applications.

A framework for a multi-agent traffic simulation using combined behavioural models

The task of simulation of urban processes often concerns city traffic and road dynamics. Many assumptions in an urban science may be made by analysing traffic flows and vehicle behaviour on city streets. Since road traffic is a dynamic and possibly spatially large system that depends on many external conditions, approaches like visual analysis and computational steering may be used. There are many traffic modelling frameworks presented in the field, but there are also several drawbacks most of them have: they are limited to a specific model or case; have abstracted logic which disregards some factors affecting vehicle movement; they ignore possible physical interactions which are not described in a model or use too many simplifications in vehicle interaction simulations. In this paper, a novel agent-based traffic modelling framework is presented, combining different classes of traffic models into a single vehicle agent and allowing the user to set a specific model for each supported class. The framework was made using a game engine, Unreal Engine 4, which allows users to model realistic interaction between vehicles and make realistic visualisations for purposes of visual analysis and computational steering.

Data reduction in scientific workflows using provenance monitoring and user steering

Scientific workflows need to be iteratively, and often interactively, executed for large input datasets. Reducing data from input datasets is a powerful way to reduce overall execution time in such workflows. When this is accomplished online (i.e., without requiring the user to stop execution to reduce the data, and then resume), it can save much time. However, determining which subsets of the input data should be removed becomes a major problem. A related problem is to guarantee that the workflow system will maintain execution and data consistent with the reduction. Keeping track of how users interact with the workflow is essential for data provenance purposes. In this paper, we adopt the “human-in-the-loop” approach, which enables users to steer the running workflow and reduce subsets from datasets online. We propose an adaptive workflow monitoring approach that combines provenance data monitoring and computational steering to support users in analyzing the evolution of key parameters and determining the subset of data to remove. We extend a provenance data model to keep track of users’ interactions when they reduce data at runtime. In our experimental validation, we develop a test case from the oil and gas domain, using a 936-cores cluster. The results on this test case show that the approach yields reductions of 32% of execution time and 14% of the data processed.

Computational steering of GEM based detector simulations

Gas based detector R&D relies heavily on full simulation of detectors and their optimization before final prototypes can be built and tested. These simulations in particular those with complex scenarios such as those involving high detector voltages or gas with larger gains are computationally intensive may take several days or weeks to complete. These long-running simulations usually run on the high-performance computers in batch mode. If the results lead to unexpected behavior, then the simulation might be rerun with different parameters. However, the simulations (or jobs) may have to wait in a queue until they get a chance to run again because the supercomputer is a shared resource that maintains a queue of other user programs as well and executes them as time and priorities permit. It may result in inefficient resource utilization and increase in the turnaround time for the scientific experiment. To overcome this issue, the monitoring of the behavior of a simulation, while it is running (or live), is essential. In this work, we employ the computational steering technique by coupling the detector simulations with a visualization package named VisIt to enable the exploration of the live data as it is produced by the simulation.

In-Situ Visualization for 3D Agent-Based Vocal Fold Inflammation and Repair Simulation

A fast and insightful visualization is essential in modeling biological system behaviors and understanding underlying inter-cellular mechanisms. High fidelity models produce billions of data points per time step, making in situ visualization techniques extremely desirable as they mitigate I/O bottlenecks and provide computational steering capability. In this work, we present a novel high-performance scheme to couple in situ visualization with the simulation of the vocal fold inflammation and repair using little to no extra cost in execution time or computing resources. The visualization component is first optimized with an adaptive sampling scheme to accelerate the rendering process while maintaining the precision of the displayed visual results. Our software employs VirtualGL to perform visualization in situ. The scheme overlaps visualization and simulation, resulting in the optimal utilization of computing resources. This results in an in situ system biology simulation suite capable of remote simulation of 17 million biological cells and 1.2 billion chemical data points, remote visualization of the results, and delivery of visualized frames with aggregated statistics to remote clients in real-time.

Computational steering for geosimulations

Geosimulations using computer simulation models provide researchers an effective way to study complex geographic phenomena and their outcomes. These simulations allow for scenario based exploration by capturing spatial and temporal relationships between various geographic processes in a region. However, current approaches to geosimulation limit manipulating model input and exploring alternative scenarios by controlling the simulation model at runtime. This paper proposes a computational steering system for geosimulation models and presents a prototype, tFUTURES , developed for the FUTURES Urban Growth Model (UGM). By allowing tangible inputs and implementing mechanisms to control model execution, this system solves the problem of lack of user-interactivity experienced at runtime. We develop a web interface and leverage the WMS, WFS-t and WPS OGC services to help visualize, modify and execute geosimulations. We define new steering controls within this interface and implement application checkpointing , allowing a user to provide new steering input and execute steering actions that can pause, advance or rollback a geosimulation and display the model outcomes in near real-time.

Advanced finite element modeling of excavation and advancement processes in mechanized tunneling

Mechanized tunneling is characterized by complex interactions between the shield machine and the surrounding ground during the TBM advancement process. In this paper, a new computational framework is developed to enable an efficient and realistic three-dimensional modeling of the tunneling process for arbitrary alignments using the finite element method. A new steering algorithm for the advancement of the Tunnel Boring Machine (TBM) for arbitrary alignments during shield tunneling is incorporated in the proposed model. This algorithm simulates the shield behavior and accordingly provides the numerical model with the required information to keep the TBM on track during the simulation. However, the utilization of this algorithm is only possible using a finite element discretization which adapts to the actual motion path of the shield machine. For this purpose, a re-meshing technique is proposed in order to automate the process of mesh generation in the vicinity of the tunnel face, denoted as the region of interest, within the advancing process. The combination of a computational steering algorithm and a 3D automatic adaptive mesh generation procedure form a novel framework for process oriented finite element simulations of the mechanized tunnel construction process. The applicability of the proposed modeling technique for predicting the shield behavior and the soil-tunnel interactions during tunneling along curved alignments is demonstrated by means of selected examples.

In situ analysis and visualization of massively parallel computations

Massively parallel simulations generate increasing volumes of big data, whose exploitation requires increasingly large storage resources, efficient networking technologies and post-processing facilities. In the coming era of exascale supercomputing, there is an emerging need for new data analysis and visualization strategies. A promising solution consists of coupling analysis with simulation, so that both are performed simultaneously. This paper describes a client–server in situ analysis for massively parallel time-evolving computations. Its application to very large turbulent transition simulations using a spectral approximation is presented. It is shown to have a low impact on the computational time with a reasonable increase of resource usage, while enriching data exploration. Computational steering is performed with real-time adjustment of the simulation parameters, thereby getting closer to a numerical experiment process. This would not have been achieved with a classical work flow using off-line visualization.

Hierarchical visual analysis and steering framework for astrophysical simulations

A framework for accelerating modern long-running astrophysical simulations is presented, which is based on a hierarchical architecture where computational steering in the high-resolution run is performed under the guide of knowledge obtained in the gradually refined ensemble analyses. Several visualization schemes for facilitating ensemble management, error analysis, parameter grouping and tuning are also integrated owing to the pluggable modular design. The proposed approach is prototyped based on the Flash code, and it can be extended by introducing user-defined visualization for specific requirements. Two real-world simulations, i.e., stellar wind and supernova remnant, are carried out to verify the proposed approach.

Isogeometric Fatigue Damage Prediction in Large-Scale Composite Structures Driven by Dynamic Sensor Data

In this paper, we combine recent developments in modeling of fatigue-damage, isogeometric analysis (IGA) of thin-shell structures, and structural health monitoring (SHM) to develop a computational steering framework for fatigue-damage prediction in full-scale laminated composite structures. The main constituents of the proposed framework are described in detail, and the framework is deployed in the context of an actual fatigue test of a full-scale wind-turbine blade structure. The results indicate that using an advanced computational model informed by in situ SHM data leads to accurate prediction of the damage zone formation, damage progression, and eventual failure of the structure. Although the blade fatigue simulation was driven by test data obtained prior to the computation, the proposed computational steering framework may be deployed concurrently with structures undergoing fatigue loading.

Model update and real-time steering of tunnel boring machines using simulation-based meta models

A method for the simulation supported steering of the mechanized tunneling process in real time during construction is proposed. To enable real-time predictions of tunneling induced surface settlements, meta models trained a priori from a comprehensive process-oriented computational simulation model for mechanized tunneling for a certain project section of interest are introduced. For the generation of the meta models, Artificial Neural Networks (ANN) are employed in conjunction with Particle Swarm Optimization (PSO) for the model update according to monitoring data obtained during construction and for the optimization of machine parameters to keep surface settlements below a given tolerance. To provide a rich data base for the training of the meta model, the finite element simulation model for tunneling is integrated within an automatic data generator for setting up, running and postprocessing the numerical simulations for a prescribed range of parameters. Using the PSO-ANN for the inverse analysis, i.e. identification of model parameters according to monitoring results obtained during tunnel advance, allows the update of the model to the actual geological conditions in real time. The same ANN in conjunction with the PSO is also used for the determination of optimal steering parameters based on target values for settlements in the forthcoming excavation steps. The paper shows the performance of the proposed simulation-based model update and computational steering procedure by means of a prototype application to a straight tunnel advance in a non-homogeneous soil with two soil layers separated by an inclined boundary.

Computational steering of CFD simulations using a grid computing environment

Simulation of complex phenomena is usually a long computing process and it has been traditionally performed in batch mode on large high performance computing (HPC) systems. However, advances in computer processing and networking capabilities can now be used to monitor and alter simulation parameters whilst it is running. This process is called computational steering. By combining this capability with advanced communication tools, it is now possible for a group of scientists located across the world to work collaboratively while visualising on-going simulations. This raise the possibility that researches can now share their experience and promote new ideas and solutions by exploring collaboratively the solution space of a complex simulation. In this paper, a collaborative computational steering environment specialised to solve CFD problems is presented.

Petri Net-Based Collaborative Simulation and Steering of Biochemical Reaction Networks

Computational steering is an interactive remote control of a long running application. The user can adopt it, e.g., to adjust simulation parameters on the fly. Simulation of large-scale biochemical networks is often computationally expensive, particularly stochastic and hybrid simulation. Such extremely time-consuming computations necessitate an interactive mechanism to permit users to try different paths and ask “what-if-questions” while the simulation is in progress. Furthermore, with the progress of computational modelling and the simulation of biochemical networks, there is a need to manage multi-scale models, which may contain species or reactions at different scales. In this context, Petri nets are of special importance, since they provide an intuitive visual representation of reaction networks. In this paper, we introduce a framework and its implementation for combining Petri nets and computational steering for the representation and interactive simulation of biochemical networks. The main merits of the developed framework are: intuitive representation of biochemical networks by means of Petri nets, distributed collaborative and interactive simulation, and tight coupling of simulation and visualisation.

Visual numerical steering in 3D AGENT code system for advanced nuclear reactor modeling and design

The AGENT simulation system is used for detailed three-dimensional modeling of neutron transport and corresponding properties of nuclear reactors of any design. Numerical solution to the neutron transport equation in the AGENT system is based on the Method of Characteristics (MOCs) and the theory of R-functions. The latter of which is used for accurately describing current and future heterogeneous lattices of reactor core configurations. The AGENT code has been extensively verified to assure a high degree of accuracy for predicting neutron three-dimensional point-wise flux spatial distributions, power peaking factors, reaction rates, and eigenvalues. In this paper, a new AGENT code feature, a computational steering, is presented. This new feature provides a novel way for using deterministic codes for fast evaluation of reactor core parameters, at no loss to accuracy. The computational steering framework as developed at the Technische Universität München is smoothly integrated into the AGENT solver. This framework allows for an arbitrary interruption of AGENT simulation, allowing the solver to restart with updated parameters. One possible use of this is to accelerate the convergence of the final values resulting in significantly reduced simulation times. Using this computational steering in the AGENT system, coarse MOC resolution parameters can initially be selected and later update them – while the simulation is actively running – into fine resolution parameters. The utility of the steering framework is demonstrated using the geometry of a research reactor at the University of Utah: this new approach provides a savings in CPU time on the order of 50%.