Runtime Engine Research Articles

Applying model-driven engineering to the domain of chatbots: The Xatkit experience

Chatbots are becoming a common component of many types of software systems. But they are typically developed as a side feature using ad-hoc tools and custom integrations. Moreover, current frameworks are efficient only when designing simple chatbot applications while they still require advanced technical knowledge to define complex interactions and are difficult to evolve along with the company needs. In addition, the deployment of a chatbot application usually requires a deep understanding of the targeted platforms, especially back-end connections, increasing the development and maintenance costs.In this paper, we discuss our experiences building, evolving and distributing the Xatkit framework. Xatkit is a model-based framework built around a Domain-Specific Language to define chatbots (and voicebots and bots in general) in a platform-independent way. Xatkit also comes with a runtime engine that automatically deploys the chatbot application and manages the defined conversation logic over the platforms of choice.Xatkit has significantly evolved since its initial release. This paper focuses on describing the evolution and the reasons (technical and non-technical) that triggered them. We believe our lessons learned can be useful to any other initiative trying to build a successful industrial-level chatbot platform, and in general, any type of model-based solution.

Merak: An Efficient Distributed DNN Training Framework With Automated 3D Parallelism for Giant Foundation Models

Foundation models are in the process of becoming the dominant deep learning technology. Pretraining a foundation model is always time-consuming due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the pretraining process is extremely memory- and communication-intensive. These challenges make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism, and tensor model parallelism, to achieve high training efficiency. However, current 3D parallelism frameworks still encounter two issues: i) they are not transparent to model developers, requiring manual model modification to parallelize training, and ii) their utilization of computation resources, GPU memory, and network bandwidth is insufficient. We propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Merak</i> , an automated 3D parallelism deep learning training framework with high resource utilization. Merak automatically deploys 3D parallelism with an automatic model partitioner, which includes a graph-sharding algorithm and proxy node-based model graph. Merak also offers a non-intrusive API to scale out foundation model training with minimal code modification. In addition, we design a high-performance 3D parallel runtime engine that employs several techniques to exploit available training resources, including a shifted critical path pipeline schedule that increases computation utilization, stage-aware recomputation that makes use of idle worker memory, and sub-pipelined tensor model parallelism that overlaps communication and computation. Experiments on 64 GPUs demonstrate Merak's capability to speed up training performance over state-of-the-art 3D parallelism frameworks of models with 1.5, 2.5, 8.3, and 20 billion parameters by up to 1.42, 1.39, 1.43, and 1.61×, respectively.

Multi-Model Running Latency Optimization in an Edge Computing Paradigm.

Recent advances in both lightweight deep learning algorithms and edge computing increasingly enable multiple model inference tasks to be conducted concurrently on resource-constrained edge devices, allowing us to achieve one goal collaboratively rather than getting high quality in each standalone task. However, the high overall running latency for performing multi-model inferences always negatively affects the real-time applications. To combat latency, the algorithms should be optimized to minimize the latency for multi-model deployment without compromising the safety-critical situation. This work focuses on the real-time task scheduling strategy for multi-model deployment and investigating the model inference using an open neural network exchange (ONNX) runtime engine. Then, an application deployment strategy is proposed based on the container technology and inference tasks are scheduled to different containers based on the scheduling strategies. Experimental results show that the proposed solution is able to significantly reduce the overall running latency in real-time applications.

General ChatBot for Medical Applications

Applications for chat bots (and voice bots) are being used more frequently in customer care or e-commerce as a direct line of communication between businesses and customers. A software programm known as a bot executes automated, repetitive, and pre-defined tasks. For people, it would be nearly impossible to respond to every single client question. The benefit of employing the notion of a bot is that it eliminates the laborious process of manually responding to customer requests. A bot handles everything automatically. Artificial intelligence is a concept that chatbots employ to communicate with people. It has security features like the ability to deactivate accounts that are sending unsolicited messages. We introduce the bot framework in this paper. Bot is a runtime engine that launches the chatbot programme automatically. and controls security. Bot is freely accessible and open source online.

A codesign framework for online data analysis and reduction

AbstractScience applications preparing for the exascale era are increasingly exploring in situ computations comprising of simulation‐analysis‐reduction pipelines coupled in‐memory. Efficient composition and execution of such complex pipelines for a target platform is a codesign process that evaluates the impact and tradeoffs of various application‐ and system‐specific parameters. In this article, we describe a toolset for automating performance studies of composed HPC applications that perform online data reduction and analysis. We describe Cheetah, a new framework for composing parametric studies on coupled applications, and Savanna, a runtime engine for orchestrating and executing campaigns of codesign experiments. This toolset facilitates understanding the impact of various factors such as process placement, synchronicity of algorithms, and storage versus compute requirements for online analysis of large data. Ultimately, we aim to create a catalog of performance results that can help scientists understand tradeoffs when designing next‐generation simulations that make use of online processing techniques. We illustrate the design of Cheetah and Savanna, and present application examples that use this framework to conduct codesign studies on small clusters as well as leadership class supercomputers.

Graphical composite modeling and simulation for multi-aircraft collision avoidance

Modeling and simulation for multi-aircraft collision avoidance to understand the mechanistic behavior is an important activity. Building models using general programming language typically requires specialist knowledge, and this limits the spread of modeling and simulation approach among multi-aircraft collision avoidance scenario. Thus, a software environment is needed to support convenient development of models by assembling components, when analysis demands changes. In this work, the graphical composite modeling and simulation software (GMAS extended) for multi-aircraft collision avoidance is introduced, with the basic graphical components and a graphical assembly editor. We define the serial and parallel execution semantics of GMASE-based model and then introduce the high-level graphical modeling interface, the low-level runtime engine of GMAS, and the simulation-based decision tree, which transforms a complex decision-making process into a collection of simpler decisions of finding the no collision or optimal sequence from some initial state to the goal state. To validate its efficiency and practicability, a three-aircraft collision avoidance model with TCAS operations is built on GMAS, which shows that using GMAS increases reusability and hiding complexity in graphical programming by splitting complex behavior into data flow and function components. The experimental result proves that GMAS not only provides a better representation for multi-aircraft collision avoidance, but also a useful approach for analyzing the potential collision occurrences.

Jointly optimizing preprocessing and inference for DNN-based visual analytics

While deep neural networks (DNNs) are an increasingly popular way to query large corpora of data, their significant runtime remains an active area of research. As a result, researchers have proposed systems and optimizations to reduce these costs by allowing users to trade off accuracy and speed. In this work, we examine end-to-end DNN execution in visual analytics systems on modern accelerators. Through a novel measurement study, we show that the preprocessing of data (e.g., decoding, resizing) can be the bottleneck in many visual analytics systems on modern hardware. To address the bottleneck of preprocessing, we introduce two optimizations for end-to-end visual analytics systems. First, we introduce novel methods of achieving accuracy and throughput trade-offs by using natively present, low-resolution visual data. Second, we develop a runtime engine for efficient visual DNN inference. This runtime engine a) efficiently pipelines preprocessing and DNN execution for inference, b) places preprocessing operations on the CPU or GPU in a hardware- and input-aware manner, and c) efficiently manages memory and threading for high throughput execution. We implement these optimizations in a novel system, Smol, and evaluate Smol on eight visual datasets. We show that its optimizations can achieve up to 5.9X end-to-end throughput improvements at a fixed accuracy over recent work in visual analytics.

Xatkit: A Multimodal Low-Code Chatbot Development Framework

Chatbot (and voicebot) applications are increasingly adopted in various domains such as e-commerce or customer services as a direct communication channel between companies and end-users. Multiple frameworks have been developed to ease their definition and deployment. While these frameworks are efficient to design simple chatbot applications, they still require advanced technical knowledge to define complex interactions and are difficult to evolve along with the company needs (e.g. it is typically impossible to change the NL engine provider). In addition, the deployment of a chatbot application usually requires a deep understanding of the targeted platforms, especially back-end connections, increasing the development and maintenance costs. In this paper, we introduce the Xatkit framework. Xatkit tackles these issues by providing a set of Domain Specific Languages to define chatbots (and voicebots and bots in general) in a platform-independent way. Xatkit also comes with a runtime engine that automatically deploys the chatbot application and manages the defined conversation logic over the platforms of choice. Xatkit's modular architecture facilitates the separate evolution of any of its components. Xatkit is open source and fully available online.

Similarity query support in big data management systems

Similarity query processing is becoming increasingly important in many applications such as data cleaning, record linkage, Web search, and document analytics. In this paper we study how to provide end-to-end similarity query support natively in a parallel database system. We discuss how to express a similarity predicate in its query language, how to build indexes, how to answer similarity queries (selections and joins) efficiently in the runtime engine, possibly using indexes, and how to optimize similarity queries. One particular challenge is how to incorporate existing similarity join algorithms, which often require a series of steps to achieve a high efficiency, including collecting token frequencies, finding matching record id pairs, and reassembling result records based on id pairs. We present a novel approach that uses existing runtime operators to implement such complex join algorithms without reinventing the wheel; doing so positions the system to automatically benefit from future improvements to those operators. The approach includes a technique to transform a similarity join plan into an efficient operator-based physical plan during query optimization by using a template expressed largely in the system’s user-level query language; this technique greatly simplifies the specification of such a transformation rule. We use Apache AsterixDB, a parallel Big Data management system, to illustrate and validate our techniques. We conduct an experimental study using several large, real datasets on a parallel computing cluster to assess the similarity query support. We also include experiments involving three other parallel systems and report the efficacy and performance results.

Towards optimisation of the number of threads in the integration platform engines using simulation models based on queueing theory

The use of applications is important to support the business processes of companies. However, most of these applications are not designed to function collaboratively. An integration solution orchestrates a group of applications, allowing data and functionality reuse. The performance of an integration solution depends on the optimum configuration of the number of threads in the runtime engine provided by the integration platforms. It is common that this configuration relies on the empirical knowledge of the software engineers, and it has a direct impact on the performance of integration solutions. The optimum number of threads may be found by means of simulation models. This article presents a methodology and a tool to assist with the generation of simulation models based on queuing theory, in order to find the optimum number of threads to execute an integration solution focusing on performance improvement. We introduce a case of study to demonstrate and experiments to evaluate our proposal.

From high-level modeling toward efficient and trustworthy circuits

Behavior–interaction–priority (BIP) is a layered embedded system design and verification framework that provides separation of functionality, synchronization, and priority concerns to simplify system design and to establish correctness by construction. BIP framework comes with a runtime engine and a suite of verification tools that use D-Finder and NuSMV as model-checkers. In this paper, we provide a method and a supporting tool that take a BIP system and a set of invariants and compute a reduced sequential circuit with a system-specific scheduler and a designated output that is \(\mathtt {true}\) when the invariants hold. Our method uses ABC, a sequential circuit synthesis and verification framework, to (1) generate an efficient circuit implementation of the system that can be readily translated into FPGA or ASIC implementations and to (2) verify the system and debug it in case a counterexample is found. Moreover, we generate a concurrent C implementation of the circuit that can be directly used for runtime verification. We evaluated our method with two benchmark systems, and our results show that, compared to existing techniques, our method is faster and scales to larger sizes.

FLEP

GPUs are widely adopted in HPC and cloud computing platforms to accelerate general-purpose workloads. However, modern GPUs do not support flexible preemption, leading to performance and priority inversion problems in multi-tasking environments. In this paper, we propose and develop FLEP, the first software system that enables flexible kernel preemption and kernel scheduling on commodity GPUs. The FLEP compilation engine transforms the GPU program into preemptable forms, which can be interrupted during execution and yield all or part of the streaming multi-processors (SMs) in the GPU. The FLEP runtime engine intercepts all kernel invocations and determines which kernels and how those kernels should be preempted and scheduled. Experimental results on two-kernel co-runs demonstrate up to 24.2X speedup for high-priority kernels and up to 27X improvement on normalized average turnaround time for kernels with the same priority. FLEP reduces the preemption latency by up to 41% compared to yielding the whole GPU when the waiting kernels only need several SMs. With all the benefits, FLEP only introduces 2.5% runtime overhead, which is substantially lower than the kernel slicing approach.

FLEP

GPUs are widely adopted in HPC and cloud computing platforms to accelerate general-purpose workloads. However, modern GPUs do not support flexible preemption, leading to performance and priority inversion problems in multi-tasking environments. In this paper, we propose and develop FLEP, the first software system that enables flexible kernel preemption and kernel scheduling on commodity GPUs. The FLEP compilation engine transforms the GPU program into preemptable forms, which can be interrupted during execution and yield all or part of the streaming multi-processors (SMs) in the GPU. The FLEP runtime engine intercepts all kernel invocations and determines which kernels and how those kernels should be preempted and scheduled. Experimental results on two-kernel co-runs demonstrate up to 24.2X speedup for high-priority kernels and up to 27X improvement on normalized average turnaround time for kernels with the same priority. FLEP reduces the preemption latency by up to 41% compared to yielding the whole GPU when the waiting kernels only need several SMs. With all the benefits, FLEP only introduces 2.5% runtime overhead, which is substantially lower than the kernel slicing approach.

A cross-layer optimized storage system for workflow applications

This paper proposes using file system custom metadata as a bidirectional communication channel between applications and the storage middleware. This channel can be used to pass hints that enable cross-layer optimizations, an option hindered today by the ossified file-system interface. We study this approach in the context of storage system support for large-scale workflow execution systems: Our workflow-optimized storage system (WOSS), exploits application hints to provide per-file optimized operations, and exposes data location to enable location-aware scheduling. We argue that an incremental adoption path for adopting cross-layer optimizations in storage exists, present the system architecture for a workflow-optimized storage system and its integration with a workflow runtime engine, and evaluate this approach using synthetic and real applications over multiple success metrics (application runtime, generated network stress, and energy). Our performance evaluation demonstrates that this design brings sizeable performance gains. On a large scale cluster (100 nodes), compared to two production class distributed storage systems (Ceph and GlusterFS), WOSS achieves up to 6× better performance for the synthetic benchmarks and 20–40% better application-level performance gain for real applications.

VirtualSoC

Architectural heterogeneity has proven to be an effective design paradigm to cope with an ever-increasing demand for computational power within tight energy budgets, in virtually every computing domain. Programmable manycore accelerators are currently widely used not only in high-performance computing systems, but also in embedded devices, in which they operate as coprocessors under the control of a general-purpose CPU (the host processor). Clearly, such powerful hardware architectures are paired with sophisticated and complex software ecosystems, composed of operating systems, programming models plus associated runtime engines, and increasingly complex user applications with related libraries. System modeling has always played a key role in early architectural exploration or software development when the real hardware is not available. The necessity of efficiently coping with the huge HW/SW design space provided by the described heterogeneous Systems on Chip (SoCs) calls for advanced full-system simulation methodologies and tools, capable of assessing various metrics for the functional and nonfunctional properties of the target system. In this article, we describe VirtualSoC, a simulation tool targeting the full-system simulation of massively parallel heterogeneous SoCs. We also describe how VirtualSoC has been successfully adopted in several research projects.

An OpenMP Extension that Supports Thread-Level Speculation

OpenMP directives are the de-facto standard for shared-memory parallel programming. However, OpenMP does not guarantee the correctness of the parallel execution of a given loop if runtime data dependences arise. Consequently, many highly-parallel regions cannot be safely parallelized with OpenMP due to the possibility of a dependence violation. In this paper, we propose to augment OpenMP capabilities, by adding thread-level speculation (TLS) support. Our contribution is threefold. First, we have defined a new speculative clause for variables inside parallel loops. This clause ensures that all accesses to these variables will be carried out according to sequential semantics. Second, we have created a new, software-based TLS runtime library to ensure correctness in the parallel execution of OpenMP loops that include speculative variables. Third, we have developed a new GCC plugin, which seamlessly translates our OpenMP speculative clause into calls to our TLS runtime engine. The result is the ATLaS C Compiler framework, which takes advantage of TLS techniques to expand OpenMP functionalities, and guarantees the sequential semantics of any parallelized loop.

DiagnosticAid: Une plateforme collaborative pour les guides de pratique en médecine générale

A doctor’s job consists in making decisions in an environment of uncertainty. Certain tools and aids can help in this decision-making process, such as medical practice guidelines. When computerized, these save time and reduce error. The DiagnosticAid project aims to provide doctors with a computerized help tool for effective decision making based on medical practice guidelines developed for general medicine. Our in-depth knowledge of consulting processes has enabled us to develop a specialized tool as well as implement specially-designed test scenarios. In order to develop it, various new technologies had to be created, including a runtime environment for practice guidelines. To this end, we have developed a model, serialization, formula-based language, and runtime engine. This paper outlines in four sections the development of the different technologies.

DiagnosticAid: Une plateforme collaborative pour les guides de pratique en médecine générale

A doctor’s job consists in making decisions in an environment of uncertainty. Certain tools and aids can help in this decision-making process, such as medical practice guidelines. When computerized, these save time and reduce error. The DiagnosticAid project aims to provide doctors with a computerized help tool for effective decision making based on medical practice guidelines developed for general medicine. Our in-depth knowledge of consulting processes has enabled us to develop a specialized tool as well as implement specially-designed test scenarios. In order to develop it, various new technologies had to be created, including a runtime environment for practice guidelines. To this end, we have developed a model, serialization, formula-based language, and runtime engine. This paper outlines in four sections the development of the different technologies.

Modeling Support for Role-Based Delegation in Process-Aware Information Systems

In the paper, an integrated approach for the modeling and enforcement of delegation policies in process-aware information systems is presented. In particular, a delegation extension for process-related role-based access control (RBAC) models is specified. The extension is generic in the sense that it can be used to extend process-aware information systems or process modeling languages with support for process-related RBAC delegation models. Moreover, the detection of delegation-related conflicts is discussed and a set of pre-defined resolution strategies for each potential conflict is provided. Thereby, the design-time and runtime consistency of corresponding RBAC delegation models can be ensured. Based on a formal metamodel, UML2 modeling support for the delegation of roles, tasks, and duties is provided. A corresponding case study evaluates the practical applicability of the approach with real-world business processes. Moreover, the approach is implemented as an extension to the BusinessActivity library and runtime engine.

The Case for Workflow-Aware Storage:An Opportunity Study

This article evaluates the potential gains a workflow-aware storage system can bring. Two observations make us believe such storage system is crucial to efficiently support workflow-based applications: First, workflows generate irregular and application-dependent data access patterns. These patterns render existing generic storage systems unable to harness all optimization opportunities as this often requires enabling conflicting optimizations or even conflicting design decisions at the storage system level. Second, most workflow runtime engines make suboptimal scheduling decisions as they lack the detailed data location information that is generally hidden by the storage system. This paper presents a limit study that evaluates the potential gains from building a workflow-aware storage system that supports per-file access optimizations and exposes data location. Our evaluation using synthetic benchmarks and real applications shows that a workflow-aware storage system can bring significant performance gains: up to 3x performance gains compared to a vanilla distributed storage system deployed on the same resources yet unaware of the possible file-level optimizations.