Multithreaded Application Real-Time Executor Research Articles

Integrating EDICAM into the MARTe framework

At the Centre for Energy Research a process has started to integrate the Event Detection Intelligent Camera (EDICAM) and its software API into the Multithreaded Application Real-time executor (MARTe) real-time framework. This framework is developed for and widely used in the fusion community (e.g. JET, MAST, COMPASS, ISTTOK etc.) and it is compatible with EPICS, which is the control system e.g. on what ITER's own control system will be based on.Throughout this contribution we integrated the basic real-time image processing functionality into MARTe by implementing a General Application Module (GAM) for EDICAM data processing, a Data Source for the offline testing of new and improved algorithms (using HDF5) and preparations made for implementing a Data Source for the live, real-time operation (using the raw camera data), which otherwise uses the same software stack. The implementation and performance measure of the aforementioned components is discussed.

Conceptual design of the COMPASS-U control systems

The COMPASS-U tokamak is an advanced and more complicated successor of the COMPASS tokamak. Almost all main technological parts will be different. Both the higher performance of COMPASS-U and the increased number of parts and systems lead to a higher risk of failure. Therefore, new control systems must be developed regarding handling dangerous situations and integrating new systems. The COMPASS-U also opens the opportunity to make significant changes in the current systems based on the experience gained from COMPASS operation and to upgrade the systems to nowadays technologies.This article focuses on the conceptual design of the COMPASS-U control systems. It shows all CODAC (Control, Data Access, and Communication) systems their purpose, functionality, operation principle, interaction, and main used technologies.The COMPASS-U will use EPICS (Experimental Physics and Industrial Control System) for Slow control and supervision system. The real-time plasma control system will utilize MARTe2 (Multi-threaded Application Real-Time executor). The base control loop period of the Real-time plasma control system will be 50 μs as of COMPASS.

Real-time plasma position reflectometry system development and integration on COMPASS tokamak

O-mode frequency-modulated continuous wave (FMCW) reflectometry provides an alternative to magnetic measurements in the determination of the plasma separatrix position for plasma position control. This type of measurement proves to be particularly attractive for the control of future fusion reactors where the harsh radiation environment may damage magnetic probes or induce non-compensable measurement drifts. Plasma position reflectometry (PPR), first demonstrated in ASDEX-Upgrade, is a control technique that is increasingly important to validate in diversified experimental devices and relevant plasma regimes. The COMPASS tokamak provides suitable conditions for such advanced demonstrations and regular PPR operation and development, thanks to its O-mode reflectometer and Multi-Threaded Application Real-Time executor (MARTe) based real-time control system. Herein we present the integration of a PPR system on COMPASS, both at hardware and software levels. Reflectometry swept measurements require signals to be acquired in bursts of data and streamed to the corresponding MARTe-PPR node through PCIe® fibre-optic links. The data transferred in real-time is used to reconstruct the radial density profiles from which the outer separatrix position is estimated. This estimate is then delivered to the central MARTe controller node via a dedicated Xilinx® Aurora® link at a rate matching COMPASS's 500 μs slow control cycle. The implemented system systematically met the required latency specifications, being able to deliver an estimation of the plasma radial position capable of successfully replacing the corresponding magnetic measurements in the plasma position feedback control loops.

Real-time multi-threaded reflectometry density profile reconstructions on COMPASS Tokamak

In COMPASS tokamak, the ordinary mode (O-mode) microwave reflectometry diagnostic has been equipped with an upgraded acquisition system and dedicated computational node running the Multi-Threaded Application Real-Time executor (MARTe). This upgrade aims at the deployment of a real-time (RT) plasma position reflectometry (PPR) diagnostic system for future application in closed-loop plasma position control experiments. The system is synchronized with COMPASS MARTe main controller slow cycle, running at 500 μs, and allows a maximum measurement delivery latency of 450 μs. The RT signal processing algorithms, used to reconstruct the O-mode density profiles are the most time consuming step in the control cycle. This is due to the computation of a large number of Fast Fourier Transforms (FFT) over long data arrays, produced for each reflectometry profile measurement. MARTe is a software control framework that allows the integration of external C/C++ libraries and code such as the FFTW3 C library, a highly optimized and performant implementation of the FFT algorithm into the real-time designs. The framework also provides a set of tools to handle the execution of parallel threads, which can be used to expedite the execution of computationally demanding signal processing algorithms. Herein, we present the processing architecture implemented for the multi-threaded reflectometry density profile reconstruction on MARTe framework, using the FTTW3 library. Consistent systematic results have shown that the implemented solution satisfies COMPASS slow control cycle maximum data latency requirement. Moreover, the obtained reduced latency opens the possibility to either increase the number of processed measurements, leading to a more robust plasma position estimate or even to provide multiple position measurements during each control cycle.

Robust configuration of the JET Real-Time Protection Sequencer

The JET Real-Time Protection Sequencer (RTPS) co-ordinates responses for magnetic and kinetic actuators to protect the ITER-Like Wall from possible melting events and other undesirable scenarios. It allows programmable stop responses per pulse, based on alarms raised by other systems.The architecture combines a modular run-time application developed using MARTe (Multithreaded Application Real-Time executor) with the top-level JET supervisory and configuration software, Level-1. Operational experience since 2011 drove a requirement to refactor the system in 2017, moving the maximum degree of functionality from compiled code to configuration data, providing more flexibility, maintainability and verifiability of action(s) to be taken during a pulse.This paper discusses the features of the architecture that made this clean separation of rule-based logic and real-time signal processing possible and practical, including how functions and interfaces between MARTe and Level-1 are organised. It also explains management of configuration data to address development, testing, commissioning and operations, each with individual ownership, responsibility and lifecycles. The core technology enabling this is the Level-1 domain specific language, able to manipulate, validate and load into plant configuration parameter sets. The language also enables implementation of advanced user interfaces, providing operators with the tools to focus on essentials tasks for their area of responsibility. It exemplifies this with recent verification and validation of the refactored protection system: unit/low-level integration tests defined by core developers and integration/behavioural tests defined by JET’s Plasma Operations Group, respectively, ensuring robust and consistent behaviour. We show how the wide scope and power of this language has enabled evolution of JET operations efficiently and correctly over decades of operational experience.

Design and Simulation of ISTTOK Real-Time Magnetic Multiple-Input Multiple-Output Control

Control of the plasma current and the position of its centroid and its shape is vital for obtaining advanced performance regimes in tokamak operation. For that aim, the use of poloidal magnetic coils for feedback control is essential. In this paper, a multiple-input multiple-output (MIMO) optimal control algorithm for the vertical and horizontal control position of the plasma current centroid at IST tokamak (ISTTOK) is presented. Using the ISTTOK magnetic system, the real-time control is made possible by the execution of generic application modules (GAMs), developed to be executed on the Multithreaded Application Real-Time executor ISTTOK controller. The design of the proposed MIMO controller is based on a state-space model of the ISTTOK magnetic response that has been identified from experimental data. Both the control design and the identification procedure rely on the reconstruction of the plasma centroid position computed by the magnetic probes (Mirnov coils) installed at ISTTOK. The centroid position is reconstructed in the Position GAM by numerical integration of the magnetic probes signals in the recently upgraded hardware based on the Advanced Telecommunications Computing Architecture. The effectiveness of the proposed control approach is presented comparing simulation results with experimental ones obtained using single-input single-output PID controllers. Moreover, due to the modularity of the ISTTOK control system, the proposed real-time reconstruction algorithm has been easily deployed on the plant, and the same approach will be adopted to test the MIMO controller in the future.

Real-Time Implementation in JET of the SPAD Disruption Predictor Using MARTe

One of the major problems in present tokamaks is the presence of disruptions. If disruptions are not mitigated, they can produce serious damage to the device. Therefore, disruption predictors are needed in order to apply the mitigation techniques in time. In this paper, the real-time implementation in JET of a new type of disruption predictor is presented. The new predictor, single signal predictor based on anomaly detection (SPAD), predicts disruptions detecting anomalies in the locked mode signal. The implementation is based on the Multithreaded Application Real-Time executor framework. Analysis over JET’s ITER-like wall campaigns (C28–C34) shows that SPAD was able to predict 83.57% of the disruptions with enough time to apply mitigation techniques. The average anticipation time was 389 ms. In this paper, the real-time implementation will be discussed, as well as the optimizations developed to make the algorithm suitable for real-time processing. Performance results and possible improvements will also be analyzed.

Real-Time Vertical Plasma Position Control Using the Heavy Ion Beam Diagnostic

A future thermonuclear fusion reactor relies on the real-time control of, among others, instabilities and plasma position. At the ISTTOK tokamak, a novel approach of a real-time diagnostic system to control the vertical plasma position (in a 100-μs closed-loop) was implemented by: 1) developing the heavy ion beam diagnostic (HIBD) to perform the position measurement and 2) combining data processing on a field-programmable gate array and Multithreaded Application Real-Time executor software framework. The results of this implementation have demonstrated the successful control of the plasma position using the HIBD. It was proven that the HIBD can be used to measure the ISTTOK vertical plasma position and to provide feedback in real time. In this paper, the overall implementation and integration is described and the experimental results are presented.

Integration of the state observer RAPTOR in the real-time MARTe framework at RFX-mod

The RAPTOR – RApid Plasma Transport simulatOR code is a model-based control-oriented code that predicts Tokamak plasma profile evolution in real-time. One of its key applications is in a state observer, where the real-time predictions are combined with the measurements of the available diagnostics, yielding a complete estimate of the plasma profiles. The state observer RAPTOR is currently installed in the real-time control system of TCV, where it has been originally developed, ASDEX-Upgrade and recently RFX-mod. The latter has pioneered its integration in the real-time MARTe – Multi-threaded Application Real-Time executor framework, which will be the topic of this work. Thanks to this, RFX-mod can now contribute to develop integrated control techniques based on the state observer RAPTOR to avoid disruptions, which are highly reproducible in q(a) <2 RFX-mod Tokamak plasmas if they are left uncontrolled.

Coupling DCS and MARTe: two real-time control frameworks in collaboration

Fusion experiments place high demands on real-time control systems. Within the fusion community two modern framework-based software architectures have emerged as powerful tools for developing algorithms for real-time control of complex systems while maintaining the flexibility required when operating a physics experiment. The two frameworks are known as DCS (Discharge Control System), from ASDEX Upgrade and MARTe (Multithreaded Application Real-Time executor), originally from JET.Based on the success of DCS and MARTe, ITER has chosen to develop a framework architecture for its Plasma Control System which will adopt major design concepts from both the existing frameworks.This paper describes a coupling of the two existing frameworks, which was undertaken to explore the degree of similarity and compliance between the concepts, and to extend their capabilities. DCS and MARTe operate in parallel with synchronised state machines and a common message logger. Configuration data is exchanged before the real-time phase. During the real-time phase, structured data is exchanged via shared memory and an existing DCS algorithm is replicated within MARTe. The coupling tests the flexibility and identifies the respective strengths of the two frameworks, providing a well-informed basis on which to move forward and design a new ITER real-time framework.

Parallel Task Management Library for MARTe

The Multithreaded Application Real-Time executor (MARTe) is a real-time framework with increasing popularity and support in the thermonuclear fusion community. It allows modular code to run in a multi-threaded environment leveraging on the current multi-core processor (CPU) technology. One application that relies on the MARTe framework is the Joint European Torus (JET) tokamak WAll Load Limiter System (WALLS). It calculates and monitors the temperature on metal tiles and plasma facing components (PFCs) that can melt or flake if their temperature gets too high when exposed to power loads. One of the main time consuming tasks in WALLS is the calculation of thermal diffusion models in real-time. These models tend to be described by very large state-space models thus making them perfect candidates for parallelisation. MARTe’s traditional approach for task parallelisation is to split the problem into several Real-Time Threads, each responsible for a self-contained sequential execution of an input-to-output chain. This is usually possible, but it might not always be practical for algorithmic or technical reasons. Also, it might not be easily scalable with an increase in the number of available CPU cores. The WorkLibrary introduces a “GPU-like approach” of splitting work among the available cores of modern CPUs that is (i) straightforward to use in an application, (ii) scalable with the availability of cores and all of this (iii) without rewriting or recompiling the source code. The first part of this article explains the motivation behind the library, its architecture and implementation. The second part presents a real application for WALLS, a parallel version of a large state-space model describing the 2D thermal diffusion on a JET tile.

Real-Time Processing System for the JET Hard X-Ray and Gamma-Ray Profile Monitor Enhancement

The Joint European Torus (JET) is currently undertaking an enhancement program, in which one of the objectives is to test relevant diagnostics for the International Thermonuclear Experimental Reactor (ITER), the reference for the next generation of fusion experiments. One of the challenges in ITER is the provision of real-time data analysis and compression capabilities, to sustain the expected long duration discharges and the high acquisition rates achieved by recent data acquisition systems. Foreseeing this real-time requirement, a new system was developed and installed at JET for the gamma-ray and hard X-ray profile monitor diagnostic. The new system, which is connected to 19 CsI(Tl) photodiodes in order to obtain the line-integrated profiles of the gamma-ray and hard X-ray emissions, was designed to overcome the data acquisition limitations of the present fast electron Bremsstrahlung diagnostic (FEB), while exploiting the required real-time features. This paper presents the real-time processing architecture for the JET gamma-ray and hard X-ray profile monitor. The system hardware, based on the Advanced Telecommunication Computer Architecture (ATCA) standard, includes reconfigurable digitizer modules with embedded Field Programmable Gate Array (FPGA) devices capable of acquiring and simultaneously processing data in real-time from the 19 detectors. A suitable algorithm was developed and implemented in the FPGAs, which are able to deliver the corresponding energy of the acquired pulses, and its associated occurrence time. The real-time processed data is sent periodically, during the discharge, through the JET real-time Asynchronous Transfer Mode (ATM) network, and stored in the JET scientific databases at the end of the pulse. Publishing the processed data in the ATM network enables it to be used for machine control purposes (e.g. the information about the line-integrated emissions of the hard X-rays in real time can be used to determine the lower hybrid current drive deposition before the main heating phase). Additionally, the real-time processed data is used for local calibration, using embedded radioactive sources to build in real-time the 19 channels spectra. The acquired raw data is also stored in the digitizer modules' local memory and retrieved after the pulse to the JET database, where it can be post-processed offline to validate the real-time algorithms. The interface between the ATCA digitizers, the JET Control and Data Acquisition System (CODAS) and the JET real-time network is provided by the Multithreaded Application Real-Time executor (MARTe). From the experimental results it was concluded that it is possible to measure in real-time the line-integrals of both hard X-ray and gamma-ray emissions, covering energy range from ∼200 keV to 8 MeV. This allows us to meet two of the major milestones: the ability to process and supply high volume data rates in real-time over a wide spectrum energy range.

A shared memory based interface of MARTe with EPICS for real-time applications

The Multithreaded Application Real-Time executor (MARTe) is a multi-platform C++ middleware designed for the implementation of real-time control systems. It currently supports the Linux, Linux+RTAI, VxWorks, Solaris and MS Windows platforms. In the fusion community MARTe is being used at JET, COMPASS, ISTTOK, FTU and RFX in fusion [1].The Experimental Physics and Industrial Control System (EPICS), a standard framework for the control systems in KSTAR and ITER, is a set of software tools and applications which provide a software infrastructure for use in building distributed control systems to operate devices.For a MARTe based application to cooperate with an EPICS based application, an interface layer between MARTe and EPICS is required. To solve this issue, a number of interfacing solutions have been proposed and some of them have been implemented. Nevertheless, a new approach is required to mitigate the functional limitations of existing solutions and to improve their performance for real-time applications.This paper describes the design and implementation of a shared memory based interface between MARTe and EPICS.

Real-time control for long ohmic alternate current discharges

The ISTTOK tokamak has a long tradition on alternate plasma current (AC) discharges, but the old control system was limiting and lacked full system integration. In order to improve the AC discharges performance the ISTTOK fast control system was updated. This control system developed on site based on the Advanced Telecommunications Computing Architecture (ATCA) standard now integrates the information gathered by all the tokamak real-time diagnostics to produce an accurate observation of the plasma parameters. The real-time actuators were also integrated, allowing a Multiple Input Multiple Output (MIMO) control environment with several synchronization strategies available.The control system software was developed in C++ on top of a Linux system with the Multi-threaded Application Real-Time executor (MARTe) Framework to synchronize the real-time code execution under a 100μs control cycle. In addition, to simplify the discharge programming, a visual Human–Machine Interface (HMI) was also developed using the BaseLib2 libraries included in the MARTe Framework.This paper presents the ISTTOK control system and the optimizations that extended the AC current discharges duration to more than 1s, corresponding to 40 semi-cycles without apparent degradation of the plasma parameters. This upgrade allows ISTTOK to be used as a low-cost material testing facility with long time exposures to nuclear fusion relevant plasmas, comparable (in duration) with medium size tokamaks.

A New Generation of Real-Time Systems in the JET Tokamak

Recently, a new recipe for developing and deploying real-time systems has become increasingly adopted in the JET tokamak. Powered by the advent of x86 multi-core technology and the reliability of JET's well established Real-Time Data Network (RTDN) to handle all real-time I/O, an official Linux vanilla kernel has been demonstrated to be able to provide real-time performance to user-space applications that are required to meet stringent timing constraints. In particular, a careful rearrangement of the Interrupt ReQuests' (IRQs) affinities together with the kernel's CPU isolation mechanism allows one to obtain either soft or hard real-time behavior depending on the synchronization mechanism adopted. Finally, the Multithreaded Application Real-Time executor (MARTe) framework is used for building applications particularly optimised for exploring multi-core architectures. In the past year, four new systems based on this philosophy have been installed and are now part of JET's routine operation. The focus of the present work is on the configuration aspects that enable these new systems' real-time capability. Details are given about the common real-time configuration of these systems, followed by a brief description of each system together with results regarding their real-time performance. A cycle time jitter analysis of a user-space MARTe based application synchronizing over a network is also presented. The goal is to compare its deterministic performance while running on a vanilla and on a Messaging Real time Grid (MRG) Linux kernel.

ISTTOK real-time architecture

The ISTTOK tokamak was upgraded with a plasma control system based on the Advanced Telecommunications Computing Architecture (ATCA) standard. This control system was designed to improve the discharge stability and to extend the operational space to the alternate plasma current (AC) discharges as part of the ISTTOK scientific program. In order to accomplish these objectives all ISTTOK diagnostics and actuators relevant for real-time operation were integrated in the control system.The control system was programmed in C++ over the Multi-threaded Application Real-Time executor (MARTe) which provides, among other features, a real-time scheduler, an interrupt handler, an intercommunications interface between code blocks and a clearly bounded interface with the external devices. As a complement to the MARTe framework, the BaseLib2 library provides the foundations for the data, code introspection and also a Hypertext Transfer Protocol (HTTP) server service.Taking advantage of the modular nature of MARTe, the algorithms of each diagnostic data processing, discharge timing, context switch, control and actuators output reference generation, run on well-defined blocks of code named Generic Application Module (GAM). This approach allows reusability of the code, simplified simulation, replacement or editing without changing the remaining GAMs.The ISTTOK control system GAMs run sequentially each 100μs cycle on an Intel® Q8200 4-core processor running at 2.33GHz located in the ATCA crate. Two boards (inside the ATCA crate) with 32 analog-to-digital converters (ADCs) were used for acquiring the diagnostics data. Each ADC operates at 2Msample/s but (for real-time operation) the acquired data is decimated in real-time on the board's Field-programmable gate array (FPGA) to a frequency defined by the control cycle time.This paper presents the ISTTOK real-time architecture and the human–machine Interface (HMI) for simplified AC discharge programming.

Overview of the COMPASS CODAC system

This paper presents an overview of the Control, Data Acquisition, and Communication system (CODAC) at the COMPASS tokamak: the hardware set-up, software implementation, and communication tools are described.The diagnostics and the data acquisition are tailored for high spatial and temporal resolution required by the COMPASS physics programme, which aims namely at studies of the plasma edge, pedestal, and Scrape-off-Layer (SOL). Studies of instabilities and turbulence are also an integral part of the programme. Therefore, the data acquisition consists of more than 1000 channels, sampled at rates from 500kS/s up to 2GS/s.Presently, the feedback system controls the plasma position and shape, plasma current, and density and it includes 32 analogue input channels as well as 1 digital input/output channel and 8 analogue outputs. The feedback control runs within the Multi-threaded Application Real-Time executor (MARTe) framework with two threads, a 500μs cycle to control slow systems and a 50μs cycle to control the fast feedback power supplies for plasma position control.In this paper, special attention is paid to the links between the systems, to the hardware and software connections, and to the communication. The hardware part is described, the software framework is addressed, and the particular implementation – the dedicated software modules, communication protocols, and links to the database are described.

Real-time algorithms for JET hard X-ray and gamma-ray profile monitor

The steady state operation with high energy content foreseen for future generation of fusion devices will necessarily demand dedicated real-time tools and mechanisms for data handling and machine control. Consequently, the real-time systems for those devices should be carefully selected and their capabilities previously established. The Joint European Torus (JET) is undertaking an enhancement program, which includes tests of relevant real-time tools for the International Thermonuclear Experimental Reactor (ITER), a key experiment for future fusion devices. In these enhancements a new Data AcQuisition (DAQ) system is included, with real-time processing capabilities, for the JET hard X-ray and gamma-ray profile monitor. The DAQ system is composed of dedicated digitizer modules with embedded Field Programmable Gate Array (FPGA) devices. The interface between the DAQ system, the JET control and data acquisition system and the JET real-time data network is provided by the Multithreaded Application Real-Time executor (MARTe). This paper describes the real-time algorithms, developed for both digitizers’ FPGAs and MARTe application, capable of meeting the DAQ real-time requirements. The new DAQ system, including the embedded real-time features, was commissioned during the 2012 experiments. Results achieved with these real-time algorithms during experiments are presented.

ISTTOK control system upgrade

The ISTTOK tokamak (Ip=4kA, BT=0.5T, R=0.46m, a=0.085m) is one of the few tokamaks with regular alternate plasma current (AC) discharges scientific programme. In order to improve the discharge stability and to increase the number of AC discharge cycles a novel control system was developed. The controller acquires data from 50 analog-to-digital converter (ADC) channels of real-time diagnostics and measurements: tomography, Mirnov coils, interferometer, electric probes, sine and cosine probes, bolometer, current delivered by the power supplies, loop voltage and plasma current. The system has a control cycle of 100μs during which it reads all the diagnostics connected to the advanced telecommunications computing architecture (ATCA) digitizers and sends the control reference to ISTTOK actuators. The controller algorithms are executed on an Intel® Q8200 chip with 4 cores running at 2.33GHz and connected to the I/O interfaces through an ATCA based environment. The real-time control system was programmed in C++ on top of the Multi-threaded Application Real-Time executor (MARTe).To extend the duration of the AC discharges and the plasma stability a new magnetising field power supply was commissioned and the horizontal and vertical field power supplies were also upgraded. The new system also features a user-friendly interface based on HyperText Markup Language (HTML) and Javascript to configure the controller parameters.This paper presents the ISTTOK control system and the consequent update of real-time diagnostics and actuators.

Evaluating performance of MARTe as a real-time framework for feed-back control system at tokamak device

The Korea Super conducting Tokamak Advanced Research (KSTAR) is performing the task of “Demonstration and Evaluation of ITER CODAC Technologies at KSTAR” whose objective is the evaluation of real-time technologies for decision making on real-time operating systems (RTOS), real-time frameworks and 10GbE networks. In this task, the Multi-threaded Application Real-Time executor (MARTe) has been evaluated as a real-time framework for real-time feedback control system.The performance of MARTe has been verified by measuring response time and jitter in a path of feedback control from an analog input of a monitoring system to an analog output of an actuator system. In addition, the evaluation has been performed in terms of applicability of MARTe and its performance depending on types of operating system and tuning of CPU affinity and priority.This paper describes the overview of MARTe as a real-time framework, the results of evaluation performance and its implementation.