EMSBench: Benchmark and Testbed for Reactive Real-Time Systems

Static Worst-Case Execution Time (WCET) analysis is currently taking a step from research to industrial use. We present a summary of three case studies where static WCET analysis has been used to analyse production code for embedded real-time systems. The primary purpose has not been to test the accuracy of the obtained WCET estimates, but rather to investigate the practical and methodological diculties that arise when applying current WCET analysis methods to these particular kind of systems. In particular, we have been interested in how laborintensive the analysis becomes, for instance by estimating the eorts to study the analysed code in detail, and measuring the number of manual annotations necessary to perform the analysis. From these observations, we draw some conclusions about what would be needed to turn static WCET analysis into a useful tool for embedded and real-time systems software development.

Worst-case execution time (WCET) analysis is one of the major tasks in timing validation of hard real-time systems. In complex systems with real-time operating systems (RTOS), the timing properties of the system are decided by both the applications and RTOS. Traditionally, WCET analysis mainly deals with application programs, while it is crucial to know whether RTOS also behaves in a timely predictable manner. In this paper, static analysis techniques are used to predict the WCET of the system calls and the Disable Interrupt regions of the μC/OS-II real-time kernel, which presents a quantitative evaluation of the real-time performance of μC/OS-II. The precision of applying existing WCET analysis techniques on RTOS is evaluated, and the practical difficulties in using static methods in timing analysis of RTOS are also discussed.

The number of embedded computers used in modern cars have increased dramatically during the last years, as they provide increased functionality to a reduced cost compared to previous technologies. These computers are often connected by one or more communication networks and the data traffic sent over the networks often has hard real-time requirements. To provide overall system timing guarantees, upper timing bounds need to be derived both for the data traffic and the embedded computer programs that controls the communication. In this article, we present a case study where static worst-case execution time (WCET) analysis was used to find upper time bounds for time-critical code in products from Volcano Communications Technologies AB (VCT). The VCT company provides tools for development of real-time communication solutions for embedded network systems, mainly used within the car industry. VCT's tool suite includes support for Controller Area Network (CAN), Local Interconnect Network (LIN), FlexRay and MOST network traffic. The primary purpose of the study was not to test the accuracy of the obtained WCET estimates, but rather to investigate the practical difficulties that arise when applying current WCET analysis methods to these particular kind of systems. A central question was if today is static WCET analysis tools can be used in the automotive software development process. In particular, we were interested in how labor-intensive the analysis becomes, measured by the number of manual annotations necessary to perform the analysis. As a result, we provide some qualitative observations on desirable research results for making static WCET analysis applicable in typical automotive software development.

Due to the dynamic behaviour of acceleration mechanisms such as caches and branch predictors, static Worst-case Execution Time (WCET) analysis methods tend to scale poorly to modern hardware architectures. As a result, a trade-off must be found between the duration and the precision of the analysis, leading to an overestimation of the WCET bounds. In turn, this reduces the schedulability and resource usage of the system. In this article, we present a new data structure to speed up the analysis: the eXecution Decision Diagram (XDD), which is an ad hoc extension of Binary Decision Diagrams tailored for WCET analysis problems. We show how XDDs can be used to represent efficiently execution states in a modern hardware platform. Moreover, we propose a new process to build the Integer Linear Programming system of the Implicit Path Enumeration Technique using XDD. We use benchmark applications to demonstrate how the use of an XDD substantially increases the scalability of WCET analysis and the precision of the obtained WCET.

Hybrid measurement-based approaches to worst-case execution time (WCET) analysis combine measured execution times of small program segments using static analysis of the larger software structure. In order to make the necessary measurements, instrumentation code is added to generate a timestamped trace from the running program. The intrusive presence of this instrumentation code incurs a timing penalty, widely referred to as the probe effect. However, recent years have seen the emergence of trace capability at the hardware level, effectively opening the door to probe-free analysis. Relying on hardware support forces the WCET analysis to the object-code level, since that is all that is known by the hardware. A major disadvantage of this is that it is expensive for a typical software engineer to interpret the results, since most engineers are familiar with the source code but not the object code. Meaningful WCET analysis involves not just running a tool to obtain an overall WCET value but also understanding which sections of code consume most of the WCET in order that corrective actions, such as optimisation, can be applied if the WCET value is too large. The main contribution of this paper is a mechanism by which hybrid WCET analysis can still be performed at the source level when the timestamped trace has been collected at the object level by state-of-the-art hardware. This allows existing, commercial tools, such as \rapitime{}, to operate without the need for intrusive instrumentation and thus without the probe effect.

Estimation of Worst Case Execution Times (WCETs) of software functions in a real-time embedded system is fundamentally important in verifying its timing behaviour. Many existing WCET analysis tools and benchmarks, based on static WCET analysis methods, are limited to analyse software functions that conform to specific programming languages and libraries. As a consequence, these tools do not support WCET estimation of software functions in legacy industrial systems that do not conform to those languages and libraries. This paper advocates the use of a statistical method based on extreme value theory for estimating WCETs of software functions in such legacy industrial embedded systems. The main advantage of this method is that it is agnostic of the languages and libraries that are used to implement the software functions. In order to provide a proof of concept, the paper incorporates an industrial use case and applies the method to estimate WCETs of its software functions. The paper also presents an extensive comparative evaluation of the statistical method and an established static analysis tool for estimating WCETs that utilises high-level flow analysis.

We have developed a new programming paradigmwhich, for conforming programs, allows the average-case execution time (ACET) to be obtained automatically by a static analysis. This is achieved by tracking the data structures and their distributions that will exist during all possible executions of a program. This new programming paradigm is called MOQA and the tool which performs the static analysis is called Distritrack. In this paper we give an overview of both MOQA and Distritrack. We then discuss the possibility of extending Distritrack for static worst-case execution time (WCET) analysis ofMOQA programs using the tight tracking of data structures already being performed. The authors wish to acknowledge the support of SFI grant SFI 02/IN.1/I181.

Powerful cache systems are required in multi- and many-core systems in order to provide a suitable performance. Moreover, these caches must provide coherent accesses to shared data. With respect to embedded real-time systems, traditional write-invalidate and write-update coherence protocols hamper a static worst case execution time analysis, since the content and state of a local cache can be modified by other cores. Hence, a meaningful cache analysis is no more possible. In previous work, we introduced the On-Demand Coherent Cache (ODC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) which provides coherent accesses to shared data without any coherence transactions and without modifications of other caches. It allows a tight worst case execution time (WCET) analysis at4 the expense of a decreased hit rate and an additional write-back procedure. In this work, we briefly describe two versions of the ODC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> before we present detailed evaluations of its performance and the cost of time predictability. We quantify the overhead of the writeback procedure of one of ODC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> 's variations and we estimate the impact on the hit rate.

Worst-Case Execution Time (WCET) analysis derives upper bounds for the execution times of programs. Such bounds are crucial when designing and verifying real-time systems. Static WCET analysis derives safe upper bounds. For complex hardware architectures the hardware modelling is still a challenge, leading to long analysis times and a risk of large WCET overestimation. Therefore, hybrid WCET analysis methods have appeared, where measurements are used to augment or replace the detailed low-level static WCET analysis. These methods do not in general yield a safe WCET estimate, but can still be appropriate in soft real-time systems where such WCET estimates are not crucial. In this paper we make two contributions. First, we develop a hybrid WCET analysis method, which uses regression to identify parameters in the common linear Implicit Path Enumeration Technique (IPET) model for WCET calculation. The method can use timing measurements of different granularity, including end-to-end measurements, which reduces the need for fine-grained timing measurement instrumentation. It uses a novel kind of regression, which guarantees that the identified model does not underestimate any observed execution times. Second, we initiate the development of an IPET-based theory for hybrid WCET analysis test coverage, and we formulate and prove a coverage criterion for the tests needed to identify a safe model.

Worst case execution time (WCET) analysis is essential for exposing timeliness defects when developing hard real-time systems. However, it is too late to fix timeliness defects cheaply since developers generally perform WCET analysis in a final verification phase. To help developers quickly identify real timeliness defects in an early programming phase, a novel interactive WCET prediction with warning for timeout risk is proposed. The novelty is that the approach not only fast estimates WCET based on a control flow tree (CFT), but also assesses the estimated WCET with a trusted level by a lightweight false path analysis. According to the trusted levels, corresponding warnings will be triggered once the estimated WCET exceeds a preset safe threshold. Hence developers can identify real timeliness defects more timely and efficiently. To this end, we first analyze the reasons of the overestimation of CFT-based WCET calculation; then we propose a trusted level model of timeout risks; for recognizing the structural patterns of timeout risks, we develop a risk data counting algorithm; and we also give some tactics for applying our approach more effectively. Experimental results show that our approach has almost the same running speed compared with the fast and interactive WCET analysis, but it saves more time in identifying real timeliness defects.

Static worst-case execution time (WCET) analysis is a technique to derive upper bounds for the execution times of programs. Such bounds are crucial when designing and verifying real-time systems. A key component for statically deriving safe and tight WCET bounds is information on the possible program flow through the program. Such flow information can be provided manually by user annotations, or automatically by a flow analysis. To make WCET analysis as simple and safe as possible, it should preferably be automatically derived, with no or very limited user interaction. In this paper we present a method for deriving such flow information called abstract execution. This method can automatically calculate loop bounds, bounds for including nested loops, as well as many types of infeasible paths. Our evaluations show that it can calculate WCET estimates automatically, without any user annotations, for a range of benchmark programs, and that our techniques for nested loops and infeasible paths sometimes can give substantially better WCET estimates than using loop bounds analysis only

In this article we give an overview of the worst-case execution time (WCET) analysis research performed by the WCET group of the ASTEC Competence Centre at Uppsala University. Knowing the WCET of a program is necessary when designing and verifying real-time systems. The WCET depends both on the program flow, such as loop iterations and function calls, and on hardware factors, such as caches and pipelines. WCET estimates should be both safe (no underestimation allowed) and tight (as little overestimation as possible). We have defined a modular architecture for a WCET tool, used both to identify the components of the overall WCET analysis problem, and as a starting point for the development of a WCET tool prototype. Within this framework we have proposed solutions to several key problems in WCET analysis, including representation and analysis of the control flow of programs, modeling of the behavior and timing of pipelines and other low-level timing aspects, integration of control flow information and low-level timing to obtain a safe and tight WCET estimate, and validation of our tools and methods. We have focussed on the needs of embedded real-time systems in designing our tools and directing our research. Our long-term goal is to provide WCET analysis as a part of the standard tool chain for embedded development (together with compilers, debuggers, and simulators). This is facilitated by our cooperation with the embedded systems programming-tools vendor IAR Systems.

Static analysis tools that are used for worst-case execution time (WCET) analysis of real-time software just provide partial information on an analyzed program. Only the longest-executing path, which currently determines the WCET bound is indicated to the programmer. This limited view can prevent a programmer (or compiler) from targeting optimizations the right way. A possible resort is to use a metric that targets WCET and which can be efficiently computed for all code parts of a program. Similar to dynamic profiling techniques, which execute code with input that is typically expected for the application, based on WCET analysis we can indicate how critical a code fragment is, in relation to the worst-case bound. Computing such a metric on top of static analysis, incurs a certain overhead though, which increases with the complexity of the underlying WCET analysis. We present our approach to estimate the Criticality metric, by relaxing the precision of WCET analysis. Through this, we can reduce analysis time by orders of magnitude, while only introducing minor error. To evaluate our estimation approach and share our garnered experience using the metric, we evaluate real-time programs, which are considered as standard WCET benchmarks. We furthermore demonstrate how the visualization of a Criticality-based profile can aid in the understanding of a program's worst-case behavior.

Static Worst-Case Execution Time (WCET) analysis is a technique to derive upper bounds for the execution times of programs. Such bounds are crucial when designing and verifying real-time systems. WCET analysis needs a program flow analysis to derive constraints on the possible execution paths of the analysed program, like iteration bounds for loops and dependences between conditionals.Current WCET analysis tools typically obtain flow information through manual annotations. Better support for automatic flow analysis would eliminate much of the need for this laborious work. However, to automatically derive high-quality flow information is hard, and solution techniques with large time and space complexity are often required.In this paper we describe how to use program slicing to reduce the computational need of flow analysis methods. The slicing identifes statements and variables which are guaranteed not to influence the program flow. When these are removed, the calculation time of our different flow analyses decreases, in some cases considerably.We also show how program slicing can be used to identify the input variables and globals that control the outcome of a particular loop or conditional. This should be valuable aid when performing WCET analysis and systematic testing of large and complex real-time programs.

Static Worst-Case Execution Time (WCET) analysis is a technique to derive upper bounds for the execution times of programs. Such bounds are crucial when designing and verifying real-time systems. WCET analysis needs a program flow analysis to derive constraints on the possible execution paths of the analysed program, like iteration bounds for loops and dependences between conditionals.Current WCET analysis tools typically obtain flow information through manual annotations. Better support for automatic flow analysis would eliminate much of the need for this laborious work. However, to automatically derive high-quality flow information is hard, and solution techniques with large time and space complexity are often required.In this paper we describe how to use program slicing to reduce the computational need of flow analysis methods. The slicing identifes statements and variables which are guaranteed not to influence the program flow. When these are removed, the calculation time of our different flow analyses decreases, in some cases considerably.We also show how program slicing can be used to identify the input variables and globals that control the outcome of a particular loop or conditional. This should be valuable aid when performing WCET analysis and systematic testing of large and complex real-time programs.