General-purpose Operating System Research Articles

Toward Linux-based safety-critical systems—Execution time variability analysis of Linux system calls

Modern transportation and industrial domain safety-critical applications, such as autonomous vehicles and collaborative robots, exhibit a combination of escalating software complexity and the need to integrate diverse software stacks and machine learning algorithms, consequently demanding complex high-performance hardware. Linux’s extensive platform support and library ecosystem make it a valuable general-purpose operating system for developing complex software systems. However, because the Linux kernel has not been designed to comply with safety standards, it has a high execution path variability and does not provide execution time guarantees. In this context, several research initiatives have studied the usage of Linux for developing complex safety-related systems, focusing on topics that include its development process, isolation architectures, or test coverage estimation. Nonetheless, execution-time analysis and providing temporal guarantees is still a challenge. This work extends the novel statistical analysis of Linux system call execution paths with the analysis of execution-time variability and proposes a method for estimating the worst-case execution time, forming a sound approach for an in-depth analysis of the Linux kernel execution paths and execution times for safety-related systems. The proposed method is applied to a representative use case that implements an Autonomous Emergency Brake application in an NVIDIA Jetson Nano board connected to the CARLA autonomous driving simulator.

Interrupt Latency Accurate Measurement in Multiprocessing Embedded Systems by Means of a Dedicated Circuit

Modern multiprocessing embedded applications require, in many cases, two different environments on the same platform: one that meets real-time requirements and another one with a general purpose operating system. Although several technologies can be used, two of the most popular are virtualization based on hypervisors and asymmetric multiprocessing software. However, using these tools introduces latency, which must be measured to verify compliance with real-time requirements. With the aim of facilitating these measurements, this work provides a hardware tool that is more precise and easier to use than other existing software solutions. The paper also studies the interrupt latency generated by different hypervisors and asymmetric multiprocessing frameworks in a Zynq UltraScale+ platform. This research work facilitates the accurate study of the temporal response of multiprocessor embedded systems, which allows for evaluating their suitability for applications with real-time requirements.

A Lightweight File System Design for Unikernel

Unikernels are specialized operating system (OS) kernels optimized for a single application or service, offering advantages such as rapid boot times, high performance, minimal memory usage, and enhanced security compared to general-purpose OS kernels. Unikernel applications must remain compatible with the runtime environment of general-purpose kernels, either through binary or source compatibility. As a result, many Unikernel projects have prioritized system call compatibility over performance enhancements. In this paper, we explore the design principles of Unikernel file systems and introduce a new file system tailored for Unikernels named ULFS (Ultra Lightweight File System). ULFS provides system call services akin to those of general-purpose OS kernels but achieves superior performance and security with significantly fewer system resources. Specifically, ULFS is developed as a lightweight file system embracing Unikernel design principles. It streamlines system calls, removes unnecessary locks, and omits permission checks for multiple users, utilizing a non-hypervisor architecture. This approach significantly reduces the memory footprint of the file system and enhances performance. Through measurement studies, we assess the performance and memory requirements of various file systems from major Unikernel projects. Our findings demonstrate that ULFS surpasses several existing Unikernel file systems, including Rumpvfs, Ramfs-u, Ramfs-q, 9pfs, and Hcfs.

Latency Reduction and Packet Synchronization in Low-Resource Devices Connected by DDS Networks in Autonomous UAVs.

Real-time flight controllers are becoming dependent on general-purpose operating systems, as the modularity and complexity of guidance, navigation, and control systems and algorithms increases. The non-deterministic nature of operating systems creates a critical weakness in the development of motion control systems for robotic platforms due to the random delays introduced by operating systems and communication networks. The high-speed operation and sensitive dynamics of UAVs demand fast and near-deterministic communication between the sensors, companion computer, and flight control unit (FCU) in order to achieve the required performance. In this paper, we present a method to assess communications latency between a companion computer and an RTOS open-source flight controller, which is based on an XRCE-DDS bridge between clients hosted in the low-resource environment and the DDS network used by ROS2. A comparison based on the measured statistics of latency illustrates the advantages of XRCE-DDS compared to the standard communication method based on MAVROS-MAVLink. More importantly, an algorithm to estimate latency offset and clock skew based on an exponential moving average filter is presented, providing a tool for latency estimation and correction that can be used by developers to improve synchronization of processes that rely on timely communication between the FCU and companion computer, such as synchronization of lower-level sensor data at the higher-level layer. This addresses the challenges introduced in GNC applications by the non-deterministic nature of general-purpose operating systems and the inherent limitations of standard flight controller hardware.

Design and implementation of a real-time simulation platform for embedded applications on general-purpose operating systems

In recent years, the number of invested resources adopted in experiments of embedded applications dropped significantly as many simulation technologies are widely used. However, the efficiency of simulations is seriously influenced by some expensive and difficult-to-obtain devices. It is urgent and of great significance to build a universal simulation platform for embedded applications on general-purpose operating systems with an objective of improving the efficiency and effectiveness of system development and implementation. Since virtualization technology can greatly enhance the simulation efficiency by providing virtual models to simulate the behaviors of real devices, this paper designs and realizes a real-time simulation platform on general-purpose operating systems with the virtualization technology such that embedded applications would be correctly and efficiently debugged and tested on the general-purpose operating systems. The proposed simulation platform contains four layers named hardware resource, virtualization, virtual runtime environment, and interface adaptation, allowing dynamic debugging and testing of embedded applications without requiring the actual presence of real devices. Experiments are conducted to verify the functionalities of the proposed simulation platform, and results demonstrate that the proposed simulation platform can meet the real-time and high reliability requirements of embedded applications.

Towards a Tractable Exact Test for Global Multiprocessor Fixed Priority Scheduling

Scheduling algorithms are called “global” if they can migrate tasks between cores. Global scheduling algorithms are the de-facto standard practice for general purpose Operating Systems, to balance the workload between cores. However, the exact schedulability analysis of real-time applications for these algorithms is proven to be weakly NP-hard. Despite such a hardness, the research community keeps investigating the methods for an exact schedulability analysis for its relevance and to tightly estimate the execution requirements of real-time systems. Due to the NP-hardness, the available exact tests are very time and memory demanding even for sets of a few tasks. On another hand, the available sufficient tests are very pessimistic, despite consuming less resources. Motivated by these observations, we propose an exact schedulability test for constrained-deadline sporadic tasks under global multiprocessor fixed-priority scheduling scheduler, which is significantly faster and consumes less memory, compared to any other available exact test. To derive a faster test, we exploit the idea of a state-space pruning, aiming at reducing the number of feasible system states to be examined by the test. The resulted test is multiple orders of magnitude faster with respect to other state-of-the-art exact tests. Our C++ implementation is publicly available.

Improving Data Reliability through Assured Digital Signing

Digital signatures are an important mechanism for ensuring data trustworthiness via source authenticity, integrity, and source non-repudiation. However, their trustworthiness guarantee can be subverted in the real world by sophisticated attacks, which can obtain cryptographically legitimate digital signatures without actually compromising the private signing key. This problem cannot be adequately addressed by a purely cryptographic approach, by the revocation mechanism of Public Key Infrastructure (PKI) because it may take a long time to detect the compromise, or by using tamper-resistant hardware because the attacker does not need to compromise the hardware. This problem will become increasingly more important and evident because of stealthy malware (or Advanced Persistent Threats). In this paper, we propose a novel solution, dubbed Assured Digital Signing (ADS), to enhancing the data trustworthiness vouched by digital signatures. In order to minimize the modifications to the Trusted Computing Base (TCB), ADS simultaneously takes advantage of trusted computing and virtualization technologies. Specifically, ADS allows a signature verifier to examine not only a signature’s cryptographic validity but also its system security validity that the private signing key and the signing function are secure, despite the powerful attack that the signing application program and the general-purpose Operating System (OS) kernel are malicious. The modular design of ADS makes it application-transparent (i.e., no need to modify the application source code in order to deploy it) and almost hypervisor-independent (i.e., it can be implemented with any Type I hypervisor). To demonstrate the feasibility of ADS, we report the implementation and analysis of an Xen-based ADS system.

Real-Time Performance Analysis and Tuning of Embedded System Virtualization Architecture Based on KVM

Embedded systems are proceeding towards exploiting virtualization technology to have the benefits of Real-Time Operating System (RTOS) and General-Purpose Operating System (GPOS) in the same system. This combination provides both a timely and deterministic behavior and a general-purpose application codebase. There still exist concerns about the real-time responsiveness of RTOS running inside a Virtual Machine (VM). In this paper, the real-time performance of Kernel-based Virtual Machine (KVM) virtualization architecture is analyzed on a multi-core system. Here, a preemptible Linux kernel with the PREEMPT_RT patch is used for RTOS, while a standard Linux kernel is used for GPOS. The interrupt latency inside the real-time guest VM is analyzed by applying various amounts of CPU, memory, and I/O stresses on the guest and host systems. A VM resource monitoring tool ‘VM_stat’ is developed to know the resource usage of the guest VMs, which is useful for effectively tuning the system. Different real-time tuning measures are applied on the host/guest systems and the performance is analyzed.

Modeling Speedup in Multi-OS Environments

For workloads that place strenuous demands on system software, novel operating system designs like unikernels, library OSes, and hybrid runtimes offer a promising path forward. However, while these systems can outperform general-purpose OSes, they have limited ability to support legacy applications. Multi-OS environments, where the application’s execution is split between a control plane and a data plane operating system, can address this challenge, but reasoning about the performance of applications that run in such a split execution environment is currently guided only by expert intuition and empirical analysis. As the level of specialization in system software and hardware continues to increase, there is both a pressing need and ripe opportunity for investigating analytical models that can predict application performance and guide programmers’ intuition when considering multi-OS environments. In this paper we present such a model to place bounds on application speedup, beginning with a simple, intuitive formulation, and progressing to a more refined model. We present an analysis of the model for a diverse set of benchmarks, as well as a prototype tool to project multi-OS speedups for applications on existing systems. Finally, we validate our model on state-of-the-art multi-OS systems, demonstrating that it reliably predicts speedup with 96% average accuracy.

Performance analysis of real-time and general-purpose operating systems for path planning of the multi-robot systems

In general, modern operating systems can be divided into two essential parts, real-time operating systems (RTOS) and general-purpose operating systems (GPOS). The main difference between GPOS and RTOS is the system istime-critical or not. It means that; in GPOS, a high-priority thread cannot preempt a kernel call. But, in RTOS, a low-priority task is preempted by a high-priority task if necessary, even if it’s executing a kernel call. Most Linux distributions can be used as both GPOS and RTOS with kernel modifications. In this study, two Linux distributions, Ubuntu and Pardus, were analyzed and their performances were compared both as GPOS and RTOS for path planning of the multi-robot systems. Robot groups with different numbers of members were used to perform the path tracking tasks using both Ubuntu and Pardus as GPOS and RTOS. In this way, both the performance of two different Linux distributions in robotic applications were observed and compared in two forms, GPOS, and RTOS.

The Kingsguard OS-level mitigation against cache side-channel attacks using runtime detection

Most of the mitigation techniques against access-driven cache side-channel attacks (CSCAs) are not very effective. This is mainly because most mitigation techniques usually protect against any given specific vulnerability of the system and do not take a system-wide approach. Moreover, they either completely remove or greatly reduce the performance benefits. Therefore, to find a security vs performance trade-off, we argue in favor of need-based protection in this paper, which will allow the operating system to apply mitigation only after successful detection of CSCAs. Thus, detection can serve as a first line of defense against such attacks. In this work, we propose a novel OS-level runtime detection-based mitigation mechanism, called the Kingsguard, against CSCAs in general-purpose operating systems. The proposed mechanism enhances the security and privacy capabilities of Linux as a proof of concept, and it can be widely used in commodity systems without any hardware modifications. We provide experimental validation by mitigating three state-of-the-art CSCAs on two different cryptosystems running under Linux. We have also provided results by analyzing the effect of the combination of multiple attacks running concurrently under variable system noise. Our results show that the Kingsguard can detect and mitigate known CSCAs with an accuracy of more than 99% and 95%, respectively.

Postmortem Analysis of Baget RTOS Processes

Despite the best efforts of programmers to create high-quality software, some errors inevitably escape even the most rigorous testing process and are first encountered by end users of the software. When this happens, developers need to quickly understand the reasons for the errors that occurred and eliminate them. Back in 1951, at the dawn of modern computing, Stanley Gill wrote that special attention should be paid to those errors that occur after the program is started, and lead to its termination. Gill is considered the founder of the so-called postmortem debugging, when a program or system is modified to record its state at the time of failure, so that the programmer can later understand what happened and why such a situation occurred. Since then, postmortem debugging technology has been used in many different systems, including all major general-purpose operating systems (OS), as well as specialized OS such as embedded systems and real-time systems. To ensure the high level of reliability expected from such critical systems, it is necessary, on the one hand, to implement the possibility of rapid recovery of the system or its part after a failure. On the other hand, it is necessary to provide a mechanism for storing as much information as possible after each failure, so that the cause of its occurrence can be determined later. To understand the real potential of postmortem debugging tools, we will first consider the current state of debugging methods and the role of postmortem analysis tools, as well as the requirements for postmortem debugging tools for critical systems. Next, we will describe the mechanism of postmortem debugging implemented by the authors in the RTOS Baget and formulate tasks for further development.

Application of Smartwatches in Elderly Care with Indoor Localization Functionality

The recent achievements in mobile technology and wearable OS makes possible to create comfortably wearable and very capable smartwatches. They have many different sensors and powerful hardware combined with general purpose OS and all this available for reasonable price. It makes it ideal device for elderly care. Monitoring the elderly’s basic health condition is very straightforward, but using smartwatch as an indoor localization device, monitoring the motion activity, recognizing the typical motion patterns of wandering is not simple. Even those watches are really capable devices, they are not equipped with direct indoor localization sensors and we would like to avoid installing special equipment’s, markers, transmitters in the home of elderly. Using only a commercially available smartwatch hardware for indoor localization is a challenging task, several filtering and data processing algorithms needs to be combined in order to provide acceptable indoor localization function. The algorithms, their connection and fine-tuning methods are explained in this article.

Design and implementation of a virtual ARINC 653 simulation platform

The ARINC 653 operation system is currently widely adopted in the avionics industry, and has become the mainstream architecture in avionics applications because of its strong agility and reliability. Although ARINC 653 can efficiently reduce the weight and energy consumption, it results in a serious development and verification problem for avionics systems. As ARINC 653 is non-open source software and lacks effective support for software testing and debugging, it is of great significance to build a real-time simulation platform for ARINC 653 on general-purpose operating systems, improving the efficiency and effectiveness of system development and implementation. In this paper, a virtual ARINC 653 platform is designed and realized by using real-time simulation technology. The proposed platform is composed of partition management, communication management, and health monitoring management, provides the same operation interfaces as the ARINC 653 system, and allows dynamic debugging of avionics applications without requiring the actual presence of real devices. Experimental results show that the platform not only simulates the functionalities of ARINC 653, but also meets the real-time requirements of avionics applications.

Statistical Test Coverage for Linux-Based Next-Generation Autonomous Safety-Related Systems

Autonomous systems represent a significant leap forward in the ongoing technological evolution of dependable and safety-related systems, integrating features such as artificial intelligence, high-performance computing devices, General Purpose Operating Systems (GPOS) (e.g., GNU/Linux) and security requirements. Nonetheless, traditionally employed safety techniques and measures were not defined for safety-related systems with such features. Consequently, the need to research new methods and measures emerges in order to be able to achieve appropriate safety assurance. In this manuscript, we explore the limitations of traditional test coverage techniques, and we provide two complementary methods to pave the way towards the testing of Linux-based complex safety-related systems. The methods, which are based on statistical analyses, are presented and applied to a Linux-based Autonomous Emergency Braking (AEB) case study, specifically focusing on the kernel execution path test coverage.

Програмно визначені системи зберігання даних

Every year the amount of generated data grows exponentially which entails an increase in both the number and capacity of data storage systems. The highest capacity is required for data storage systems that are used to store backups and archives, file storages with shared access, testing and development environments, virtual machine storages, corporate or public web services. To solve such tasks, nowadays manufacturers offer three types of storage systems: block and file storages which have already become a standard used for implementing IT infrastructures, and software-defined storage systems. They allow to create data storages on non-specialized equipment, such as a group of x86-64 server nodes managed by general-purpose operating systems. The main feature of software-defined data storages is the transfer of storage functions from the hardware level to the software level where these storage functions are defined not by physical features of the hardware but by the software selected for specific tasks solving. Today there are three main singled out technologies characterized by scalable architecture that allow to in-crease efficiency and storage volume through adding new nodes to a single pool: Ceph, DELL EMC VxFlex OS, HP StoreVirtual VSA. Software-defined data storages have the following advantages: fault tolerance, efficiency, flexibility and economy. Utilization of software-defined storages allows to increase efficiency of IT infrastructure and reduce its maintenance costs; to build a hybrid infrastructure that would allow to use internal and external cloud resources; to increase efficiency of both services and us-ers by providing reliable connection by using the most convenient devices; to build a portal as a single point of services and resources control.

A Performance-Stable NUMA Management Scheme for Linux-Based HPC Systems

Linux is becoming the de-facto standard operating system for today’s high-performance computing (HPC) systems because it can satisfy the demands of many HPC systems for rich operating system (OS) features. However, owing to features intended for the general-purpose OS, Linux has many OS noise sources such as page faults or thread migrations that can result in the unstable performance of HPC application. Furthermore, in the case of the non-uniform memory access (NUMA) architecture, which has different memory access latencies to local and remote memory nodes, the performance stability of the application can be more exacerbated by the OS noise. In this paper, we address the OS noise caused by Linux in the NUMA architecture and propose a novel performance-stable NUMA management scheme called Stable-NUMA . Stable-NUMA comprises three techniques for improving performance stability: two-level thread clustering, state-based page placement, and selective page profiling. Our proposed Stable-NUMA scheme significantly alleviates OS noise and enhances the local memory access ratio of the NUMA system as compared to Linux. We implemented Stable-NUMA in Linux and experimented with various HPC workloads. The evaluation results demonstrated that Stable-NUMA outperforms Linux with and without its NUMA-aware feature by up to 25% in terms of average performance and 73% in terms of performance stability.

PacketUsher: Exploiting DPDK to accelerate compute-intensive packet processing

Many compute-intensive network applications such as application-layer traffic generator, Deep Packet Inspection(DPI) and web servers are widely deployed on commodity PC for the reason of flexibility and cheap price. However, how to improve their performance on general purpose OS is challenging due to the high packet I/O related overheads. This paper presents PacketUsher, a high-performance packets processing framework to remove these performance bottlenecks. In building PacketUsher, we constructed a DPDK wrapper as the underlying packet I/O engine to accelerate packet transmission, and utilized the strategies of zero copy, batch processing and parallelism to improve packet processing. Through RFC2544 benchmark, we demonstrate that DPDK wrapper has excellent packets transmission capability. As a case study of PacketUsher, we design and implement a commercial application-layer traffic generator. The experiment results show that the FPS (Flow Per Second) value of our traffic generator over PacketUsher is more than 4 times of that over standard Linux platform. By comparison, the FPS value over PacketUsher is about 3 times of that over existing methods (Netmap and PF_RING).

Build real-time communication for hybrid dual-OS system

Abstract In automotive/avionic/robot systems, consolidating multiple domains on a powerful SoC is an important trend. From the perspectives of safety and security, each of these domains must execute independently with a self-governed Operating System (OS). Such systems are termed as hybrid multi-OS systems. Generally, in a commonly used hybrid dual-OS system, a real-time OS (RTOS) and a general-purpose OS (GPOS) are integrated to satisfy different requirements. Inter-OS communication is the key to such systems. Specifically, services are often requested by tasks in RTOS from GPOS, but it is challenging to satisfy the requirements on time efficiency and predictability in RTOS, due to the uncertainty in communication maintenance and the time-sharing policy of the GPOS. Current existing approaches can not provide predictable time guarantees to the circumstances. This paper aims to build a time predictable and secure RPC mechanism (i.e., the primary and critical communication unit in a hybrid multi-OS system). On the foundation of a detailed analysis of the time delays in each phase of a RPC from RTOS to GPOS, we point out the drawbacks in conventional RPC mechanisms and propose an effective model and some methods to guarantee the predictability in all the phases of RPC. Furthermore, a real-time RPC scheme (termed RTRG-RPC) is proposed and applied to our ready-built TrustZone-based consolidation system (i.e., TZDKS). RTRG-RPC achieves accurate time control through three mechanisms: SGI (Software Generate Interrupt) message transforming, interrupt handler RPC servicing, and priority-swapping. Evaluations show that RTRG-RPC can achieve real-time predictability and managed priority inversion. Moreover, the design of RTRG-RPC also provides lightweight and essential protection on the security, without violation of the efficiency.

METHOD FOR ASSESSING OF RELIABILITY CHARACTERISTICS IN DESIGNING OF FAILURERESISTANT REAL-TIME OPERATING SYSTEMS

For many years, real-time OS-based applications have been used in embedded special-purpose systems. Recently they have been used everywhere, from on-board control systems for aircraft, to household appliances. The development of multiprocessor computing systems usually aims to increase either the level of reliability or the level of system performance to values that are inaccessible or difficult to implement in traditional computer systems. In the first case, the question of the availability of special means of ensuring the fault tolerance of computer systems arises, the main feature (and advantage) of which is the absence of any single resource, failure of which leads to a fatal failure of the entire system. The use of a real-time operating system is always associated with equipment, with an object and with events occurring at an object. A real-time system, as a hardware-software complex, includes sensors that record events at an object, input / output modules that convert sensor readings into a digital form suitable for processing these readings on a computer, and finally, a computer with a program that responds to events occurring at the facility. The RTOS is focused on processing external events. It is this that leads to fundamental differences (compared with general-purpose OS) in the structure of the system as well as in the functions of the kernel and in the construction of the input-output system. The RTOS can be similar in its user interface to general-purpose operating systems, but it is completely different in its structure. In addition, the use of RTOS is always specific. If users (not developers) usually perceive a general-purpose OS as a ready-made set of applications, then the RTOS serves only as a tool for creating a specific hardware-software complex in real time. Therefore, the widest class of users of RTOS is the developers of real-time complexes, people designing control and data collection systems. When designing and developing a specific real-time system, the programmer always knows exactly what events can occur at the facility, and he knows the critical terms for servicing each of these events. We call a real-time system (SRV) a hardware-software complex that responds in predictable times to an unpredictable stream of external events. The system must have time to respond to the event that occurred at the facility, during the time critical for this event. The critical time for each event is determined by the object and by the event itself, and, of course, it can be different, but the response time of the system must be predicted (calculated) when creating the system. Lack of response at the predicted time is considered an error for real-time systems. The system must have time to respond to simultaneously occurring events. Even if two or more external events occur simultaneously, the system must have time to respond to each of them during time intervals critical for these events. In this study, as part of a network fault-tolerant technology, the RTOS becomes a special type of control software that is used to organize the operation of embedded applications, which are characterized by limited memory resources, low productivity and the requirements of a guaranteed response time (T<4 μs), high level availability and availability of auto-monitoring facilities.