Linux Kernel Module Research Articles

Decentralized Scheduling for Data-Parallel Tasks in the Cloud

For latency-sensitive data processing applications in the cloud, concurrent data-parallel tasks need to be scheduled and processed quickly. A data-parallel task usually consists of a set of sub-tasks, generating a set of flows that are collectively referred to as coflows . The state-of-the-art schedulers collect coflow information in the cloud to optimize coflow-level performance. However, most of the coflows, classified as small coflows because they consist of only short flows, have been largely overlooked. This paper presents OptaX , a decentralized network scheduling service that collaboratively schedules data-parallel tasks’ small coflows. OptaX adopts a cross-layer, COTS (commercial off-the-shelf) switch-compatible design that leverages the sendbuffer information in the kernel to adaptively optimize flow scheduling in the network. Specifically, OptaX (i) monitors the system calls (syscalls) in the hosts to obtain their sendbuffer footprints, and (ii) recognizes small coflows and assigns high priorities to their flows. OptaX transfers these flows in a FIFO manner by adjusting TCP’s two attributes: window size and round-trip time. We have implemented OptaX as a Linux kernel module. The evaluation shows that OptaX is at least 2.2 × faster than fair sharing and 1.2 × faster than only assigning small coflows with the highest priority. We further apply OptaX to improve the small I/O performance of Ursa , a distributed block storage system that provides virtual disks where small I/O is dominant. URSA with OptaX achieves significant improvement compared to the original URSA for small I/O latency.

An Online Learning Assisted Packet Scheduler for MPTCP in Mobile Networks

Multipath TCP is designed to utilize multiple network paths to achieve improved throughput and robustness against network failure. These features are supposed to make MPTCP preferable to single-path TCP in mobile networks. However, it fails to achieve the expected performance in practice. A key challenge of using MPTCP in mobile networks is how to effectively spread packets over heterogeneous and unstable network paths to mobile devices with limited buffers. If packets are not sent in an effective way, MPTCP may only provide equal or even lower throughput than single-path TCP. Several packet scheduling algorithms have been designed to tackle this challenge. Unfortunately, they still cannot achieve the expected performance in dynamic scenarios such as mobile networks. In this paper, we propose an Online-Learning Assisted Packet Scheduler (OLAPS) to solve the packet scheduling problem by modeling it as a multi-armed bandit problem. Over time, OLAPS can adaptively learn from current network conditions to make the best scheduling policy to provide the highest possible throughput in a dynamic environment. Moreover, when the inbuilt reward monitor detects the mismatch between network conditions and the learned policy, OLAPS aborts the outdated policy and switches to a new one swiftly. We implement OLAPS as a Linux kernel module and evaluate it over a wide range of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ns-3</i> -simulated network conditions. The results show that OLAPS retains MPTCP’s ability to provide higher throughput and also significantly improves the throughput performance of MPTCP when other in-kernel schedulers suffer a dramatic throughput decline.

Объединение одинаковых данных страниц оперативной памяти, сжатых модулем ядра Linux

Объединение одинаковых данных страниц оперативной памяти, сжатых модулем ядра Linux

An INT-based TCP window modulator for congestion control in data center networks

Modern data centers carry a huge number of diverse services and applications, which are developed by Transmission Control Protocol (TCP). The early TCP versions, such as Reno, were designed decades ago and not efficient enough to accommodate the data center environment, especially congestion control algorithms (CCAs). Data center TCP (DCTCP) was the classical CCA, which proposed for the data center networks (DCNs). All the conventional CCAs use packet loss and/or delay as congestion signals, however, these signals are not accuracy enough to conduct the CCAs to compute appropriate congestion window. In this paper, the In-band Network Telemetry (INT) data is used as congestion signal. The INT data can indicate the congestion location and level, which is more accuracy than end-to-end signals. Based on the advanced signal, an INT-based Receiver-driven TCP congestion window modulator, named IRTCP, is proposed and implemented as a loadable Linux kernel module, without any modification of the TCP/IP stack. Either in small-scale Mininet emulation or in large-scale NS3 simulation, the results show that IRTCP could provide significant performance improvement. The average FCT of IRTCP for mice flows is about 89.5%, 91.6%, 91.7% lower than Cubic-RED, Cubic-FIFO, and DCTCP. The average throughput of IRTCP for elephant flows is about 11.5%, 31%, 20.6% higher than DCTCP, Cubic-RED, Cubic-FIFO.

A STUDY ON LINUX KERNEL MODIFICATION FOR PERFORMANCE IMPROVEMENT OF SSD WEAR-OUT-AWARE SYSTEM

This study aims to improve the solid-state drive (SSD) wear-out-aware system by modifying the virtual file system (VFS) layer of the Linux kernel. Recently, according to the usage of big data and the development of artificial intelligence, the public can access it from a general PC rather than a research center. They installed several large-capacity SSDs required for learning individually or in a redundant array of inexpensive disks (RAID) 0 format on a PC to store the desired large amount of data. When multiple SSDs are connected and used, if one SSD suffers from wear-out, the input/output performance of other SSDs is adjusted to the corresponding device. Therefore, it is essential to ensure that wear out occurs evenly across multiple SSDs to prevent performance deterioration of the entire system. In this paper, we propose a method of distributing I/O to each SSD by modifying VFS layer of the Linux kernel to improve performance when the SSDs with the same performance but different performance due to wear-out. In other words, VFS was changed to detect each SSD wear out and distribute I/O to SSDs according to the performance ratio in a user-transparent way. Through the experimental results, when using the modified Linux kernel proposed in the paper, we found that the performance improved by 52% in the read and write functions compared to the original Linux kernel. Additionally, compared to RAID 0, it was found that there was a performance improvement of 26% in the read function and 16.7% in the write function. Keywords: Linux Kernel, Solid-State Drive, Redundant Array of Inexpensive Disks, Virtual File System DOI： https://doi.org/10.35741/issn.0258-2724.58.1.40

SealFSv2: combining storage-based and ratcheting for tamper-evident logging

Tamper-evident logging is paramount for forensic audits and accountability subsystems. It is based on a forward integrity model: upon intrusion, the attacker is not able to counterfeit the logging data generated before controlling the system. There are local and distributed solutions to this problem. Distributed solutions are suitable for common scenarios, albeit not appropriate for autonomous and loosely connected systems. Moreover, they can be complex and introduce new security issues. Traditional local tamper-evident logging systems use cryptographic ratchets. In previous works, we presented SealFS (from now on, SealFSv1), a system that follows a radically different approach for local tamper-evident logging based on keystream storage. In this paper, we present a new version, SealFSv2, which combines ratcheting and storage-based log anti-tamper protection. This new approach is flexible and enables the user to decide between complete theoretical security (like in SealFSv1) and partial linear degradation (like in a classical ratchet scheme), exchanging storage for computation with user-defined parameters to balance security and resource usage. We also describe an implementation of this scheme. This implementation, which showcases our approach, is an optimized evolution of the original sealfs Linux kernel module. It implements a stackable file system that enables transparent tamper-evident logging to all user space applications and provides instant deployability. Last, we present a complete performance evaluation of our current implementation and a fair performance comparison of the two opposite approaches for local tamper-evident logging (i.e., storage-based vs. ratcheting). This comparison suggests that, on current systems and general-purpose hardware, the storage-based approach and hybrid schemes perform better than the traditional ratchet approach.

Towards QoS-Based Embedded Machine Learning

Due to various breakthroughs and advancements in machine learning and computer architectures, machine learning models are beginning to proliferate through embedded platforms. Some of these machine learning models cover a range of applications including computer vision, speech recognition, healthcare efficiency, industrial IoT, robotics and many more. However, there is a critical limitation in implementing ML algorithms efficiently on embedded platforms: the computational and memory expense of many machine learning models can make them unsuitable in resource-constrained environments. Therefore, to efficiently implement these memory-intensive and computationally expensive algorithms in an embedded computing environment, innovative resource management techniques are required at the hardware, software and system levels. To this end, we present a novel quality-of-service based resource allocation scheme that uses feedback control to adjust compute resources dynamically to cope with the varying and unpredictable workloads of ML applications while still maintaining an acceptable level of service to the user. To evaluate the feasibility of our approach we implemented a feedback control scheduling simulator that was used to analyze our framework under various simulated workloads. We also implemented our framework as a Linux kernel module running on a virtual machine as well as a Raspberry Pi 4 single board computer. Results illustrate that our approach was able to maintain a sufficient level of service without overloading the processor as well as providing an energy savings of almost 20% as compared to the native resource management in Linux.

Aggregating Without Bloating: Hard Times for TCP on Wi-Fi

Since the definition of the bufferbloat phenomenon, several Linux kernel modules have been introduced in its TCP/IP stack, and there is a lack of experimental studies on their effects when coupled with WLAN technologies, in particular, IEEE 802.11n and IEEE 802.11ac. One essential algorithm introduced is named TCP Small Queues (TSQ) and has the role of limiting the number of packets that a TCP socket can enqueue in the stack, waiting for the physical layer to send the packets before enqueueing extra data. A second significant TCP algorithm is named TCP Pacing (TP) and regulates the pace used by the socket to enqueue packets in the stack, regulating the formation of bursts of data. These mechanisms affect the frame aggregation logic on WLAN networks and compromise the throughput-latency tradeoff of all the TCP variants. This paper presents an experimental evaluation of these techniques investigating the wireless network performance of several TCP congestion control variants under the presence of different TSQ and TP policies, modeling also their interaction.

Reducing Minor Page Fault Overheads through Enhanced Page Walker

Application virtual memory footprints are growing rapidly in all systems from servers down to smartphones. To address this growing demand, system integrators are incorporating ever larger amounts of main memory, warranting rethinking of memory management. In current systems, applications produce page fault exceptions whenever they access virtual memory regions that are not backed by a physical page. As application memory footprints grow, they induce more and more minor page faults. Handling of each minor page fault can take a few thousands of CPU cycles and blocks the application till the OS kernel finds a free physical frame. These page faults can be detrimental to the performance when their frequency of occurrence is high and spread across application runtime. Specifically, lazy allocation-induced minor page faults are increasingly impacting application performance. Our evaluation of several workloads indicates an overhead due to minor page faults as high as 29% of execution time. In this article, we propose to mitigate this problem through a hardware, software co-design approach. Specifically, we first propose to parallelize portions of the kernel page allocation to run ahead of fault time in a separate thread. Then we propose the Minor Fault Offload Engine (MFOE), a per-core hardware accelerator for minor fault handling. MFOE is equipped with a pre-allocated page frame table that it uses to service a page fault. On a page fault, MFOE quickly picks a pre-allocated page frame from this table, makes an entry for it in the TLB, and updates the page table entry to satisfy the page fault. The pre-allocation frame tables are periodically refreshed by a background kernel thread, which also updates the data structures in the kernel to account for the handled page faults. We evaluate this system in the gem5 architectural simulator with a modified Linux kernel running on top of simulated hardware containing the MFOE accelerator. Our results show that MFOE improves the average critical path fault handling latency by 33× and tail critical path latency by 51×. Among the evaluated applications, we observed an improvement of runtime by an average of 6.6%.

Network Policy Enforcement With Commodity Multiqueue NICs for Multitenant Data Centers

Data centers are the fundamental component in the Internet of Things (IoT) system architecture. Data center servers where IoT services are co-located require hierarchical network policy enforcement to ensure fair bandwidth sharing among tenants and to prioritize latency-sensitive traffic within a tenant simultaneously. Meanwhile, emerging network interface cards (NICs) in servers make use of multiple hardware queues to drive increasing line rates. Unfortunately, multiqueue NICs make it hard to enforce hierarchical policies because the NIC packet scheduler dequeues packets in a static round-robin (RR) fashion for per-flow fairness. In this article, we enable hierarchical network policy enforcement with existing commodity multiqueue NICs. We design TONIC, a multiqueue NIC packet scheduling solution that approximates hierarchical packet scheduling by manipulating the packet dequeueing sequence of the NIC scheduler through dynamic packet enqueueing decisions. Specifically, TONIC leverages multiple hardware queues and the double-ended queue structure of qdiscs to express different tenant weights and application priorities without hardware modifications. We implement a TONIC prototype as a Linux kernel module and evaluate it on a testbed with commodity multiqueue NICs. Our experiment results show that TONIC can enforce hierarchical policies consisting of weighted fair sharing and traffic prioritization while maintaining robustness to various network conditions.

Fault Injection for Linux Device Drivers on x86, MIPS and ARM Architectures

The security of device drivers should be verified by system testing. This paper presents QDFaultInjector, a fault injection tool that can work on three architectures (x86 64, AArch64 and MIPS64) to test Linux kernel modules (including device drivers) at runtime. Firstly, overall framework of QDFaultInjector is introduced in this paper, along with monitoring of LKM operation based on notifier chain and callback function and flexible mechanisms of retriggering faults and automatic recovering based on debugfs pseudo file system. Moreover, it is explained in detail how instruction replacement is implemented for fault injection on different architectures, because it is architecture-specific, i.e., it is related to the machine codes and the formats of instructions. Thereafter, a char device driver is used to verify every function of QDFaultInjector on the above architectures respectively and the results show that QDFaultInjector can inject faults successfully and effectively. In addition, a few of realistic network card drivers such as hns3 and pcnet32 are used to do the tests and some events of system breakdown occur when some faults that have been injected into device drivers by QDFaultInjector are triggered. That means that some potential problems are existed in the corresponding device drivers, which ought to be revised and perfected. Finally, QDFaultInjector is compared with some other fault injection tools and a conclusion is drawn that it supports more computer architectures and it is more flexible for testing operations.

Comparison of Fault Simulation Over Custom Kernel Module Using Various Techniques

To test the behavior of the Linux kernel module, device drivers and file system in a faulty situation, scientists tried to inject faults in different artificial environments. Since the rarity and unpredictability of such events are pretty high, thus the localization and detection of Linux kernel, device drivers, file system modules errors become unfathomable. ‘Artificial introduction of some random faults during normal tests’ is the only known approach to such mystifying problems. A standard method for performing such experiments is to generate synthetic faults and study the effects. Various fault injection frameworks have been analyzed over the Linux kernel to simulate such detection. The following paper highlights the comparison of different approaches and techniques used for such fault injection to test Linux kernel modules that include simulating low resource conditions and detecting memory leaks. The frameworks chosen to be used in these experiments are; Linux Text Project (LTP), KEDR, Linux Fault-Injection (LFI), and SCSI.

Reducing traffic burstiness for MPTCP in data center networks

Modern industrial and data center networks adopt multipath TCP (MPTCP) to achieve low communication latency. To make full use of multiple paths, MPTCP divides one TCP connection into multiple subflows. Due to the connection-level ACK blocking, however, MPTCP suffers from the traffic burstiness, which leads to large queueing delay, frequent packet drops, and even TCP timeout. To mitigate the traffic burstiness caused by ACK blocking, we propose MPTCP-RB (Multipath TCP with Reduced Burstiness), which interleaves consecutive packets across multiple subflows, instead of assigning consecutive packets to individual subflows. MPTCP-RB is implemented as a loadable Linux kernel module, working on any traditional MPTCP-based schemes without any other modifications on existing network stacks. The testbed and NS2 experiments show that, MPTCP-RB reduces the average flow completion time by up to 91% compared with the state-of-the-art MPTCP-based schemes.

Thread-Aware Mechanism to Enhance Inter-Node Load Balancing for Multithreaded Applications on NUMA Systems

NUMA multi-core systems divide system resources into several nodes. When an imbalance in the load between cores occurs, the kernel scheduler’s load balancing mechanism then migrates threads between cores or across NUMA nodes. Remote memory access is required for a thread to access memory on the previous node, which degrades performance. Threads to be migrated must be selected effectively and efficiently since the related operations run in the critical path of the kernel scheduler. This study focuses on improving inter-node load balancing for multithreaded applications. We propose a thread-aware selection policy that considers the distribution of threads on nodes for each thread group while migrating one thread for inter-node load balancing. The thread is selected for which its thread group has the least exclusive thread distribution, and thread members are distributed more evenly on nodes. This has less influence on data mapping and thread mapping for the thread group. We further devise several enhancements to eliminate superfluous evaluations for multithreaded processes, so the selection procedure is more efficient. The experimental results for the commonly used PARSEC 3.0 benchmark suite show that the modified Linux kernel with the proposed selection policy increases performance by 10.7% compared with the unmodified Linux kernel.

Observing the Invisible: Live Cache Inspection for High-Performance Embedded Systems

The vast majority of high-performance embedded systems implement multi-level CPU cache hierarchies. But the exact behavior of these CPU caches has historically been opaque to system designers. Absent expensive hardware debuggers, an understanding of cache makeup remains tenuous at best. This enduring opacity further obscures the complex interplay among applications and OS-level components, particularly as they compete for the allocation of cache resources. Notwithstanding the relegation of cache comprehension to proxies such as static cache analysis, performance counter-based profiling, and cache hierarchy simulations, the underpinnings of cache structure and evolution continue to elude software-centric solutions. In this paper, we explore a novel method of studying cache contents and their evolution via snapshotting. Our method complements extant approaches for cache profiling to better formulate, validate, and refine hypotheses on the behavior of modern caches. We leverage cache introspection interfaces provided by vendors to perform live cache inspections without the need for external hardware. We present CacheFlow, a proof-of-concept Linux kernel module which snapshots cache contents on an NVIDIA Tegra TX1 SoC (system on chip).

MDev-NVMe: Mediated Pass-Through NVMe Virtualization Solution With Adaptive Polling

The fast access to data and high parallel processing in high-performance computing instigates an urgent demand on the improvement of the NVMe storage within modern data centers. However, the former NVMe virtualization&#x2019;s unsatisfactory performance demonstrates that NVMe devices are often underutilized within cloud platforms. An NVMe virtualization mechanism with high performance and device sharing has captured researchers and developers&#x2019; attention. This article introduces MDev-NVMe, a new virtualization solution for NVMe storage device with (1) full NVMe storage virtualization for VMs running native NVMe driver, (2) a mediated pass-through mechanism for NVMe management, and (3) adaptive configuration of active polling optimization to simultaneously achieve high throughput, low latency performance, and substantial device scalability. We practically implement the MDev-NVMe as a Linux kernel module. This article subsequently evaluates MDev-NVMe with Intel OPTANE and P3600 SSD by comparing several mainstream NVMe virtualization mechanisms using application-level I/O benchmarks. MDev-NVMe with active polling can demonstrate a 142 percent improvement over native (interrupt-driven) throughput and over 2.5 &#x00D7; the <i>Virtio</i> throughput with only 70 percent native average latency and 31 percent <i>Virtio</i> average latency. Finally, the advantages of MDev-NVMe and the importance of adaptive polling are discussed, offering evidence that MDev-NVMe is a superior virtualization choice for cloud storage.

On Architectural Support for Instruction Set Randomization

Instruction Set Randomization (ISR) is able to protect against remote code injection attacks by randomizing the instruction set of each process. Thereby, even if an attacker succeeds to inject code, it will fail to execute on the randomized processor. The majority of existing ISR implementations is based on emulators and binary instrumentation tools that unfortunately: (i) incur significant runtime performance overheads, (ii) limit the ease of deployment, (iii) cannot protect the underlying operating system kernel, and (iv) are vulnerable to evasion attempts that bypass the ISR protection itself. To address these issues, we present the design and implementation of ASIST, an architecture with both hardware and operating system support for ISR. ASIST uses our extended SPARC processor that is mapped onto a FPGA board and runs our modified Linux kernel to support the new features. In particular, before executing a new user-level process, the operating system loads its randomization key into a newly defined register, and the modified processor decodes the process’s instructions with this key. Besides that, ASIST uses a separate randomization key for the operating system to protect the base system against attacks that exploit kernel vulnerabilities to run arbitrary code with elevated privileges. Our evaluation shows that ASIST can transparently protect both user-land applications and the operating system kernel from code injection and code reuse attacks, with about 1.5% runtime overhead when using simple encryption schemes, such as XOR and Transposition; more secure ciphers, such as AES, even though they are much more complicated for mapping them to hardware, they are still within acceptable margins,with approximately 10% runtime overhead, when efficiently leveraging the spatial locality of code through modern instruction cache configurations.

DALI LED Driver Control System for Lighting Operations Based on Raspberry Pi and Kernel Modules

Light emitting diodes (LEDs) as an efficient low-consumption lighting technology are being used increasingly in many applications. The move to LED lighting is also changing the way the lighting control systems are designed. Currently, most electronic ballasts and other digital lighting devices implement the Digital Addressable Lighting Interface (DALI) standard. This paper presents a low-cost, low-power effective DALI LED driver controller, based on open-source Raspberry Pi3 microcontroller prototyping platform. The control software is developed as a Linux kernel module under UBUNTU 18.04.2 LTS patched with PREEMPT_RT (Preemptive Real-time) for real-time processing. This dynamically loaded kernel module performs all the processing, communication and control operations of the Raspberry Pi3-based DALI controller with the DALI LED driver and LED luminaire. Software applications written in C and Python were developed for performance testing purposes. The experimental results showed that the proposed system could efficiently and effectively manage DALI LED drivers and perform lighting operations (e.g. dimming). The system can be used for a variety of purposes from personal lighting control needs and experimental research in control of electronic ballasts and other control gears, devices and sensors, to advanced requirements in professional buildings, including energy management, lighting maintenance and usage.

A Container-Based Technique to Improve Virtual Machine Migration in Cloud Computing

Hypervisor-enabled virtualization is a promising technology to deploy applications and implement Virtual Machine (VM) migration on cloud data centers. However, one critical aspect in hypervisor-based solution is that to deploy applications each VM is installed with a dedicated guest OS along with the requirement of binaries and library files and substantially large VM's image size. Therefore, it is a significant concern for data center administrators because it causes performance overheads. To solve these issues, a new paradigm of virtualization technology that has attracted considerable attention is containerization. Although, progress has been made to address container-based virtualization, there has been less attention to explore the issue of performance evaluation of container and VM in a migration environment. The main contribution of this research study is to develop an experimental setup to implement and evaluate the performance of LXD/CR container-based migration technique and compared it with the pre-copy VM migration scheme. To conduct this performance evaluation, first we modified Linux kernel and implemented LXD which uses liblxc to launch container, then migration of the running container is facilitated using a checkpoint/restore mechanism of CRIU technique. To ensure the applicability and validity for real-time cloud infrastructure, we executed a wide variety of benchmarks such as web server, CPU-intensive, disk I/O and database. From the results obtained, it is found that compared to pre-copy VM migration scheme, LXD/CR container migration technique reduces the downtime, migration time, amount of data transferred and the number of pages transferred by 75.66%, 65.55%, 76.63%, and 76.78%, respectively. In addition, the migration overhead caused is reduced by 55.89% w.r.t. CPU utilization and 76.52% w.r.t. RAM utilization. This being an important contribution for migrating the running applications efficiently and consequently increase the business value of the cloud infrastructure providers.

Augmenting Operating Systems with OpenCL Accelerators

Heterogeneous computing leverages more than one kind of processors to boost the performance of user-space applications with the heterogeneous programming languages, e.g., OpenCL. While some works have been done to accelerate the computations required by Linux kernel software, they are either application-specific solutions or tightly coupled with the certain computing platforms and are not able to support the general-purpose in-kernel accelerations using different types of processors. In this article, the general-purpose software framework called Kernel acceleration with OpenCL (KOCL), is proposed to tackle the problem. KOCL exposes a set of the high-level programming interfaces for the Linux kernel module developers to offload compute-intensive tasks on different hardware accelerators without managing and coordinating the platform-specific computing and memory resources. The simplified programming efforts are achieved by the developed platform management and memory models, which provide a systematic means of managing the heterogeneous hardware resources. In addition, the one- and zero-copy data-buffering schemes are offered by KOCL, so that the offloaded tasks deliver high performance on the platforms with different memory architectures. We have developed the prototype system to accelerate the Network-Attached Storage server applications. Significant performance improvements are achieved with the three different types of accelerators, i.e., the multicore processor, the integrated GPU, and the discrete GPU, respectively. We believe that KOCL is useful for the design of embedded appliances to evaluate the performance of design alternatives.