Hardware-assisted Virtualization Research Articles

Hardware-Software Collaborative Tiered-Memory Management Framework for Virtualization

The tiered-memory system can effectively expand the memory capacity for virtual machines (VMs). However, virtualization introduces new challenges specifically in enforcing performance isolation, minimizing context switching, and providing resource overcommit. None of the state-of-the-art designs consider virtualization and address these challenges; we observe that a VM with tiered memory incurs up to a 2× slowdown compared to a DRAM-only VM. We propose vTMM , a hardware-software collaborative tiered-memory management framework for virtualization. A key insight in vTMM is to leverage the unique system features in virtualization to meet the above challenges. vTMM automatically determines page hotness and migrates pages between fast and slow memory to achieve better performance. Specially, vTMM optimizes page tracking and migration based on page-modification logging (PML), a hardware-assisted virtualization mechanism, and adaptively distinguishes hot/cold pages through the page “temperature” sorting. vTMM also dynamically adjusts fast memory among multi-VMs on demand by using a memory pool. Further, vTMM tracks huge pages at regular-page granularity in hardware and splits/merges pages in software, realizing hybrid-grained page management and optimization. We implement and evaluate vTMM with single-grained page management on an Intel processor, and the hybrid-grained page management on a Sunway processor with hardware mode supporting hardware/software co-designs. Experiments show that vTMM outperforms existing tiered-memory management designs in virtualization.

Designing Robust API Monitoring Solutions

Tracing the sequence of library calls and system calls that a program makes is very helpful to characterize its interactions with the surrounding environment and, ultimately, its semantics. However, due to the entanglements of real-world software stacks, accomplishing this task can be surprisingly challenging as we take accuracy, reliability, and transparency into the equation. In this article, we identify six challenges that API monitoring solutions should overcome in order to manage these dimensions effectively and outline actionable design points for building robust API tracers that can be used even for security research. We then detail and evaluate SNIPER, an open-source API tracing system available in two variants based on dynamic binary instrumentation (for simplified in-guest deployment) and hardware-assisted virtualization (realizing the first general user-space tracer of this kind), respectively.

SmartFVM: A Fast, Flexible, and Scalable Hardware-based Virtualization for Commodity Storage Devices

A computational storage device incorporating a computation unit inside or near its storage unit is a highly promising technology to maximize a storage server’s performance. However, to apply such computational storage devices and take their full potential in virtualized environments, server architects must resolve a fundamental challenge: cost-effective virtualization . This critical challenge can be directly addressed by the following questions: (1) how to virtualize two different hardware units (i.e., computation and storage), and (2) how to integrate them to construct virtual computational storage devices, and (3) how to provide them to users. However, the existing methods for computational storage virtualization severely suffer from their low performance and high costs due to the lack of hardware-assisted virtualization support. In this work, we propose SmartFVM-Engine , an FPGA card designed to maximize the performance and cost-effectiveness of computational storage virtualization. SmartFVM-Engine introduces three key ideas to achieve the design goals. First, it achieves high virtualization performance by applying hardware-assisted virtualization to both computation and storage units. Second, it further improves the performance by applying hardware-assisted resource orchestration for the virtualized units. Third, it achieves high cost-effectiveness by dynamically constructing and scheduling virtual computational storage devices. To the best of our knowledge, this is the first work to implement a hardware-assisted virtualization mechanism for modern computational storage devices.

Toward an Analysable, Scalable, Energy-Efficient I/O Virtualization for Mixed-Criticality Systems

In mixed-criticality systems (MCSs), timely handling of I/O operations is a key for the system being successfully implemented and appropriately functioned. The I/O system for an MCS must simultaneously enable different features, including isolation/separation, timing-predictability, performance, scalability, and energy-efficiency. Moreover, such an I/O system also requires to manage I/O resource in an adaptive manner to facilitate efficient yet safe resource sharing among components of different criticality levels. Existing approaches cannot achieve all of these requirements simultaneously. This article presents a mixed-criticality I/O management framework, termed MCS-IOV. MCS-IOV is based on hardware-assisted virtualization, which provides temporal and spatial isolation and prohibits fault propagation with limited extra overhead. MCS-IOV extends a real-time I/O virtualization system, by supporting the concept of mixed criticalities and customized interfaces for schedulers, which offers good timing predictability and scalability. Finally, we introduce an energy management framework for MCS-IOV, ensuring the power-efficiency of the design. The MCS-IOV is the first systematical solution that fulfills all the requirements as a mixed-criticality I/O system.

Virtualization Overhead of Multithreading in X86 State-of-the-Art & Remaining Challenges

Despite great advancements in hardware-assisted virtualization of the x86 architecture, certain workloads still suffer significant overhead. This article dissects said overhead in the context of multi-threading. We describe the state-of-the-art, pinpoint challenges, and suggest improvements, aiming to provide a valuable reference to developers and users of virtualization systems alike. We study the virtualization overhead of the PARSEC and SPLASH2X multithreaded benchmarks in a variety of scenarios using a state-of-the-art system. Through controlled experiments, source code analysis and literature review, we quantify the virtualization overhead multithreading still induces and link it to its root causes, after which we suggest possible mitigation strategies. Multithreading still induces high virtualization overhead, mainly caused by synchronization, spinning at user level and NUMA management. The overhead is diverse in nature and embodiment as it is a function of many system and workload properties. System-level solutions are feasible, but often imply difficult trade-offs. Systematic workload optimization is a promising alternative.

Detecting Hardware-Assisted Virtualization With Inconspicuous Features

Recent years have witnessed the proliferation of the deployment of virtualization techniques. Virtualization is designed to be transparent, that is, unprivileged users should not be able to detect whether a system is virtualized. Such detection can result in serious security threats such as evading virtual machine (VM)-based malware dynamic analysis and exploiting vulnerabilities for cross-VM attacks. The traditional software-based virtualization leaves numerous artifacts/fingerprints, which can be exploited without much effort to detect the virtualization. In contrast, current mainstream hardware-assisted virtualization significantly enhances the virtualization transparency, making itself more transparent and difficult to be detected. Nonetheless, we showcase three new identified low-level inconspicuous features, which can be leveraged by an unprivileged adversary to effectively and stealthily detect the hardware-assisted virtualization. All three features come from the chipset fingerprints, rather than the traces of software-based virtualization implementations (e.g., Xen or KVM). The identified features include i) Translation-Lookaside Buffer (TLB) stores an extra layer of address translations; ii) Last-Level Cache (LLC) caches one more layer of page-table entries; and iii) Level-1 Data (L1D) Cache is unstable. Based on the above features, we develop three corresponding virtualization detection techniques, which are then comprehensively evaluated on three native environments and three popular cloud providers: i) Amazon Elastic Compute Cloud, ii) Google Compute Engine and iii) Microsoft Azure. Experimental results validate that these three adversarial detection techniques are effective (with no false positive) and stealthy (without triggering suspicious system events, e.g., VM-exit ) in detecting the above commodity virtualized environments.

PrOS: Light-Weight Privatized Se cure OSes in ARM TrustZone

TrustZone is a hardware security technique in ARM mobile devices. Using TrustZone, software components running within the secure world can be completely isolated from the normal world, which ensures hardware-enforced security access control over the underlying computing resources. In order to support multiple trusted applications, TrustZone runs its own operating system, called the secure OS, within the secure world. Unfortunately, attackers have been exploiting privilege escalation vulnerabilities in a secure OS, as reported in most of major secure OSes from product vendors including Samsung, Huawei, and Qualcomm. More critically, as all trusted applications are running on the same secure OS instance, compromising the secure OS leads to compromising all trusted applications, rendering the secure OS as a single point of failure endangering the entire TrustZone's security. This paper presents PrOS, our mechanism to privatize secure OSes through direct virtualization of TrustZone. PrOS allows each trusted application to run with its own secure OS such that the secure OS is no longer a single point of security failure. One particular challenge for PrOS lies in how efficiently to implement software-only virtualization for TrustZone for a practical deployment in real systems despite the condition that the current ARM architectures do not support hardware-assisted virtualization for TrustZone. As opposed to the common belief that software-only virtualization is inefficient and sluggish, we have found several common design features inherent in the secure OS to leverage for optimally tailoring the TrustZone virtualization scheme. We implemented PrOS on a 64-bit ARM development board. According to our evaluation, PrOS incurs 0.02 and 1.18 percent performance overheads on average in the normal and secure worlds, respectively, demonstrating its effectiveness in the field.

Demystifying Arm TrustZone

The world is undergoing an unprecedented technological transformation, evolving into a state where ubiquitous Internet-enabled “things” will be able to generate and share large amounts of security- and privacy-sensitive data. To cope with the security threats that are thus foreseeable, system designers can find in Arm TrustZone hardware technology a most valuable resource. TrustZone is a System-on-Chip and CPU system-wide security solution, available on today’s Arm application processors and present in the new generation Arm microcontrollers, which are expected to dominate the market of smart “things.” Although this technology has remained relatively underground since its inception in 2004, over the past years, numerous initiatives have significantly advanced the state of the art involving Arm TrustZone. Motivated by this revival of interest, this paper presents an in-depth study of TrustZone technology. We provide a comprehensive survey of relevant work from academia and industry, presenting existing systems into two main areas, namely, Trusted Execution Environments and hardware-assisted virtualization. Furthermore, we analyze the most relevant weaknesses of existing systems and propose new research directions within the realm of tiniest devices and the Internet of Things, which we believe to have potential to yield high-impact contributions in the future.

GPUCloudSim: an extension of CloudSim for modeling and simulation of GPUs in cloud data centers

Recent years have witnessed an increasing growth in the usage of GPUs in cloud data centers. It is known that conventional virtualization techniques are not directly applicable to GPUs, making it a challenge to effectively take advantage of virtualization benefits. API remoting, full, para and hardware-assisted virtualization methods are adopted to empower VMs with GPU capabilities. With such a diversity in approaches, there is a need for a simulation environment to study the effectiveness of GPU virtualization techniques and evaluate GPU provisioning and scheduling policies in cloud data centers. In order to model and simulate GPU-enabled VMs in cloud data centers, this work proposes and describes a simulator architecture implemented as an extension of CloudSim. The extension eases up conducting experimental studies that otherwise need to be carried out in real cloud infrastructures. It includes models to simulate interference among co-running applications, the overhead of virtualization and power consumption of GPUs. To demonstrate the usefulness of our extension, we study NVIDIA GRID, a hardware-assisted GPU virtualization solution. We show that for situations where the number of VMs outperforms the number of hosts, the first-fit VM placement of VMware Horizon may not be effective. Instead, we suggest a first-fit increasing VM placement algorithm which increases the acceptance rate by 59%, shortens makespan by 25% and saves energy by 21%.

Leveraging Hardware-Assisted Virtualization for Deterministic Replay on Commodity Multi-Core Processors

Deterministic replay, which provides the ability to travel backward in time and reconstruct the past execution flow of a multiprocessor system, has many prominent applications. Prior research in this area can be classified into two categories: hardware-only schemes and software-only schemes. While hardware-only schemes deliver high performance, they require significant modifications to the existing hardware. In contrast, software-only schemes work on commodity hardware, but suffer from excessive performance overhead and huge logs. In this article, we present the design and implementation of a novel system, Samsara, which uses the hardware-assisted virtualization (HAV) extensions to achieve efficient deterministic replay without requiring any hardware modification. Unlike prior software schemes which trace every single memory access to record interleaving, Samsara leverages HAV on commodity processors to track the read-set and write-set for implementing a chunk-based recording scheme in software. By doing so, we avoid all memory access detections, which is a major source of overhead in prior works. Evaluation results show that compared with prior software-only schemes, Samsara significantly reduces the log file size to 1/70th on average, and further reduces the recording overhead from about $10 \times$ , reported by state-of-the-art works, to $2.1 \times$ on average.

Native Simulation of MPSoC Using Hardware-Assisted Virtualization

Integration of multiple heterogeneous processors into a single system-on-a-chip is a clear trend in embedded devices. Designing and verifying these devices requires high-speed and easy-to-build simulation platforms. Among the software simulation approaches, native simulation is a good candidate since the embedded software is executed natively on the host machine, and no instruction set simulator development effort is necessary. However, existing native simulation approaches are such that the simulated software shares the memory space of the modeled hardware modules and the host operating system, making impractical the support of legacy code running on the target platform. To overcome this issue seldom mentioned in the literature, we propose the addition of a transparent address space translation layer to separate the target address space from the host simulator one. For this, we exploit the hardware-assisted virtualization technology now available on most general-purpose processors. Experiments show that this solution does not degrade the native simulation speed, while keeping the ability to accomplish software performance evaluation.

Optimizing virtual machines using hybrid virtualization

Minimizing virtualization overhead and improving the reliability of virtual machines are challenging when establishing virtual machine cluster. Paravirtualization and hardware-assisted virtualization are two mainstream solutions for modern system virtualization. Hardware-assisted virtualization is superior in CPU and memory virtualization and becoming the leading solution, yet paravirtualization is still valuable in some aspects as it is capable of shortening the disposal path of I/O virtualization. Thus we propose the hybrid virtualization which runs the paravirtualized guest in the hardware-assisted virtual machine container to take advantage of both. Experiment results indicate that our hybrid solution outweighs origin paravirtualization by nearly 30% in memory intensive test and 50% in microbenchmarks. Meanwhile, compared with the origin hardware-assisted virtual machine, hybrid guest owns over 16% improvement in I/O intensive workloads.

Perfctr-Xen

Virtualization is a powerful technique used for variety of application domains, including emerging cloud environments that provide access to virtual machines as a service. Because of the interaction of virtual machines with multiple underlying software and hardware layers, the analysis of the performance of applications running in virtualized environments has been difficult. Moreover, performance analysis tools commonly used in native environments were not available in virtualized environments, a gap which our work closes. This paper discusses the challenges of performance monitoring inherent to virtualized environments and introduces a technique to virtualize access to low-level performance counters on a per-thread basis. The technique was implemented in perfctr-xen, a framework for the Xen hypervisor that provides an infrastructure for higher-level profilers. This framework supports both accumulative event counts and interrupt-driven event sampling. It is light-weight, providing direct user mode access to logical counter values. perfctr-xen supports multiple modes of virtualization, including paravirtualization and hardware-assisted virtualization. perfctr-xen applies guest kernel-hypervisor coordination techniques to reduce virtualization overhead. We present experimental results based on microbenchmarks and SPEC CPU2006 macrobenchmarks that show the accuracy and usability of the obtained measurements when compared to native execution.

Enforcing kernel constraints by hardware-assisted virtualization

This article deals with kernel security protection. We propose a characterization of malicious kernel-targeted actions, based on how the way they act to corrupt the kernel. Then, we discuss security measures able to counter such attacks. We finally expose our approach based on hardware-virtualization that is partially implemented into our demonstrator Hytux, which is inspired from bluepill (Rutkowska in subverting vista kernel for fun and profit. In: Black Hat in Las Vegas, 2006), a malware that installs itself as a lightweight hypervisor--on a hardware-virtualization compliant CPU--and puts a running Microsoft Windows Operating System into a virtual machine. However, in contrast with bluepill, Hytux is a lightweight hypervisor that implements protection mechanisms in a more privileged mode than the Linux kernel.

A comparison of software and hardware techniques for x86 virtualization

Until recently, the x86 architecture has not permitted classical trap-and-emulate virtualization. Virtual Machine Monitors for x86, such as VMware ® Workstation and Virtual PC, have instead used binary translation of the guest kernel code. However, both Intel and AMD have now introduced architectural extensions to support classical virtualization.We compare an existing software VMM with a new VMM designed for the emerging hardware support. Surprisingly, the hardware VMM often suffers lower performance than the pure software VMM. To determine why, we study architecture-level events such as page table updates, context switches and I/O, and find their costs vastly different among native, software VMM and hardware VMM execution.We find that the hardware support fails to provide an unambiguous performance advantage for two primary reasons: first, it offers no support for MMU virtualization; second, it fails to co-exist with existing software techniques for MMU virtualization. We look ahead to emerging techniques for addressing this MMU virtualization problem in the context of hardware-assisted virtualization.

A comparison of software and hardware techniques for x86 virtualization

Until recently, the x86 architecture has not permitted classical trap-and-emulate virtualization. Virtual Machine Monitors for x86, such as VMware ® Workstation and Virtual PC, have instead used binary translation of the guest kernel code. However, both Intel and AMD have now introduced architectural extensions to support classical virtualization.We compare an existing software VMM with a new VMM designed for the emerging hardware support. Surprisingly, the hardware VMM often suffers lower performance than the pure software VMM. To determine why, we study architecture-level events such as page table updates, context switches and I/O, and find their costs vastly different among native, software VMM and hardware VMM execution.We find that the hardware support fails to provide an unambiguous performance advantage for two primary reasons: first, it offers no support for MMU virtualization; second, it fails to co-exist with existing software techniques for MMU virtualization. We look ahead to emerging techniques for addressing this MMU virtualization problem in the context of hardware-assisted virtualization.

A comparison of software and hardware techniques for x86 virtualization

Until recently, the x86 architecture has not permitted classical trap-and-emulate virtualization. Virtual Machine Monitors for x86, such as VMware ® Workstation and Virtual PC, have instead used binary translation of the guest kernel code. However, both Intel and AMD have now introduced architectural extensions to support classical virtualization.We compare an existing software VMM with a new VMM designed for the emerging hardware support. Surprisingly, the hardware VMM often suffers lower performance than the pure software VMM. To determine why, we study architecture-level events such as page table updates, context switches and I/O, and find their costs vastly different among native, software VMM and hardware VMM execution.We find that the hardware support fails to provide an unambiguous performance advantage for two primary reasons: first, it offers no support for MMU virtualization; second, it fails to co-exist with existing software techniques for MMU virtualization. We look ahead to emerging techniques for addressing this MMU virtualization problem in the context of hardware-assisted virtualization.