Mobile Virtual Reality Devices Research Articles

A 92 fps and 2.56 mJ/Frame Computing-In-Memory-Based Human Pose Estimation Accelerator With Resource-Efficient Macro for Mobile Devices

Human pose estimation (HPE) is a promising solution for accurately understanding the state and context of human actions in virtual reality (VR). A high frame rate with low-power HPE processing is required for a realistic user interaction experience in battery-limited mobile devices. The proposed HPE accelerator is a computing-in-memory (CIM) based accelerator that computes depth-wise separable convolution (DWSC) of a lightweight HPE network. Three key features contribute to a resource-efficient CIM accelerator: 1) Dual-mode CIM computes DWSC with a reconfigurable homogenous architecture, resulting in 2.68 times higher throughput than previous analog CIMs. 2) Effective layeraware unrolling performs bit-parallel computation on dualmode CIM with fewer ADC operations, achieving 46 times higher throughput than before. 3) Adaptive fused inter-macro balancing improves latency balance in layer fusion execution, leading to a 57.0 % higher frame rate than before. The proposed HPE accelerator is implemented in 28nm CMOS technology. It achieves higher computation resource utilization and operates HPE with a low energy-delay product of 27.6 uJ7s in mobile VR devices.

Wireless Multiplayer Interactive Virtual Reality Game Systems With Edge Computing: Modeling and Optimization

Wireless multiplayer interactive virtual reality (VR) game has the high computing workload of VR and unpredictable interaction among players, which brings severe challenges to the design of wireless communication systems. In this paper, we propose a wireless multiplayer interactive VR game transmission framework based on mobile edge computing (MEC) that is able to model the interaction among players and compute the post-processing procedures at the MEC server or the mobile VR device. In the framework, the absolute delay of each player is used to avoid VR vertigo and the inter-player delay among players is used to model the fairness of the interactive game. Aiming to minimize the average inter-player delay, we optimize the computing resource allocation of the MEC server, the wireless bandwidth allocation and the post-processing decision policy subject to the constraints of the absolute delay requirements, the local energy limits of players, the total bandwidth limit and the computing resources limit. To tackle the non-convex problem efficiently, we design an iterative algorithm based on the NESTT-G algorithm which iteratively optimizes the truncated first-order Taylor approximation of the objective. Numerical results demonstrate the proposed algorithm can reduce the average inter-player delay significantly with lower complexity, and also reveal the impact of different parameters and the channel state conditions on the post-processing decision and edge resource allocation.

Digital reconstruction of Elmina Castle for mobile virtual reality via point-based detail transfer

Reconstructing 3D models from large, dense point clouds is critical to enable Virtual Reality (VR) as a platform for entertainment, education, and heritage preservation. Existing 3D reconstruction systems inevitably make trade-offs between three conflicting goals: the efficiency of reconstruction (e.g., time and memory requirements), the visual quality of the constructed scene, and the rendering speed on the VR device. This paper proposes a reconstruction system that simultaneously meets all three goals. The key idea is to avoid the resource-demanding process of reconstructing a high-polygon mesh altogether. Instead, we propose to directly transfer details from the original point cloud to a low polygon mesh, which significantly reduces the reconstruction time and cost, preserves the scene details, and enables real-time rendering on mobile VR devices. While our technique is general, we demonstrate it in reconstructing cultural heritage sites. We for the first time digitally reconstruct the Elmina Castle, a UNESCO world heritage site at Ghana, from billions of laser-scanned points. The reconstruction process executes on low-end desktop systems without requiring high processing power, making it accessible to the broad community. The reconstructed scenes render on Oculus Go in 60 FPS, providing a real-time VR experience with high visual quality.

QoE Evaluation Methods for 360-Degree VR Video Transmission

Virtual reality (VR) videos, especially 360-degree VR videos, have attracted substantial research interests with the explosion of mobile VR devices in recent years. In this work, we propose a VR quality of experience (QoE) evaluation framework including online, offline and mixed scenarios that can meet the requirement of real applications. Under this framework, we further develop a subjective evaluation method and an objective QoE evaluation model based on network transmission parameters. A VR panoramic video database is established with nine different contents. This database contains 468 different samples coming from nine original contents. Then, in the subjective evaluation part, we propose to adopt four dimensions, i.e., quality score, immersion score, non-spinning sensation score and global score for the subjective evaluation purpose. These dimensions can meet the interaction and immersion features of the VR video service. In the objective evaluation part, an improved two-step neural network (INN) model is developed by using the features of the physiological psychology and cognitive neurology. This model captures the intrinsic link between the initial input network parameters and the final perception in VR transmission. Our results suggest that the proposed INN can achieve a higher pearson correlation coefficients than the other methods. Especially, it increases by 8.9% than the traditional NN method. Our work can make up for the shortage of traditional evaluation methods in term of the VR QoE subjective and objective evaluation in transmission scopes.

Communications, Caching, and Computing for Mobile Virtual Reality: Modeling and Tradeoff

Virtual reality (VR) over wireless is emerging as an important use case of 5G networks. Immersive VR experience requires the delivery of huge data at ultra-low latency, thus demanding ultra-high transmission rate. This challenge can be largely addressed by the recent network architecture known as mobile edge computing (MEC), which enables caching and computing capabilities at the edge of wireless networks. This paper presents a novel MEC-based mobile VR delivery framework that is able to cache parts of the field of views (FOVs) in advance and run certain post-processing procedures at the mobile VR device. To optimize resource allocation at the mobile VR device, we formulate a joint caching and computing decision problem to minimize the average required transmission rate while meeting a given latency constraint. When FOVs are homogeneous, we obtain a closed-form expression for the optimal joint policy which reveals interesting communications-caching-computing tradeoffs. When FOVs are heterogeneous, we obtain a local optima of the problem by transforming it into a linearly constrained indefinite quadratic problem then applying concave convex procedure. Numerical results demonstrate great promises of the proposed mobile VR delivery framework in saving communication bandwidth while meeting low latency requirement.

Joint Radio Communication, Caching, and Computing Design for Mobile Virtual Reality Delivery in Fog Radio Access Networks

The emerging virtual reality (VR) experience demands ultra-high-transmission-rate and ultra-low-latency deliveries, which is challenging for the current cellular networks. Since fog radio access networks (F-RANs) take full advantages of both edge fog computing and caching technologies and benefit different quality-of-service requirements, it is anticipated that high-quality VR experience could be well addressed in F-RANs. This paper presents an F-RAN-based mobile VR delivery framework, in which the core idea is to cache parts of the VR videos in advance and run a certain processing procedure at the edge of F-RANs. To optimize resource allocation at both mobile VR devices and fog access points (F-APs), a joint radio communication, caching and computing decision problem is formulated to maximize the average tolerant delay with meeting a given transmission rate constraint. This problem is formulated as a multiple choice multiple dimensional knapsack problem and solved with the Lagrangian dual decomposition approach. Furthermore, the optimal joint caching and computing decision is analyzed in a specific case with a closed-form expression of the average tolerant delay. The communications-caching-computing tradeoff at both mobile VR devices and F-APs is revealed, and the numerical results demonstrate that local caching and computing capabilities have significant impacts on the average tolerant delay. The proposed mobile VR delivery framework is promising in improving spectral efficiency by maximizing average tolerant delay while meeting high transmission rate requirements.

An effective FoV restriction approach to mitigate VR sickness on mobile devices

AbstractIt has been noted that restricting field of view (FoV) is effective to mitigate virtual reality (VR) sickness, but it is prone to reduce the immersiveness. We propose an effective FoV restriction approach to mitigate VR sickness on mobile devices. The proposed approach includes a practical and reliable method for measuring VR sickness and a method of effectively limiting the FoV on mobile VR devices. Experimental result shows that the proposed method reduces VR sickness by 31.4%. The method was successfully applied to a VR application.

Communication-Constrained Mobile Edge Computing Systems for Wireless Virtual Reality: Scheduling and Tradeoff

Mobile edge computing (MEC) is expected to be an effective solution to deliver virtual reality (VR) videos over wireless networks. In contrast to previous computation-constrained MEC, which reduces the computation-resource consumption at the mobile device by increasing the communication-resource consumption, we develop a communications-constrained MEC framework to reduce communication-resource consumption by fully exploiting the computation and caching resources at the mobile VR device in this paper. Specifically, according to a task modularization, the MEC server only delivers the components which have not been stored in the VR device, and then the VR device uses the received components and other cached components to construct the task, yielding low communication cost but high delay. The MEC server also computes the task by itself to reduce the delay, however, it consumes more communication-resource due to the delivery of entire task. Therefore, we propose a task scheduling strategy to decide which computation model should the MEC server operates to minimize the communication-resource consumption under the delay constraint. Finally, the tradeoffs among communications, computing, and caching are also discussed, and we analytically find that given a target communication-resource consumption, the transmission rate is inversely proportional to the computing ability of mobile VR device.

QoE-Aware Mobile VR HAS Cache Management With Coding Helper

Virtual reality (VR) video has attracted substantial research interests with the explosion of mobile VR devices in recent years. In this paper, we propose a novel cache management architecture with coding helper to meet the VR’s high bit rate requirement and the multi-rate feature in the HTTP adaptive streaming (HAS) transmission. The coding helper has not only storage capacity just as traditional caching helper, but also video coding function to encode the VR video into different rates. In this new architecture of VR HAS cache management, we consider the problem that how to distribute the bit rate of different VR videos in a certain cache size by taking into account the quality of experience (QoE) in different situations. In order to decrease the computing time of VR video service and guarantee a high level QoE as well, we propose a new maximum QoE increase (MQI) cache management method to obtain the solution of the optimal problem in multi-video and multi-cache (MVMC) situation. Furthermore, we propose a scheduling algorithm of coding helper to decrease the backhaul traffic and develop four different request scenarios in the experiments. Simulation results are finally provided to show that the coding helper scheme has about 16%–100% gain of usefulness than the traditional caching helper scheme. The proposed MQI cache allocation algorithm has more than 50% decreasing of computing time and almost has the same QoE compared with the exhaustive search. The framework can provide useful guidelines to distribute rate of VR video in cache over wireless networks and get better QoE.

Computing Offloading Over mmWave for Mobile VR: Make 360 Video Streaming Alive

Compared with previous video streaming technologies, 360 video streaming is an emerging technology in commercial platforms that promise greater immersive video service. The use of 360 video streaming requires larger bandwidth and rapid responsiveness to users. In addition, mobile virtual-reality (VR) devices locally process video decoding, post-processing, and rendering. However, its performance is insufficient to stream high-resolution videos such as 4k–8k. Therefore, we propose an adaptive computing offloading scheme using millimeter wave (mmWave) communication. This offloading scheme helps mobile device sharing video decoding tasks to a powerful GPU-based PC. It allows the mobile VR device enhancing the capability of playing high-resolution videos. The mmWave 802.11ad wireless technology promises the use of high-bandwidth wireless communication to improve the capacity of multimedia systems. However, the current studies on video transmission over 802.11ad mmWave are specifically focused on supporting the outdoor environment. Furthermore, synchronization is an essential issue for ensuring the quality of service (QoS) of 360 video transmission. Thus, the proposed scheme concentrates on two parts: 1) 360 video streaming in an indoor environment, and offloading mechanism using mmWave 802.11ad 60-GHz wireless communication to offload video coding tasks and 2) adaptive video synchronization mechanisms, and high-efficiency video coding (HEVC) using scalable HEVC extension for 360 videos with equirectangular projection. The experiments prove that the proposed scheme provides high performance 360 video streaming in QoS-sensitive video streaming applications.