Immersive Media Research Articles

Demonstrating Immersive Media Delivery on 5G Broadcast and Multicast Testing Networks

Demonstrating Immersive Media Delivery on 5G Broadcast and Multicast Testing Networks

Virtual reality and augmented reality in social learning spaces: a literature review

In this survey, we explore Virtual Reality and Augmented Reality within social learning spaces, such as classrooms and museums, while also extending into relevant social interaction concepts found within more reality-based and social immersive media frameworks. To provide a foundation for our findings we explore properties and interactions relevant to educational use in social learning spaces; in addition to several learning theories such as constructivism, social cognitive theory, connectivism, and activity theory, within a CSCL lens, to build a theoretical foundation for future virtual reality/augmented reality educational frameworks. Several virtual reality/augmented reality examples for learning are explored, and several promising areas to further research, such as a greater focus on accessibility, the interplay between the physical and virtual environments, and suggestions for updated learning theory foundations, are proposed.

Real-time non-photorealistic animation for immersive storytelling in “Age of Sail”

Immersive media such as virtual and augmented reality pose some interesting new challenges for non-photorealistic animation: we must not only balance the screen-space rules of a 2D visual style against 3D motion coherence, but also account for stereo spatialization and interactive camera movement, at a rate of 90 frames per second. We introduce two new real-time rendering techniques: MetaTexture, an example-based texturing method that adheres to the movement of 3D geometry while preserving the texture’s screen-space characteristics, and Edge Breakup, a method for roughening edges by warping with structured noise. We also describe a custom rendering pipeline featuring art-directable coloring, shadow filtering, and texture indication, and our approach to animating and rendering a painterly ocean in real time. We show how we have used these techniques to achieve the “moving illustration” style of the real-time immersive short film “Age of Sail”.

실감형미디어기반 불교의 열반 체험 VR콘텐츠 융합연구

실감형미디어기반 불교의 열반 체험 VR콘텐츠 융합연구

Sound design inducing attention in the context of audiovisual immersive environments

Sound design has been a fundamental component of audiovisual storytelling in linear media. However, with recent technological developments and the shift towards non-linear and immersive media, things are rapidly changing. More sensory information is available and, at the same time, the user is gaining agency upon the narrative, being offered the possibility of navigating or making other decisions. These new characteristics of immersive environments bring new challenges to storytelling in interactive narratives and require new strategies and techniques for audiovisual narrative progression. Can technology offer an immersive environment where the user has the sensation of agency, of choice, where her actions are not mediated by evident controls but subliminally induced in a way that it is ensured that a narrative is being followed? Can sound be a subliminal element that induces attentional focus on the most relevant elements for the narrative, inducing storytelling and biasing search in an immersive non-linear audiovisual environment? Herein, we present a literature review that has been guided by this prospect. With these questions in view, we present our exploration process in finding possible answers and potential solution paths. We point out that consistency, in terms of coherency across sensory modalities and emotional matching may be a critical aspect. Finally, we consider that this review may open up new paths for experimental studies that could, in the future, provide new strategies in the practice of sound design in the context of non-linear media.

Augmented Reality and Virtual Reality Applications Based on Satellite-Borne and ISS-Borne Remote Sensing Data for School Lessons

Facing global challenges, a qualified education in remote sensing technologies needs to start in school to sensitise teachers and thus young people for ecological issues and develop their technological skills. Remote sensing is part of the STEM (Science, Technology, Engineering and Mathematics) curricular topics, all of which are either a requirement for or benefit from remote sensing. However, implementing its data and methods into regular curricula poses technological and educational challenges, given the lack of appropriate school IT infrastructure and remote sensing education for aspiring STEM teachers. Immersive media can overcome these structural issues, using the teachers’ and pupils’ smartphones to display and interact, e.g. with video data, astronaut photographs, or complex hyperspectral data. The data needs to be pre-processed for Augmented and Virtual Reality applications (AR and VR, resp.) to fit topic, available devices and software. Game development software allows for easy integration in apps for AR and VR, but does not contain functions for remote sensing methods. These are either implemented directly in scripts, worked around using less computing-intensive methods or replaced by methods unique to augmented and virtual reality. Following these premises, five AR apps are presented in this report, teaching about tropical cyclones, anthropogenic desertification, energy consumption, gravitation, and algal blooms, all of which use remote sensing data from different sensors aboard the ISS. One VR app teaching about Mount Fuji was developed using a DEM derived from astronaut photographs. All have different levels of interactivity and are embedded in worksheets that fill a double period.

실감미디어 기반의 사용자경험 특징과 의미에 관한 융합연구

오늘날 고도화된 정보통신의 발전과 스마트 미디어산업의 성장으로 실감미디어(Immersive Media)는 사용자에게 현실과 가장 근접한 가상의 환경을 제공하고 있다.<BR> 이에 본 연구는 실감미디어 환경에서 다차원적으로 변화하고 있는 사용자경험을 살펴보기 위해 시작되었다.<BR> 본 연구의 목적은 먼저 급격히 변화하고 있는 실감미디어 생태계를 이해하고, 그 속에서 다양한 감각의 형태로 경험을 전달하고 있는 실감미디어 기반의 사용자경험 특징과 의미를 파악하는 것에 목적을 두고 있다. 따라서 본 논문의 연구 방법은 실감미디어의 생태계와 기술적인 부분을 다학제적으로 이해하고, 사용자경험의 개념과 요소를 살펴본다. 이를 바탕으로 경험의 특징과 의미를 파악하기 위한 분석의 툴로 8개의 키워드를 유형화하였다. 이를 토대로 각 사례에서 나타나는 사용자경험의 과정과 가치, 변화 및 속성을 파악하여 분석을 시도하였다.<BR> 연구 결과 및 내용은 다음과 같다. 첫째, 물리적·심리적 한계로부터의 극복이다. 때때로 오프라인 또는 온라인 환경에서 사용자경험에 단절이 생긴다. 이때 실감미디어와의 기술적 결합으로 시각적 자극과 감각적 체험을 융복합한 가상의 공간에서 경험의 한계를 극복할 수 있다. 둘째, 경험이 몰입도 높은 체험으로 진화하고 있다. 스마트 기기를 사용해 사용자는 직접몸을 움직이며, 신체 일부분을 사용해 자신이 원하는 것을 선택해 체험할 수 있다. 셋째, 개인화된 서비스이다. 사용자는 개인화된 가상의 공간에서 자신의 취향과 라이프스타일이 반영된 제품을 선택하고, 원하는 시간과 장소에서 서비스를 경험할 수 있으며, 미디어와의 상호작용으로 개인 맞춤형 디지털 콘텐츠를 받을 수 있다.<BR> 이러한 연구 결과를 바탕으로 앞으로 더욱 활성화될 실감미디어 기반의 사용자경험이 물리적 한계를 넘어서 사용자의 다양한 니즈가 반영된 경험을 설계하는 데 있어 중요한 시사점이 되길 바란다.

Demonstrating Immersive Media Delivery on 5G Broadcast and Multicast Testing Networks

This work presents eight demonstrators and one showcase developed within the 5G-Xcast project. They experimentally demonstrate and validate key technical enablers for the future of media delivery, associated with multicast and broadcast communication capabilities in 5th Generation (5G). In 5G-Xcast, three existing testbeds: IRT in Munich (Germany), 5GIC in Surrey (UK), and TUAS in Turku (Finland), have been developed into 5G broadcast and multicast testing networks, which enables us to demonstrate our vision of a converged 5G infrastructure with fixed and mobile accesses and terrestrial broadcast, delivering immersive audio-visual media content. Built upon the improved testing networks, the demonstrators and showcase developed in 5G-Xcast show the impact of the technology developed in the project. Our demonstrations predominantly cover use cases belonging to two verticals: Media & Entertainment and Public Warning, which are future 5G scenarios relevant to multicast and broadcast delivery. In this paper, we present the development of these demonstrators, the showcase, and the testbeds. We also provide key findings from the experiments and demonstrations, which not only validate the technical solutions developed in the project, but also illustrate the potential technical impact of these solutions for broadcasters, content providers, operators, and other industries interested in the future immersive media delivery.

A location‐aware network virtualization and reconfiguration for 5G core network based on SDN and NFV

SummaryIn 5G networks, mobile network and service functions run as software on general‐purpose servers that are distributed across remote sites. Dedicated networks, such as Internet of Things (IoT) network, disaster network, and immersive media network, are virtualized as software solutions on common hardware equipment. This means a new network can be easily provisioned by modifying the configurations of the network based on software‐defined networking (SDN) and network functions virtualization (NFV). Furthermore, the 5G core network architecture toward flat architecture with distributed cloud is a key solution for providing service flexibility and scalability in order to handle an explosive increase in mobile data traffic. To create heterogeneous networks and services on 5G networks, we propose a location‐aware network virtualization method that aims at minimizing the network load and resource cost by appropriately selecting the relevant cloud (including Virtual Machines (VMs), virtual network function [VNF] nodes, applications, and resources) and mobile core entities to provide the required virtual networking services that facilitate secure isolation of the virtual networks with automated resource convergence. In addition, we propose a selective mobility management scheme for service continuity in cases of user equipment (UE) handover. The numerical results show the trade‐off between the different cost factors, ie, network cost and service resources cost in terms of the number of deployed clouds on 5G core networks.

Design methodology for 360\xb0 immersive video applications: the case study of a cultural heritage virtual tour

Three hundred sixty–degree (360°) immersive video applications for Head Mounted Display (HMD) devices offer great potential in providing engaging forms of experiential media solutions especially in Cultural Heritage education. Design challenges emerge though by this new kind of immersive media due to the 2D form of resources used for their construction, the lack of depth, the limited interaction and the need to address the sense of presence. In addition, the use of Virtual Reality (VR) headsets often causes nausea, or motion sickness effects imposing further implications in moderate motion design tasks. This paper introduces a methodological categorisation of tasks and techniques for the design of 360° immersive video applications. Following the design approach presented, a testbed application has been created as an immersive interactive virtual tour at the historical centre of the city of Rethymno in Crete, Greece, which has undergone user trials. Based on the analysis of the results of this study, a set of design guidelines for the implementation of 360° immersive video virtual tours is proposed.

Bringing People Closer: The Prosocial Effects of Immersive Media on Users’ Attitudes and Behavior

This experimental study examined how varying the degree of immersiveness of a short documentary about a remote health issue influenced users’ reported spatial presence, empathic parasocial interaction, and individual issue involvement. Higher-order responses, namely, the desire for information and willingness to donate to the cause, were also analyzed. The documentary was shown to 85 participants using three different technologies with varying degrees of immersiveness (high, moderate, and low). The results show that the level of the technology’s immersiveness gradually increases the spatial presence, empathic parasocial interaction, and issue involvement of the user. While participants of the highly immersive condition reported a higher desire for additional information, the results on donation behavior were less conclusive.

Overview of current development in depth map coding of 3D video and its future

3D videos have attracted attention from academia and industry after great success in the film industry. Multiview video plus depth (MVD) is the most popular 3D video format to provide vivid 3D feeling and has been adopted as an international 3D video coding standard, namely 3D extension of high efficiency video coding (HEVC). MVD includes a limited number of textures and depth maps to synthesise virtual views. In MVD, depth samples describe the distance between a camera and an actual object as a grey-level image. The characteristics of depth maps are quite different from texture images. Consequently, new coding tools are designed for depth maps in 3D-HEVC to improve the coding efficiency at the expense of high-computational complexity, which faces great challenges in coding systems. Depth map coding is also an important technique in immersive media to support three degrees of freedom 3DoF+/6DoF applications such as virtual reality/augmented reality. The study starts with an overview of what has been done over the last decade in 3D-HEVC, especially depth map coding, regarding theories, methodologies, current research and state-of-the-art fast approaches. Following this, a comprehensive comparison of the reviewed techniques is presented, and an outlook on its future trends is provided.

Quality Evaluation of 3D Objects in Mixed Reality For Different Lighting Conditions

This paper presents a study on Quality of Experience (QoE) evaluation of 3D objects in Mixed Reality (MR) scenarios. In particular, a subjective test was performed with Microsoft HoloLens, considering different degradations affecting the geometry and texture of the content. Apart from the analysis of the perceptual effects of these artifacts, given the need for recommendations for subjective assessment of immersive media, this study was also aimed at: 1) checking the appropriateness of a single stimulus methodology (ACR-HR) for these scenarios where observers have less references than with traditional media, and 2) analyzing the possible impact of environment lighting conditions on the quality evaluation of 3D objects in mixed reality (MR), and 3) benchmark state-of-the-art objective metrics in this context. The subjective results provide insights for recommendations for subjective testing in MR/AR, showing that ACR-HR can be used in similar QoE tests and reflecting the influence among the lighting conditions, the content characteristics, and the type of degradations. The objective results show an acceptable performance of perceptual metrics for geometry quantization artifacts and point out the need of further research on metrics covering both geometry and texture compression degradations.

LectureCast as 360 Degree Video: What Impact do Immersion and Presence Experience have on Learning Performance?

In times of PodCast and MOOC's, a rebirth of the "classical" academic lecture in the form of modern technology-mediated communication as video-conserve can be observed. Current technological developments in the field of audiovisual media reduce the distance between a recorded action and its reception. By using high immersive technologies presence experiences can introduc new qualities of communication. In a study undertaken with freshmen we first explored the everyday use of video technologies by students and their experiences and expectations with regard to their use in formal learning contexts. I addition we investigated the question whether and how learning abilities of the recipients are impacted by immersive technologies (360-degree video on Head Mounted Display/VR headset) compared to classical LectureCast technologies (16:9 Video via Desktop). All things considered, it can be stated that an immersive media format alone does not generate added value: there is no increase of attention that further enhances learning, which can be derived from the feeling (of presence) of sitting in a lecture room, even beyond the possibly initial distraction of the appeal of novelty. Furthermore, the teacher-centered mode is unsuitable for a spherical projection due to its spatial setting, where only one direction is being employed. In order to exploit possible media-specific values of 360-degree video in the context of LectureCasts, it is consequently necessary to work either with a modified concept of a teacher-centered lecture, using the entire space, or to set a focus on the content by combining the recorded lecture with additional visual elements. Keywords: LectureCast, Immersion, Sense of Presence, 360-degree video, VR headset, Video-Learning

Accessible scenic arts and Virtual Reality: A pilot study with aged people about user preferences when reading subtitles in immersive environments

The technological advancements in virtual reality and other forms of immersive content such as 360o video are triggering the concept of liveness in the scenic arts, and deserve in-depth consideration in audiovisual translation studies. The reception of subtitles in this new VR-based media needs to be studied. In the specific field of media accessibility, accessibility in immersive media much research needs to be done. This article explores the preferences of two subtitle options: Subtitles for the D/deaf and hard-of-hearing and Easy-to-Read in a 360o video of Romeo and Juliette the opera, played at the Gran Teatre del Liceu during the 2017/2018 season. A pilot study was prepared and conducted with the aim of determining which subtitling options are preferred by users aged between 50 and 79.

HUMAN4D: A Human-Centric Multimodal Dataset for Motions and Immersive Media

We introduce HUMAN4D, a large and multimodal 4D dataset that contains a variety of human activities simultaneously captured by a professional marker-based MoCap, a volumetric capture and an audio recording system. By capturing 2 female and $2$ male professional actors performing various full-body movements and expressions, HUMAN4D provides a diverse set of motions and poses encountered as part of single- and multi-person daily, physical and social activities (jumping, dancing, etc.), along with multi-RGBD (mRGBD), volumetric and audio data. Despite the existence of multi-view color datasets captured with the use of hardware (HW) synchronization, to the best of our knowledge, HUMAN4D is the first and only public resource that provides volumetric depth maps with high synchronization precision due to the use of intra- and inter-sensor HW-SYNC. Moreover, a spatio-temporally aligned scanned and rigged 3D character complements HUMAN4D to enable joint research on time-varying and high-quality dynamic meshes. We provide evaluation baselines by benchmarking HUMAN4D with state-of-the-art human pose estimation and 3D compression methods. For the former, we apply 2D and 3D pose estimation algorithms both on single- and multi-view data cues. For the latter, we benchmark open-source 3D codecs on volumetric data respecting online volumetric video encoding and steady bit-rates. Furthermore, qualitative and quantitative visual comparison between mesh-based volumetric data reconstructed in different qualities showcases the available options with respect to 4D representations. HUMAN4D is introduced to the computer vision and graphics research communities to enable joint research on spatio-temporally aligned pose, volumetric, mRGBD and audio data cues. The dataset and its code are available https://tofis.github.io/myurls/human4d.

Tensor-Based Light Field Compressed Sensing and Epipolar Plane Images Reconstruction via Deep Learning

Light field (LF) can capture the spatial and angular information of the light in one single exposure. And the LF images are widely used in various fields, especially in immersive media. The rich imaging information in the LF poses great challenges for transmission. However, LF images are sparse and redundant to some extent, which makes LF compression possible. Besides, the compressed sensing (CS) theory shows that images can be recovered from a small number of measurements when they are sparse. In this paper, we propose a Tensor-based Compressed Sensing method to compress images and Epipolar Plane Images for reconstruction (TCSEPI). This method divides the viewpoints of LF images into several regions and stacks the images in each region into a 4D tensor, which conduct CS together and yields measurements with common characteristics. Subsequently, the epipolar plane images are used to reconstruct the LF images and restore the geometric consistency information. To achieve better reconstruction results, we design two cascaded convolutional neural networks to implement the measurement matrix optimization and LF images reconstruction sequentially. Experimental results show the superior performance of TCSEPI, which achieves at least 3dB gain in PSNR and outperforms state-of-the-art in the reconstruction quality.

Tensor Oriented No-Reference Light Field Image Quality Assessment.

Light field image (LFI) quality assessment is becoming more and more important, which helps to better guide the acquisition, processing and application of immersive media. However, due to the inherent high dimensional characteristics of LFI, the LFI quality assessment turns into a multi-dimensional problem that requires consideration of the quality degradation in both spatial and angular dimensions. Therefore, we propose a novel Tensor oriented No-reference Light Field image Quality evaluator (Tensor-NLFQ) based on tensor theory. Specifically, since the LFI is regarded as a low-rank 4D tensor, the principal components of four oriented sub-aperture view stacks are obtained via Tucker decomposition. Then, the Principal Component Spatial Characteristic (PCSC) is designed to measure the spatial-dimensional quality of LFI considering its global naturalness and local frequency properties. Finally, the Tensor Angular Variation Index (TAVI) is proposed to measure angular consistency quality by analyzing the structural similarity distribution between the first principal component and each view in the view stack. Extensive experimental results on four publicly available LFI quality databases demonstrate that the proposed Tensor-NLFQ model outperforms state-of-the-art 2D, 3D, multi-view, and LFI quality assessment algorithms.

A Survey on Adaptive 360° Video Streaming: Solutions, Challenges and Opportunities

Omnidirectional or 360° video is increasingly being used, mostly due to the latest advancements in immersive Virtual Reality (VR) technology. However, its wide adoption is hindered by the higher bandwidth and lower latency requirements than associated with traditional video content delivery. Diverse researchers propose and design solutions that help support an immersive visual experience of 360° video, primarily when delivered over a dynamic network environment. This paper presents the state-of-the-art on adaptive 360° video delivery solutions considering end-to-end video streaming in general and then specifically of 360° video delivery. Current and emerging solutions for adaptive 360° video streaming, including viewport-independent, viewport-dependent, and tile-based schemes are presented. Next, solutions for network-assisted unicast and multicast streaming of 360° video content are discussed. Different research challenges for both on-demand and live 360° video streaming are also analyzed. Several proposed standards and technologies and top international research projects are then presented. We demonstrate the ongoing standardization efforts for 360° media services that ensure interoperability and immersive media deployment on a massive scale. Finally, the paper concludes with a discussion about future research opportunities enabled by 360° video.

Reconstructing Personal Stories in Virtual Reality sas a Mechanism to Recover the Self.

Advances in virtual reality present opportunities to relive experiences in an immersive medium that can change the way we perceive our life stories, potentially shaping our realities for the better. This paper studies the role of virtual reality as a tool for the creation of stories with the concept of the self as a narrator and the life of the self as a storyline. The basis of the study is the philosophical notion of the self-narrative as an explanatory story of the events in one’s life that constitutes the notion of one’s self. This application is suitable for cases when individuals need to recreate their self, such as during recovery after traumatic events. The analysis of the effects of virtual reality shows that it enables a person to engage in a process of deeper self-observation to understand and explain adverse events and to give meaning to these events to form a new story, which can complement the therapeutic outcomes of exposure treatments. This study proposes concrete examples of immersive scenarios used to reconstruct personal stories. Several possible levels of experience are proposed to suggest that recovery can be achieved through the gradual retelling of the self-narrative, addressing all of the underlying narratives. Considering the ethical challenges that might arise, this paper explores the ways in which immersion in virtual reality can benefit a person’s view toward life as a story and his or her self as its author, comparing this idea with previous research on the application of virtual reality for trauma treatment. The analysis also emphasizes the perception of narrative authorship in virtual reality as an essential method for recovering the self-narrative and improving a patient’s mental health during self-actualization.