Immersive Media Experiences Research Articles

MPEG-I Scene Description: A Scene Description Framework for Interactive and Immersive Media Experiences

MPEG-I Scene Description: A Scene Description Framework for Interactive and Immersive Media Experiences

Design Dimensions of Co-Located Multi-Device Audio Experiences

The widespread distribution of mobile computing presents new opportunities for the consumption of interactive and immersive media experiences using multiple connected devices. Tools now exist for the creation of these experiences; however, there is still limited understanding of the best design practices and use cases for the technology, especially in the context of audio experiences. In this study, the application space of co-located multi-device audio experiences is explored and documented through a review of the literature and a survey. Using the obtained information, a set of seven design dimensions that can be used to characterise and compare experiences of this type is proposed; these are synchronisation, context, position, relationship, interactivity, organisation, and distribution. A mapping of the current application space is presented where four categories are identified using the design dimensions, these are public performances, interactive music, augmented broadcasting, and social games. Finally, the overlap between co-located multi-device audio and audio-augmented reality (AAR) experiences is highlighted and discussed. This work will contribute to the wider discussion about the role of multiple devices in audio experiences and provide a source of reference for the design of future multi-device audio experiences.

Spatial audio production for immersive media experiences: Perspectives on practice-led approaches to designing immersive audio content

Sound design with the goal of immersion is not new. However, sound design for immersive media experiences (IMEs) utilizing spatial audio can still be considered a relatively new area of practice with less well-defined methods requiring a new and still emerging set of skills and tools. There is, at present, a lack of formal literature around the challenges introduced by this relatively new content form and the tools used to create it, and how these may differ from audio production for traditional media. This article, through the use of semi-structured interviews and an online questionnaire, looks to explore what audio practitioners view as defining features of IMEs, the challenges in creating audio content for IMEs and how current practices for traditional stereo productions are being adapted for use within 360 interactive soundfields. It also highlights potential direction for future research and technological development and the importance of practitioner involvement in research and development in ensuring future tools and technologies satisfy the current needs.

Immersive media experiences

An introduction to the QUALINET White Paper on Definitions of Immersive Media Experience (IMEx) [1].

QoE column: Solving complex issues through immersive narratives

A transdisciplinary dialogue and innovative research, including technical and artistic research as well as digital humanities are necessary to solve complex issues. We need to support and produce creative practices, and engage in a critical reflection about the social and ethical dimensions of our current technology developments. At the core is an understanding that no single discipline, technology, or field can produce knowledge capable of addressing the complexities and crises of the contemporary world. Moreover, we see the arts and humanities as critical tools for understanding this hyper-complex, mediated, and fragmented global reality. As a use case, we will consider the complexity of extreme weather events, natural disasters and failure of climate change mitigation and adaptation, which are the risks with the highest likelihood of occurrence and largest global impact (World Economic Forum, 2017). Through our project, World of Wild Waters (WoWW), we are using immersive narratives and gamification to create a simpler holistic understanding of cause and effect of natural hazards by creating immersive user experiences based on real data, realistic scenarios and simulations. The objective is to increase societal preparedness for a multitude of stakeholders. Quality of Experience (QoE) modeling and assessment of immersive media experiences are at the heart of the expected impact of the narratives, where we would expect active participation, engagement and change, to play a key role [1]. Here, we present our views of immersion and presence in light of Quality of Experience (QoE). We will discuss the technical and creative considerations needed for QoE modeling and assessment of immersive media experiences. Finally, we will provide some reflections on QoE being an important building block in immersive narratives in general, and especially towards considering Extended Realities (XR) as an instantiation of Digital storytelling.

Adaptive multi-view video streaming using side information over peer-to-peer networks

Multi-view plus-depth-map (MVD) video streaming with autostereoscopic displays provides multi-user immersive media experiences. In this context, delivery of MVD representation to multiple clients remains a challenging problem because of the high-volume of data involved and the inherent limitations imposed by the delivery networks. To this end, this paper investigates the side information (SI) assisted adaptation algorithm using peer-to-peer (P2P) systems. P2P delivery systems for MVD video can maximize link utilization, preventing the transport of multiple video copies of the same packet for many users. However, the quality of experience (QoE) can be significantly degraded by dynamic variations caused by network congestions. To this end, our solution comprises the extraction of low-overhead metadata at the encoding server that is distributed through the P2P network as SI and used by P2P clients performing network adaptation. In the proposed adaptation strategy, pre-selected views are discarded at times of network congestion and reconstructed with an optimal reconstruction performance using the delivered SI and the delivered neighboring camera views. The experimental results show that the robustness of P2P multi-view streaming using the proposed adaptation scheme is significantly increased in the P2P network.

Closing the Gap: Human Factors in Cross-Device Media Synchronization

The continuing growth in the mobile phone arena, particularly in terms of device capabilities and ownership, is having a transformational impact on media consumption. It is now possible to consider orchestrated multistream experiences delivered across many devices, rather than the playback of content from a single device. However, there are significant challenges in realizing such a vision, particularly around the management of synchronicity between associated media streams. This is compounded by the heterogeneous nature of user devices, the networks upon which they operate, and the perceptions of users. This paper introduces IMSync, an open interstream synchronization framework that incorporates human factors to capture the quality of experience (QoE). IMSync adopts efficient monitoring and control mechanisms, alongside a QoE perception model that has been derived from a series of subjective user experiments. Based on an observation of lag, IMSync is able to use this model of impact to determine an appropriate strategy to catch-up with playback while minimizing the potential detrimental impacts on a users QoE. The impact model adopts a balanced approach: Trading off the potential impact on QoE of initiating a resynchronization process compared with retaining the current levels of nonsynchronicity, in order to maintain high levels of QoE. A series of experiments demonstrate the potential of the framework as a basis for enabling new immersive media experiences.

A reproducible olfactory display for exploring olfaction in immersive media experiences

Olfaction, the sense of smell, is an important perceptual function. It has been shown to enhance the quality of life and to facilitate memory recall. However, despite its importance and potential benefits, olfaction is often neglected when creating immersive media experiences (IMEs). The probable reason is that olfactory displays are not readily available. We have developed a reproducible olfactory display that is simple and affordable to make olfactory displays more available. More importantly, we have conducted three exploratory studies on the effects of olfaction. The results of these studies indicate that olfaction is a nuanced sense that can have both positive and negative effects on user experiences. Hence, further research is needed to better understand and design for olfaction in IMEs.

Modeling immersive media experiences by sensing impact on subjects

As immersive technologies target to provide higher quality of multimedia experiences, it is important to understand the quality of experience (QoE) perceived by users from various multimedia rendering schemes, in order to design and optimize human-centric immersive multimedia systems. In this study, various QoE-related aspects, such as depth perception, sensation of reality, content preference, and perceived quality are investigated and compared for presentation of 2D and 3D contents. Since the advantages of implicit over explicit QoE assessment have become essential, the way these QoE-related aspects influence brain and periphery is also investigated. In particular, two classification schemes using electroencephalography (EEG) and peripheral signals (electrocardiography and respiration) are carried out, to explore if it is possible to automatically recognize the QoE-related aspects under investigation. In addition, a decision-fusion scheme is applied to EEG and peripheral features, to explore the advantage of integrating information from the two modalities. The results reveal that the highest monomodal average informedness is achieved in the high beta EEG band (0.14 % ± 0.09, p < 0.01), when recognizing sensation of reality. The highest and significantly non-random multimodal average informedness is achieved when high beta EEG band is fused with peripheral features (0.17 % ± 0.1, p < 0.01), for the case of sensation of reality. Finally, a temporal analysis is conducted to explore how the EEG correlates for the case of sensation of reality change over time. The results reveal that the right cortex is more involved when sensation of reality is low, and the left when sensation of reality is high, indicating that approach and withdrawal-related processes occur during sensation of reality.

Architecture and Methods for Innovative Heterogeneous Wireless Sensor Network Applications

Nowadays wireless sensor netwoks (WSN) technology, wireless communications and digital electronics have made it realistic to produce a large scale miniaturized devices integrating sensing, processing and communication capabilities. The focus of this paper is to present an innovative mobile platform for heterogeneous sensor networks, combined with adaptive methods to optimize the communication architecture for novel potential applications in multimedia and entertainment. In fact, in the near future, some of the applications foreseen for WSNs will employ multi-platform systems with a high number of different devices, which may be completely different in nature, size, computational and energy capabilities, etc. Nowadays, in addition, data collection could be performed by UAV platforms which can be a sink for ground sensors layer, acting essentially as a mobile gateway. In order to maximize the system performances and the network lifespan, the authors propose a recently developed hybrid technique based on evolutionary algorithms. The goal of this procedure is to optimize the communication energy consumption in WSN by selecting the optimal multi-hop routing schemes, with a suitable hybridization of different routing criteria. The proposed approach can be potentially extended and applied to ongoing research projects focused on UAV-based sensing with WSN augmentation and real-time processing for immersive media experiences.