Immersive Media Research Articles

MPEG column: 116th MPEG meeting

At its 116th meeting, MPEG successfully organised a workshop on its 5-year standardisation roadmap. Various industry representatives presented their views and reflected on the need for standards for new services and applications, specifically in the area of immersive media. The results of the workshop (roadmap, presentations) and the planned phases for the standardisation of "immersive media" are available at http://mpeg.chiariglione.org/. A follow-up workshop will be held on 18 January 2017 in Geneva, co-located with the 117th MPEG meeting. The workshop is open to all interested parties and free of charge. Details on the program and registration will be available at http://mpeg.chiariglione.org/.

Adaptive Multiview Video Delivery Using Hybrid Networking

Multiview entertainment is the next step in 3D immersive media networking owing to its improved depth perception and free-viewpoint viewing capability whereby users can observe the scene from the desired viewpoint. This paper outlines a delivery system for multiview plus depth video, combining the broadcast and broadband networks. The digital video broadcast (DVB) network is used along with adaptive peer-to-peer (P2P) distribution over the Internet to deliver high-volume multimedia to users. The DVB network has been used to deliver part of the 3D service, owing to its robustness and wide availability, as a mechanism to guarantee the minimum 3D quality of experience. The developed system brings key contributions in the P2P transport for real-time multimedia delivery, including a user preference-aware adaptation mechanism, adaptive redundant chunk scheduling for robustness, incentives to decrease the load on the content server for improved system scalability, and resynchronization capability with the DVB transmission. The introduced features are compared with those of some other well-known P2P solutions to highlight the quantitative gains. A subjective testing campaign has also been organized on the developed hybrid platform, which proves the effectiveness of user-aware adaptation over network-based adaptation on a mean opinion score scale.

Parisflâneur. Artistic Approaches to Binaural Technology and Their Evaluation

This article approaches binaural interactive environments from an artistic research perspective. Beyond content production, an aesthetic reflection of binaural media requires pervasive access to digital processing means and ways to employ them in composition. However, most conventional workflows separate media-specific rendering algorithms from object-based scene authoring. Such a delimitation between binaural engineering and its application restricts transdisciplinary creation that crosses both areas. This article assumes that the full potential of immersive media cannot be explored without investigating technology in the context of aesthetic experience. A case study is presented in which artistic references are regarded together with its technical realization. Contemporary user experience evaluation methods are adopted and refined with reference to the aims of the artist. A subsequent revision of the work is discussed along with implementation adjustments and conceptual alterations. The presented project shall exemplify how artistic research may bridge scholarly investigation and the creative acquirement of media technology beyond its mere application. A point of departure shall be provided for further cross-fertilization between engineering and the arts by identifying mutual implications.

The effect of reverberation and audio spatialization on egocentric distance estimation of objects in stereoscopic virtual reality

It has been reported by numerous studies on distance perception in VR that a compression of visual space occurs in virtual environments presented using stereoscopic techniques. Other studies have shown that modified environmental auditory cues can affect egocentric spatial perception and that increased order of modality improved the experience of immersive media. Work was conducted to measure the effect of spatialized acoustic cues on egocentric distance estimation in head mounted display VR. Results suggest that the although early reflection content was not found to have significant effect on estimation of distance, presence of reverberation increases the perception of distance for objects further than 5m from the user and can compensate for the spatial compression observed in the use of stereoscopic VR.

Using consensus group methods to improve simulation using immersive media in paramedicine

Problem: Simulation is a cornerstone of paramedic pedagogy. It facilitates a learning-centred approach which explicitly links the learning and teaching environment to the real world and enables the student to construct new understandings of paramedic practice threshold concepts. Although the benefits of simulation in medical and health education have been well researched, there is a paucity of research into how to deliver simulation using immersive media (IM) due to its recency as a strategy in paramedicine. This study explored student feedback of simulations using IM to improve the efficacy of learning activities designed for the recently commissioned University of the Sunshine Coast (USC) immersive simulation studio. Method: Feedback was taken using a modified Nominal Group Technique (NGT) consensus method post implementation of an IM strategy. Participants were second year students of the Bachelor of Paramedic Science program at USC who had undertaken simulation using immersive media in 2015. Results: The research has revealed the value of listening to the student voice in enhancing simulation design. Two major themes arose regarding the value of IM according to students. Firstly, the students perceived that benefits were derived from the closer approximation of an authentic practice environment through the use of immersive media. Secondly, that students felt that the immersive style allowed them to practice of a broader range of technical and non-technical skills than in a conventional classroom. Conclusion: IM was perceived as a positive experience by students as they reported IM to be more engaging, increasing their confidence, situational awareness and improved their non-technical skills. The USC paramedic program will continue to develop the IM strategy as a signature pedagogy.

Closing the Gap: Human Factors in Cross-Device Media Synchronization

The continuing growth in the mobile phone arena, particularly in terms of device capabilities and ownership, is having a transformational impact on media consumption. It is now possible to consider orchestrated multistream experiences delivered across many devices, rather than the playback of content from a single device. However, there are significant challenges in realizing such a vision, particularly around the management of synchronicity between associated media streams. This is compounded by the heterogeneous nature of user devices, the networks upon which they operate, and the perceptions of users. This paper introduces IMSync, an open interstream synchronization framework that incorporates human factors to capture the quality of experience (QoE). IMSync adopts efficient monitoring and control mechanisms, alongside a QoE perception model that has been derived from a series of subjective user experiments. Based on an observation of lag, IMSync is able to use this model of impact to determine an appropriate strategy to catch-up with playback while minimizing the potential detrimental impacts on a users QoE. The impact model adopts a balanced approach: Trading off the potential impact on QoE of initiating a resynchronization process compared with retaining the current levels of nonsynchronicity, in order to maintain high levels of QoE. A series of experiments demonstrate the potential of the framework as a basis for enabling new immersive media experiences.

Immersion, authenticity and the theme park as social space: Experiencing the Wizarding World of Harry Potter

This article examines the visitor experience of the Wizarding World of Harry Potter (WWOHP) theme park at Universal Studios Orlando. The park is hugely popular and has been embraced by the series’ devoted but critical fanbase. Prior research on theme parks has generally focused on critiques of their form, leading to a limited understanding of their appeal. This article asks how fan-visitors interpret this simulated environment, and what leads them to embrace it. It does this with an ethnographic approach, utilizing in-depth interviews with 15 visitors combined with participant observation. We show how WWOHP is understood by its visitors as an adaptation of the series into physical space, via the medium of the theme park, and how the visitor’s experience is shaped through use of ironic imagination. In doing so, we present a new understanding of the immersive media experience of theme parks.

실감미디어 동영상정보를 이용한 실내 공간 정보 제공 시스템 구현

This paper presents the interior space information using 6D-360 degree immersive media video information. And we implement the augmented reality, which includes a variety of information such as position information, movement information of the specific location in the interior space GPS signal does not reach the position information. Augmented reality containing the 6D-360 degree immersive media video information provides the position information and the three dimensional space image information to identify the exact location of a user in an interior space of a moving object as well as a fixed interior space. This paper constitutes a three dimensional image database based on the 6D-360 degree immersive media video information and provides augmented reality service. Therefore, to map the various information to 6D-360 degree immersive media video information, the user can check the plant in the same environment as the actual. It suggests the augmented reality service for the emergency escape and repair to the passengers and employees.

Extrapolating on McLuhan: How Media Environments of the Given, the Represented, and the Induced Shape and Reshape Our Sensorium

The article develops Marshall McLuhan’s approach to the interplay between media, the sensorium, and reality. McLuhan’s concepts of “acoustic space” and “visual space” are unfolded with regard to the consequences that digital media will have on the human ability to perceive reality. Reality–sensorium interaction is systematized in the article. This systematization includes the environments of the given, the represented, and the induced. These environments are shaped by sequential stages of media evolution, which relate to preliterate media, alphabet-based media, and digital media. Existing and upcoming media technologies are presumed to alter human biology and transcend it. Within the set of media technologies that alter human biology, artificial flavours, electrically induced senses, immersive media, augmented reality, and virtual reality are treated. Within the set of media impacts that will change the human sensorium, the dismissal of gravity (related to the McLuhanian “angelism” of electronic discarnate man), the switch in navigation from biological networking to social networking, the sense of others, and the thirst for response are treated. Plato, Lenin, Wittgenstein, Benveniste, Logan, Carr, Shirky, and other thinkers are employed in the article to support these McLuhanian speculations, and sketch out prospective trends in the evolution of media and the sensorium.

Στο Άντρο Της Φαινομενικότητας: Προς Μία Φιλοσοφία & Αισθητική Του Τεχνητού - Από την Παρμενίδεια Οντο-φυσιο-λογία Στις Τεχνοεπιστήμες της Προσομοίωσης (Into the Den of Phenomenality: Toward a Philosophy & Aesthetics of The Artificial - From Parmenidean Onto-physio-logy to Simulation Technosciences)

Greek Abstract: Eλάχιστα αναθeωρημένη, eπαυξημένη κατά μeρικές σeλίδeς προοιμίου και eπίμeτρου, διδακτορική διατριβή (100.000 λέξeις) του Δημήτρη Γκινοσάτη, η οποία υποστηρίχθηκe και βαθμολογήθηκe μe άριστα παμψηφeί στο Τμήμα Eπικοινωνίας, Μέσων & Πολιτισμού του Παντeίου Πανeπιστημίου Κοινωνικών & Πολιτικών Eπιστημών το 2008. Σημeίο eκκίνησης της διδακτορικής διατριβής αποτeλeί η σύγχρονη θeωρητική διαμάχη που λαμβάνeι χώρα στις ανθρωπιστικές eπιστήμeς γύρω από το καθeστώς της πραγματικότητας, όπως αυτή διαμορφώνeται και υπeρκαθορίζeται υπό την πανταχού κυριαρχία των ψηφιακών πληροφορικών/υπολογιστικών μιντιοτeχνολογιών. Μe τη ραγδαία, μαζική eπέλeυση κατά τις τρeις τeλeυταίeς δeκαeτίeς των ψηφιακών τeχνολογιών στον αλφαβητικά δομημένο βιόκοσμό μας, η έννοια της –αποκύημα η ίδια eνός πολιτισμού αλφαβητικά συγκροτημένου– αλλάζeι ριζικά. Και πράγματι, από τη στιγμή που ο βιόκοσμός μας προσλαμβάνeι τη μορφή eνός τόσο πυκνά διατeταγμένου δικτύου διασυνδeδeμένων τeχνικών μέσων και καναλιών πληροφοριών, η παραδοσιακή έννοια, αλλά και η φύση αυτού που άλλοτe αποκαλούσαμe πραγματικότητα αλλάζουν άρδην: η ιστορία μας διδάσκeι ότι η έννοια της πραγματικότητας καθίσταται προβληματική, όταν μeταβάλλeται το σημασιακό πeριeχόμeνο της συμμeτρικά αντίθeτης προς αυτήν έννοιας, eν προκeιμένω της έννοιας της φαινομeνικότητας. Στόχος της διατριβής eίναι η προβληματοποίηση των παραπάνω ζητημάτων μέσα από το κeιμeνικό σώμα της δυτικής οντομeταφυσικής παράδοσης (όπως αυτή eγκαθιδρύθηκe από την Παρμeνίδeια οντο-φυσιο-λογία κι eν συνeχeία από την Πλατωνική θeωρία του eίναι, της και της φαινομeνικότητας, για να αποδομηθeί από τις Καντιανές και μeτέπeιτα Νιτσeϊκές κριτικές) έως τις eπιστήμeς του τeχνητού (H. Simon) και τις σύγχρονeς θeωρίeς των τeχνικών μέσων, αποβλέποντας στη συγκρότηση eνός γeνικού eπιστημολογικού και eυρύτeρα φιλοσοφικο-αισθητικού πλαισίου πραγμάτeυσης του τeχνητού, όπως αυτό προκύπτeι, αναδeικνύeται και τίθeται μe ριζικό τρόπο μέσα από τις τeχνο-eπιστήμeς ψηφιακής μοντeλοποίησης και προσομοίωσης συστημάτων. English Abstract: Minimally reworked Ph.D. thesis (100.000 words) by Dimitris Ginosatis, obtained in 2008 from the Department of Communication, Media & Culture of the Panteion University of Social & Political Sciences (Athens-Greece). The task undertaken in the thesis was to delve into, problematize and thematize the ontological status of what we traditionally call reality, as determined by today's advanced computational/information mediotechnologies, through our western onto-metaphysical tradition (as established by foundational Parmenidean and Platonic doctrines of being, and appearance and deconstructed by Kantian and Nietzschean critiques) up to 20th century sciences of the artificial (H. Simon) and contemporary media theory, with view to outlining a solid conceptual framework for the construction of a general epistemological and aesthetic theory of how a reality should be, henceforth, conceived and thought of under the sovereignty of today's all-encompassing, immersive media technologies.

A Framework for QoE-Aware 3D Video Streaming Optimisation over Wireless Networks

The delivery of three-dimensional immersive media to individual users remains a highly challenging problem due to the large amount of data involved, diverse network characteristics, and user terminal requirements, as well as user’s context. This paper proposes a framework for quality of experience-aware delivering of three-dimensional video across heterogeneous wireless networks. The proposed architecture combines a Media-Aware Proxy (application layer filter), an enhanced version of IEEE 802.21 protocol for monitoring key performance parameters from different entities and multiple layers, and a QoE controller with a machine learning-based decision engine, capable of modelling the perceived video quality. The proposed architecture is fully integrated with the Long Term Evolution Enhanced Packet Core networks. The paper investigates machine learning-based techniques for producing an objective QoE model based on parameters from the physical, the data link, and the network layers. Extensive test-bed experiments and statistical analysis indicate that the proposed framework is capable of modelling accurately the impact of network impairments to the perceptual quality of three-dimensional video user.

A reproducible olfactory display for exploring olfaction in immersive media experiences

Olfaction, the sense of smell, is an important perceptual function. It has been shown to enhance the quality of life and to facilitate memory recall. However, despite its importance and potential benefits, olfaction is often neglected when creating immersive media experiences (IMEs). The probable reason is that olfactory displays are not readily available. We have developed a reproducible olfactory display that is simple and affordable to make olfactory displays more available. More importantly, we have conducted three exploratory studies on the effects of olfaction. The results of these studies indicate that olfaction is a nuanced sense that can have both positive and negative effects on user experiences. Hence, further research is needed to better understand and design for olfaction in IMEs.

Modeling immersive media experiences by sensing impact on subjects

As immersive technologies target to provide higher quality of multimedia experiences, it is important to understand the quality of experience (QoE) perceived by users from various multimedia rendering schemes, in order to design and optimize human-centric immersive multimedia systems. In this study, various QoE-related aspects, such as depth perception, sensation of reality, content preference, and perceived quality are investigated and compared for presentation of 2D and 3D contents. Since the advantages of implicit over explicit QoE assessment have become essential, the way these QoE-related aspects influence brain and periphery is also investigated. In particular, two classification schemes using electroencephalography (EEG) and peripheral signals (electrocardiography and respiration) are carried out, to explore if it is possible to automatically recognize the QoE-related aspects under investigation. In addition, a decision-fusion scheme is applied to EEG and peripheral features, to explore the advantage of integrating information from the two modalities. The results reveal that the highest monomodal average informedness is achieved in the high beta EEG band (0.14 % ± 0.09, p < 0.01), when recognizing sensation of reality. The highest and significantly non-random multimodal average informedness is achieved when high beta EEG band is fused with peripheral features (0.17 % ± 0.1, p < 0.01), for the case of sensation of reality. Finally, a temporal analysis is conducted to explore how the EEG correlates for the case of sensation of reality change over time. The results reveal that the right cortex is more involved when sensation of reality is low, and the left when sensation of reality is high, indicating that approach and withdrawal-related processes occur during sensation of reality.

Novel Televisual Environments

This article focuses on one of the most ground-breaking technological attempts in creating novel immersive media environments for heightened televisual user experiences: 3DTV, a Network of Excellence funded by the European Commission 6th Framework Information Society Technologies Programme. Based on the theoretical framework outlined by the works of Jonathan Crary and Brian Winston, and on empirical data obtained from author’s fieldwork and laboratory visit notes, as well as discussions with practitioners, the article explores the history of stereoscopic vision and technological progress related with it, and looks for possible reasons of 3DTV’s dramatic commercial failure.

Designing for Immersive Technology: Integrating Art and STEM Learning

Students struggle to learn science, technology, engineering and mathematics concepts. The arts have been proposed as a means to engage students in STEM education, resulting in the idea of STEAM. This study investigates how two students in a six-week summer program solved technological and design production problems to create public service announcements for the immersive fulldome on the topic of water conservation. Qualitative data were collected, including interviews, observations, artifacts of student work and reflections. Qualitative analysis focused on integration of STEM content and practices with the arts. The study contributes to what is known about how people learn when they design for immersive media, and identify potential barriers and affordances for learning STEM through the arts.

Documentation and dissemination of the sculptural elements of Canada's Parliamentary Buildings: Methodology development and evolution, a case study

Abstract. Parliament Hill consists of four historic gothic revival buildings, which form part of the Parliament Buildings National Historic Site of Canada in the National Capital of Ottawa. There are more than 2000 masonry sculptural elements throughout the four buildings. Three of the buildings are in the middle of multi-year rehabilitation projects. Extensive Heritage Documentation is being undertaken to support various activities and conservation teams throughout the interior and exterior of the buildings while also serving as a key posterity records. One of the significant heritage documentation projects is the 3D digitization of the 2000+ heritage character defining sculptural elements. The Heritage Conservation Directorate (HCD) of Public Works and Government Services Canada (PWGSC) was tasked by the Parliamentary Precinct Branch (PPB) of PWGSC to document these character defining elements. The sculptures vary in size from as small as 100mm in width to up to 2 meters in size. This project is in its third year and much has been learned and researched about the most appropriate and efficient means by which to document these elements. Although a methodology was in place to document the sculptures at the inception of the project, it has gone through several iterations in order to improve the gathered data, and in turn increase the efficiency, quality and speed of data acquisition. This paper will describe the evolution of the methodology, as well as the rationale for the alterations in technique. With over 600 of the approximate 2000 (heritage character defining) sculptural elements captured to date, the project is entering a critical phase where an efficient and effective method for sharing and disseminating the information to a wide audience is being explored and evaluated. The end result is intended to allow the client (PPB) and the general public a way to look at and interactively manipulate the viewpoint of each digital model. This will provide a unique opportunity for a wide audience to evaluate and appreciate these elaborate works of art. This database of information can also be of use for academic study and research. To date the data captured by HCD has been utilized by conservation professionals and by the Carleton Immersive Media Studio (CIMS) of Carleton University. HCD has partnered with CIMS to explore a wide variety of potential uses for the data including the creation of 2D-drawings, stone carving profile templates, 3D printed scale models, CNC carved maquettes, robotic stone carving, and digitally-designed replacement elements. CIMS and HCD have tested the limits of digital technologies to assist stone carvers in restoring carved elements while also learning from the sculptors and masons to better understand their approach when conserving or restoring a sculptural element. Examples of how the conservation team, specifically the stone carvers have been working with the data and technology will also be presented.

Documentation for emergency condition mapping of Decorated historic surfaces at the Caid Residence, the Kasbah of Taourirt (Ouarzazate, Morocco)

Abstract. As it is broadly understood, recording serves as a basis for the diagnosis, treatment and preservation of historic places and contributes to record our built cultural heritage for posterity. This work is not a stand-alone practice but a part of the overall conservation process of cultural heritage at imminent risk of irreversible damage. Recording of heritage places should be directly related to the needs, skills and the technology that are available to the end users that are responsible for the management and care of these sites. They should be selected in a way that the future managers of these sites can also access and use the data collected. This paper explains an innovative heritage recording approach applied by the Getty Conservation Institute (GCI) and Carleton Immersive Media Studio (CIMS) in the documentation of historic decorated surfaces at the Caïd Residence, located at Tighermt (Kasbah) Taourirt in Ouarzazate, Morocco; as part of a collaborative project between the GCI and the Centre de Conservation et Réhabilitation du Patrimoine Architectural des Zones Atlasiques et Sub-Atlasiques (CERKAS) to rehabilitate the entire architectural ensemble. The selected recording techniques were used for the rapid mapping of conditions of the decorated surfaces at the Caïd Residence using international standards. The resulting work is being used by GCI staff, consultants and CERKAS team to conduct emergency stabilization and protection measures for these important decorated surfaces.

Integrated Digital Technologies for the Architectural Rehabilitation &amp; Conservation of Beinn Bhreagh Hall &amp; Surrounding Site, Nova Scotia, Canada

Abstract. This contribution summarizes the collaboration between Carleton Immersive Media Studio (CIMS) and Heritage Standing in the use of digital technologies to document the Beinn Bhreagh Hall historic site (house and immediate surroundings) in Baddeck, Nova Scotia for its rehabilitation and protection. The project objectives were to develop a prototype for research and training using 3D scanning, and Building Information Modelling (BIM), as well as other emerging surveying tools to understand the state of conservation of Historic Places in Canada, providing relevant and appropriate information for their rehabilitation and maintenance. The vehicle of this research was the documentation and modelling of this important landmark, the summer home of Dr. Alexander Graham Bell in Victoria County, Nova Scotia. This unique 19thC building, currently under review for designated as a national historic site.

Digitally-Assisted Stone Carving of a Relief Sculpture for the Parliament Buildings National Historic Site of Canada

Abstract. The emerging field of digital fabrication is a process where three-dimensional datasets can be directly transferred to fabrication equipment to create models or even 1:1 building elements. In this paper, we will discuss the results of a collaboration between the Carleton Immersive Media Studio (CIMS), the Dominion Sculptor of Canada, and the Heritage Conservation Directorate (HCD) of Public Works and Government Services Canada (PWGSC), that utilizes digital fabrication technologies in the development of a digitally-assisted stone carving process. The collaboration couples the distinguished skill of the Dominion Sculptor with the latest digital acquisition and digital fabrication technologies for the reconstruction of a deteriorated stone bas-relief on the façade of the East Block building of the Parliament Buildings National Historic Site of Canada. The intention of the research is to establish a workflow of hybrid digital/analogue methodologies from acquisition through rehabilitation and ultimately to the fabrication of stone elements.

해양 채널 환경에서 3D 입체영상의 전송을 위한 계층변조 기법

Recently, Due to the rapid growth of broadcasting communication and video coding technologies, the demands for immersive media contents based on 3D stereoscopic video will increase steadily. And the demands must ultimately provide the contents for users which are in wireless channel such as vehicle, train, and ship. Thus, in this paper, we transmit the 3D stereoscopic video over the maritime Rician channel that direct wave is more dominant than reflective wave. Besides, we present unequel error protection (UEP) by applying hierarchical 4/16-QAM to V+D(Video plus Depth) format which can represent 3D stereoscopic video. We expect our system to provide seamless broadcasting service for users with poor reception condition.