Creating explorable extended reality environments with semantic annotations

Jakub Flotyński

doi:10.1007/s11042-020-09772-y

Jakub Flotyński

Open Access

https://doi.org/10.1007/s11042-020-09772-y

Copy DOI

Abstract

The main element of extended reality (XR) environments is behavior-rich 3D content consisting of objects that act and interact with one another as well as with users. Such actions and interactions constitute the evolution of the content over time. Multiple application domains of XR, e.g., education, training, marketing, merchandising, and design, could benefit from the analysis of 3D content changes based on general or domain knowledge comprehensible to average users or domain experts. Such analysis can be intended, in particular, to monitor, comprehend, examine, and control XR environments as well as users’ skills, experience, interests and preferences, and XR objects’ features. However, it is difficult to achieve as long as XR environments are developed with methods and tools that focus on programming and 3D modeling rather than expressing domain knowledge accompanying content users and objects, and their behavior. The main contribution of this paper is an approach to creating explorable knowledge-based XR environments with semantic annotations. The approach combines description logics with aspect-oriented programming, which enables knowledge representation in an arbitrary domain as well as transformation of available environments with minimal users’ effort. We have implemented the approach using well-established development tools and exemplify it with an explorable immersive car showroom. The approach enables efficient creation of explorable XR environments and knowledge acquisition from XR.

Highlights

Extended reality (XR) covers different forms of combined real and virtual environments, ranging from augmented reality (AR) to virtual reality (VR) in the reality-virtuality continuum [34], and encompassing different types of presentation and interaction with objects in Multimedia Tools and Applications (2021) 80:6959–6989 such environments [27]
The performance of the tool has been evaluated in terms of transforming XR environments, inserting statements derived from annotations to a triplestore, and the number of FPS rendered for the generated explorable environments
The obtained results show that the implemented environment efficiently transforms XR environments to their explorable counterparts, as the time of transformation varies from 3 seconds for smaller environments to 13 seconds for larger environments

Summary

Introduction

Extended reality (XR) covers different forms of combined real and virtual environments, ranging from augmented reality (AR) to virtual reality (VR) in the reality-virtuality continuum [34], and encompassing different types of presentation and interaction with objects in Multimedia Tools and Applications (2021) 80:6959–6989 such environments [27]. XR attracts users in multiple application domains, such as marketing, merchandising, simulation, design, engineering, medicine, education, and training. Multiple application domains can benefit from registering behavior of users and 3D content in XR, including their actions and interactions described using general or domain knowledge. Registered behavior can be subject to exploration with queries about 3D content states at different moments and periods in time. It may be especially useful in XR intended to acquire knowledge about the environment behavior as well as users’ behavior, experience, interests, and preferences. In marketing and merchandising, collected information about actions of customers interacting with products and salesmen in virtual stores can provide knowledge of their interests and preferences. Information collected while training can be used to consider diverse situations and teach beginners

Methods

Results

Conclusion