Mixed Reality Applications Research Articles

Supporting the Conservation and Restoration OpenLab of the Acropolis of Ancient Tiryns through Data Modelling and Exploitation of Digital Media

Open laboratories (OpenLabs) in Cultural Heritage institutions are an effective way to provide visibility into the behind-the-scenes processes and promote documentation data collected and produced by domain specialists. However, presenting these processes without proper explanation or communication with specialists may cause issues in terms of visitors’ understanding. To support OpenLabs and disseminate information, digital media and efficient data management can be utilized. The CAnTi (Conservation of Ancient Tiryns) project seeks to design and implement virtual and mixed reality applications that visualize conservation and restoration data, supporting OpenLab operations at the Acropolis of Ancient Tiryns. Semantic Web technologies will be used to model the digital content, facilitating organization and interoperability with external sources in the future. These applications will be part of the OpenLab activities on the site, enhancing visitors’ experiences and understanding of current and past conservation and restoration practices.

VanityX: An Agile 3D Rendering Platform Supporting Mixed Reality

VanityX is a prototype, low-level, real-time 3D rendering and computing platform. Unlike most XR solutions, which integrate several commercial and/or open-source products, such as game engines, XR libraries, runtime, and services, VanityX is a platform ready to adapt to any business domain including anthropology and medicine. The design, architecture, and implementation are presented, which are based on CPU and GPU asymmetric multiprocessing with explicit synchronization and collaboration of parallel tasks and a predictable transfer of pipeline resources between processors. The VanityX API is based on DirectX 12 and native programming languages C++20 and HLSL 6, which, in conjunction with explicit parallel processing, the asynchronous loading and explicit managing of graphic resources, and effective algorithms, results in great performance and resource utilization close to metal. Surface-based rendering, direct volume rendering (DVR), and mixed reality (MR) on the HoloLens 2 immersive headset are currently supported. Our MR applications are directly compiled and deployed to HoloLens 2 allowing for better programming experiences and software engineering practices such as testing, debugging, and profiling. The VanityX server provides various computational and rendering services to its clients running on HoloLens 2. The use and test cases are in many business domains including anthropology and medicine. Our future research challenges will primarily, via the MetaverseMed project, focus on opening new opportunities for implementing innovative MR-based scenarios in medical procedures, especially in education, diagnostics, and surgical operations.

Full body video-based self-avatars for mixed reality: from E2E system to user study

In this work, we explore the creation of self-avatars through video pass-through in mixed reality (MR) applications. We present our end-to-end system, including custom MR video pass-through implementation on a commercial head-mounted display (HMD), our deep learning-based real-time egocentric body segmentation algorithm, and our optimized offloading architecture, to communicate the segmentation server with the HMD. To validate this technology, we designed an immersive VR experience where the user has to walk through a narrow tile path over an active volcano crater. The study was performed under three-body representation conditions: virtual hands, video pass-through with color-based full-body segmentation, and video pass-through with deep learning full-body segmentation. This immersive experience was carried out by 30 women and 28 men. To the best of our knowledge, this is the first user study focused on evaluating video-based self-avatars to represent the user in a MR scene. Results showed no significant differences between the different body representations in terms of presence, with moderate improvements in some Embodiment components between the virtual hands and full-body representations. Visual Quality results showed better results from the deep-learning algorithms in terms of the whole body perception and overall segmentation quality. In this study, we provide some discussion regarding the use of video-based self-avatars and some reflections on the evaluation methodology. The proposed E2E solution is in the boundary of the state-of-the-art, so there is still room for improvement before it reaches maturity. However, this solution serves as a crucial starting point for MR applications where users can feel immersed and interact with their own bodies.

Effectiveness of computer-based interventions for community-dwelling people with cognitive decline: a systematic review with meta-analyses

BackgroundCognitive deficits arise with age and can increase the risk for subjective cognitive decline (SCD) and mild cognitive impairment (MCI), which may result in dementia, leading to health problems, care dependency and institutionalization. Computer-based cognitive interventions (CCIs) have the potential to act as important counteraction functions in preserving or improving cognition concomitant to available pharmacological treatment. The aim was to assess the effectiveness of CCIs performed individually with a personal or tablet computer, game console, virtual, augmented, or mixed reality application on cognition in community-dwelling people with SCD, MCI and dementia.MethodsA systematic review with meta-analyses of randomized controlled trials (RCTs) was performed. The systematic literature search was conducted in MEDLINE, CINAHL, Embase, Cochrane CENTRAL, IEEE Xplore Digital Library, Web of Science, Scopus and PsycINFO. In addition, a search for gray literature and backward citation searching were carried out. To judge on the evidence, two reviewers independently used the Cochrane Risk of Bias Tool. The standardized mean difference (SDM) for pooling comparable studies using the random-effects model was applied.ResultsTwenty-four RCTs were identified, of which 1 RCT examined CCIs in individuals with SCD, 18 RCTs with MCI, and 6 RCTs with dementia. Most interventions were conducted with personal computers. Meta-analyses with 12 RCTs showed significant effects of computer-based cognitive interventions for people with MCI in the domains memory, working memory, attention/concentration/processing speed and executive functioning, but no significant improvements in global cognition and language. Regarding dementia a meta-analysis pooled with 4 RCTs demonstrated a tendency towards, but no significant increase of memory functions (SMD 0.33, CI 95% [-0.10, 0.77]). One RCT regarding SCD reported significant improvements in memory functions for participants conducting a cognitive training on a personal computer.ConclusionsThe results demonstrated that CCIs have beneficial effects on domain-specific cognition in people with MCI but no significant effects on people with dementia. In terms of SCD, one study showed significant improvements in memory functions. It seems that the beneficial effect for cognitive preservation or improvement due to CCIs occurs at the earliest intervention state. However, more research on SCD is needed.Trial registrationPROSPERO International Prospective Register of Systematic Reviews CDR42020184069.

Passthrough Mixed Reality With Oculus Quest 2: A Case Study on Learning Piano

Mixed reality (MR) in standalone headsets has many advantages over other types of devices. With the recent appearance of the Passthrough of Oculus Quest 2, new possibilities open up. This work details the features of the current Passthrough and how its potential was harnessed and its drawbacks minimized for developing a satisfying MR experience. It has been applied to learning to play the piano as a use case. A total of 33 piano students participated in a study to compare participants' interpretation outcomes and subjective experience when using a MR application for learning piano with two visualization modes (border lines on all the keys (Wireframe) versus solid color hiding the real keys (Solid)). The two visualization modes provided a satisfying experience. Even though there were no significant differences in the analyzed variables, the students preferred the Solid mode, indicating that short-distance Passthrough limitations should be minimized in application development.

Mixed Reality Platforms in Telehealth Delivery: Scoping Review

Background The distinctive features of the digital reality platforms, namely augmented reality (AR), virtual reality (VR), and mixed reality (MR) have extended to medical education, training, simulation, and patient care. Furthermore, this digital reality technology seamlessly merges with information and communication technology creating an enriched telehealth ecosystem. This review provides a composite overview of the prospects of telehealth delivered using the MR platform in clinical settings. Objective This review identifies various clinical applications of high-fidelity digital display technology, namely AR, VR, and MR, delivered using telehealth capabilities. Next, the review focuses on the technical characteristics, hardware, and software technologies used in the composition of AR, VR, and MR in telehealth. Methods We conducted a scoping review using the methodological framework and reporting design using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. Full-length articles in English were obtained from the Embase, PubMed, and Web of Science databases. The search protocol was based on the following keywords and Medical Subject Headings to obtain relevant results: “augmented reality,” “virtual reality,” “mixed-reality,” “telemedicine,” “telehealth,” and “digital health.” A predefined inclusion-exclusion criterion was developed in filtering the obtained results and the final selection of the articles, followed by data extraction and construction of the review. Results We identified 4407 articles, of which 320 were eligible for full-text screening. A total of 134 full-text articles were included in the review. Telerehabilitation, telementoring, teleconsultation, telemonitoring, telepsychiatry, telesurgery, and telediagnosis were the segments of the telehealth division that explored the use of AR, VR, and MR platforms. Telerehabilitation using VR was the most commonly recurring segment in the included studies. AR and MR has been mainly used for telementoring and teleconsultation. The most important technical features of digital reality technology to emerge with telehealth were virtual environment, exergaming, 3D avatars, telepresence, anchoring annotations, and first-person viewpoint. Different arrangements of technology—3D modeling and viewing tools, communication and streaming platforms, file transfer and sharing platforms, sensors, high-fidelity displays, and controllers—formed the basis of most systems. Conclusions This review constitutes a recent overview of the evolving digital AR and VR in various clinical applications using the telehealth setup. This combination of telehealth with AR, VR, and MR allows for remote facilitation of clinical expertise and further development of home-based treatment. This review explores the rapidly growing suite of technologies available to users within the digital health sector and examines the opportunities and challenges they present.

The Application of Mixed Reality in Root Canal Treatment

The priority of modern dentistry is to keep patients’ teeth for as long as possible. Tooth extraction is a procedure performed as a last resort when conservative methods and endodontic surgery procedures have not brought the expected results. As a consequence, the number of patients in dental offices, who require first and repeated endodontic treatment, is increasing. Thus, the development of new technologies in the medical industry, including microscopy, computer tomography (CT), as well as diode and neodymium-YAG-erbium lasers, enables dentists to increase the percentage of successful treatments. Moreover, mixed reality (MR) is a very new technology, in which the 3D view can help plan or simulate various types of tasks before they will be carried out in real life. In dentistry, 3D holography can be applied to display CT data to plan endodontic treatment. The most important element in effective root canal treatment is the precise imaging of the root canal. The CT scans allow dentists to view the anatomy of the patient’s tooth with much higher precision and understanding than using 2D radiography (RTG-radiographic photo) pictures. Recently, the development of new 3D technologies allows dentists to obtain even more data from existing CT scans. In this paper, the CT scan data were applied to generate patient teeth in 3D and simulate the view of the root canal’s anatomy in MR devices, i.e., Microsoft HoloLens 2. Using DICOM RAW data from the CT exam, we generated a 3D model of the jaw with a tooth. In the next step, the crown of the tooth was removed in a similar way to how a dentist would do this using a dental handpiece. Furthermore, all root canals were cleaned of everything inside. This way we achieved empty tunnels, namely root canals. Finally, we added appropriate lighting, similar to the type of lighting that dentists use. The proposed approach enables to display of the root canals in the same way as during the endodontic procedure using a microscope. It allows for the visualization of the root canal and changing its direction, in which dimensional accuracy is crucial. It turns out that mixed reality can be considered a complementary method to the traditional approach, which reduces the amount of time for the root canal treatment procedure by up to 72.25%, depending on the complexity of the case, and increases its effectiveness. Thus, the mixed reality-based system can be considered an effective tool for planning dental treatment.

Extended reality (XR) technologies in the construction safety: systematic review and analysis

PurposeExtended reality (XR) is an emerging technology, with its popularity rising in different industry sectors, where its application has been recently considered in construction safety. This study aims to investigate the applications of XR technologies in the safety of construction through projects lifecycle perspective.Design/methodology/approachScientometric analysis was conducted to discover trends, keywords, contribution of countries and publication outlets in the literature. The content analysis was applied to categorize previous studies into three groups concerning the phase of lifecycle in which they used XR.FindingsResults of the content analysis showed that the application of XR in the construction safety is mostly covered in two areas, namely, safety training and risk management. It was found that virtual reality was the most used XR tool with most of its application dedicated to safety training in the design phase. The amount of research on the application of augmented reality and mixed reality in safety training, and risk management in all phases of lifecycle is still insignificant. Finally, this study proposed three main areas for using the XR technologies regarding the safety issues in future research, namely, control of safety regulations and safety coordination in construction phase, and safety reports in the operation phase.Originality/valueThis paper inspected the utilization of all types of XR for safety in each phase of construction lifecycle and proposed future directions for research by addressing the safety challenges in each phase.

마커 기반 상호작용 동기화 및 혼합현실 협업 콘텐츠 개발

As non-face-to-face services spread in the post-COVID-19, expanded reality technologies such as mixed reality are attracting attention in various fields. In mixed reality, each user forms a unique coordinate system with the device location at the time the application is executed as the origin. The positions of virtual objects thus cannot be shared or synchronized by using absolute coordinates. In particular, for a co-located collaboration, virtual objects augmented on each user’s device should exist exactly at the same spot in the real world, which means the objects rendered on each user’s display should be aligned in the same position and rotation in the real world as well as synchronized. For this, we propose a method for creating a standard coordinate system for a collaborative workspace and synchronizing interactions. All users create a standard coordinate system at the location of a marker, and then register virtual objects and avatars as its sub-objects. Afterwards interactions in the collaborative space are synchronized among users by exchange of local position relative to the standard coordinate system. We developed the mixed reality application in which multiple users collaborate for 3D-drone assembly, and verified our proposal by showing that it synchronizes interactions in real-time successfully.

PAV-SOD: A New Task towards Panoramic Audiovisual Saliency Detection

Object-level audiovisual saliency detection in 360° panoramic real-life dynamic scenes is important for exploring and modeling human perception in immersive environments, also for aiding the development of virtual, augmented, and mixed reality applications in fields such as education, social network, entertainment, and training. To this end, we propose a new task, p anoramic a udio v isual s alient o bject d etection, ( PAV-SOD 1 ), which aims to segment the objects grasping most of the human attention in 360° panoramic videos reflecting real-life daily scenes. To support the task, we collect PAVS10K , the first p anoramic video dataset for a udio v isual s alient object detection, which consists of 67 4K-resolution equirectangular videos with per-video labels including hierarchical scene categories and associated attributes depicting specific challenges for conducting PAV-SOD , and 10,465 uniformly sampled video frames with manually annotated object-level and instance-level pixel-wise masks. The coarse-to-fine annotations enable multi-perspective analysis regarding PAV-SOD modeling. We further systematically benchmark 13 state-of-the-art salient object detection (SOD)/video object segmentation (VOS) methods based on our PAVS10K . Besides, we propose a new baseline network, which takes advantage of both visual and audio cues of 360° video frames by using a new conditional variational auto-encoder (CVAE). Our C VAE-based a udio v isual net work, namely, CAV-Net , consists of a spatial-temporal visual segmentation network, a convolutional audio-encoding network, and audiovisual distribution estimation modules. As a result, our CAV-Net outperforms all competing models and is able to estimate the aleatoric uncertainties within PAVS10K . With extensive experimental results, we gain several findings about PAV-SOD challenges and insights towards PAV-SOD model interpretability. We hope that our work could serve as a starting point for advancing SOD towards immersive media.

Adopting metaverse‐related mixed reality technologies to tackle urban development challenges: An empirical study of an Australian municipal government

Fast-growing urban areas bring challenges to government authorities. The urban planning system in Australia is still very analogue with paper-based systems as standard practice. Two-dimensional document-based blueprints and Geographic Information System (GIS) maps do not always give stakeholders and investors the necessary insights. By exploring Mixed Reality (MR) application at the municipal government level, this research explains how metaverse-related immersive technologies can assist the government in better engaging with stakeholders and potential partners to improve their understanding of the urban environment and showcase future city developments to foster new opportunities and contribute to the development of the real living world.

Dynamic Mixed Reality Assembly Guidance Using Optical Recognition Methods

Augmented (AR) and Mixed Reality (MR) technologies are enablers of the Industry 4.0 paradigm and are spreading at high speed in production. Main applications include design, training, and assembly guidance. The latter is a pressing concern, because assembly is the process that accounts for the biggest portion of total cost within production. Teaching and guiding operators to assemble with minimal effort and error rates is pivotal. This work presents the development of a comprehensive MR application for guiding novice operators in following simple assembly instructions. The app follows innovative programming logic and component tracking in a dynamic environment, providing an immersive experience that includes different guidance aids. The application was tested by experienced and novice users, data were drawn from the performed experiments, and a questionnaire was submitted to collect the users’ perception. Results indicate that the MR application was easy to follow and even gave confidence to inexperienced subjects. The guidance support was perceived as useful by the users, though at times invasive in the field of view. Further development effort is required to draw from this work a complete and usable architecture for MR application in assembly, but this research forms the basis to achieve better, more consistent instructions for assembly guidance based on component tracking.

„Mixed Reality“ – erste Erfahrungen aus der Leberchirurgie

BackgroundDevelopments in imaging technology enable better diagnostics and, thus, also safer visceral interventions. One of the most recent developments is the use of mixed reality in the operating room, a “mixture” of reality with three-dimensional (3D) reconstructed preoperative images. Thus, we present our first experiences with the application of mixed reality in the field of liver surgery.Materials and methodsEight patients (age 62–82 years) with primary liver malignancy (n = 4) or liver metastases (n = 4) were studied and documented in this work. The time required to fully prepare the preoperative reconstructions was analyzed and the benefits of the new technology were graded by the members of the treatment team for each case. We use an open-source software for image analysis and scientific visualization preoperatively.ResultsThe time required to prepare for the operation is considerable and relevant for the everyday clinical practice (150–360 min). We were able to gain more information about tumor location and the relationship to other important intra-abdominal structures preoperatively and intraoperatively through the use of the technology. We have not yet been able to demonstrate a benefit in the area of risk minimization for the patient, faster operation times or faster recovery in this study.ConclusionMixed reality technology offers an enormous range of possible applications. In particular, it has great potential for visceral surgery, patient education, but also medical training. Further investigations are necessary to more precisely assess the advantages and disadvantages in larger case series.

Isolated Points Prediction via Deep Neural Network on Point Cloud Lossless Geometry Compression

The point cloud is one of the indispensable data structures of virtual and mixed reality applications. Vivid scene requirement means that millions of points need to be encoded while transmitting these point clouds. Therefore, a highly efficient point cloud compression (PCC) technology is the key to promote the above applications. In the tree structure geometry partition based PCC framework, nodes of the tree containing isolated points are further subdivided. This sub-division decreases the compression ratio. Furthermore, the location uncertainty of the isolated points makes the above coding very challenging. In this paper, we propose a deep neural network model to predict the isolated points. These point clouds are generated by LiDAR in the autonomous driving applications. The proposed neural network model includes nodes feature extraction modules and parent-child nodes feature aggregation modules. The nodes feature extraction modules extract the hierarchical features from the inputs generated from the current node or its ancestor nodes. The parent-child nodes feature aggregation modules aggregate the hierarchical feature maps of the parent node and the current node. Finally, this proposed neural network model suggests the probability that the current node contains the isolated points. As a result, the nodes containing isolated points will not be further subdivided. The experimental results show that, compared with the latest isolated points prediction method, the accuracy of the proposed isolated points prediction neural network model is much higher on average. Compared with the state-of-the-art, the proposed model can achieve 5.29% bpip coding gain on average on lossless geometry point cloud compression.

Mixed Reality or Simply Mobile? A Case Study on Enabling Less Skilled Workers to Perform Routine Maintenance Tasks

The availability of skilled workers has become a severe issue for production companies in developed countries like Germany in recent years. Specialists for maintenance tasks are in strong demand and their workforce should be steered towards the most challenging tasks in maintenance. In this case study we analyze, whether less skilled or not specifically instructed workers can perform routine maintenance tasks without being directly supported by experts. In our setup ten participants try to handle a maintenance routine checklist in three different ways: paper-based printouts, on a mobile tablet, and with a Microsoft HoloLens 2 Mixed Reality (MR) device. The results show that workers fail to complete the task with a mere paper-based approach, but are able to succeed both with the mobile tablet and MR application. Quantitative time measurements and qualitative analyses indicate that the mobile tablet app was easier to learn, did not impair the worker as much as the HoloLens and worked more stable. However, the HoloLens solution showed advantages as hands-free and spatial guiding device, which could be important aspects when better MR hardware will be available and MR interaction principles become common knowledge.

Outcomes of VR, AR and MR Technologies in K-12 Language Education: A Review

Since the 21st century, Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) technologies have been increasingly used in the field of language education to promote student motivation, engagement, and so on. This paper reviews the results of 180 reviews of empirical studies on the use of VR, AR, and MR to enhance K-12 teaching or learning from 2012 to 2022 and summarizes the trends. Based on Bloom’s classification theory of instructional goals, namely Cognitive Goals, Affective Goals, and Behavioral Goals, the outcomes of 53 papers are summarized and analyzed from these three dimensions to find the function of VR, AR, and MR in K-12 Language Education, which can improve literacy, creative thinking, communication, collaboration, confidence, participation, and enjoyment in the digital era. And it also promotes self-learning, enables multisensory learning, and reduces cognitive load. Despite the technological complexity and problems associated with the use of AR in education, VR, AR, and MR applications can successfully improve language learning. Compared with previous research approaches, this study first constructs a VR, AR, and MR-supported learning model based on their characteristics respectively and the relevance of these technologies.

A Mixed Reality application to support the design of custom prostheses

The definition of an innovative category of implantable devices, characterized not only by a fully customization but also by improved osteoconductive and functionalized bioactive surfaces, needs to be supported by a systematic approach. In the present work, a new human-machine interface based on the Mixed Reality (MR) is proposed and focused on the implantation of a fully customized prosthesis. By means of an informal and exploratory focus group (i.e., without using a structured questionnaire), the limitations belonging to the current procedure were highlighted and a first list of user needs was subsequently defined. The MR interface was then considered to be the most suitable solution to match the gathered requirements. However, the doctors proposed also to develop a desktop interface for a finer and easier manipulation of 3D models. The proposed MR application offers several advantages from the possibility to display 3D anatomical structures and 3D models of custom prostheses in an immersive environment to the optimized communication and data exchange among the players (medical staff, doctors and engineers). A mock-up of the MR applications is presented in this work to show the results of the design stage, before the deployment of the application.

How to provide work instructions to reduce the workers’ physical and mental workload

The transition to Industry 5.0 emphasizes the human-centric perspective and pushes for ergonomic risks prevention and mitigation. The diffusion of the production paradigms of mass customization and mass personalization also increases the need for methods and tools to support workers in manufacturing activities. The growing number of product variants requires a higher cognitive load and increases the likelihood of human errors. Work instructions become fundamental, as well as their digitalization. The technologies enabling industry 4.0 have opened up new scenarios for the use of this information, from extended reality to digital assistance systems. This work deals with the human-centered redesign of a workstation for the wire harness assembly with the aim of mitigating the ergonomic risks for operators, both physical and cognitive. Four different scenarios for using the work instructions were implemented and tested: paper-based instructions, workstation display, mixed reality application usable with a head-mounted display, and augmented reality application that only requires a projection system. The experimentation involved user experience evaluation, performance assessment, and objective postural analysis through inertial sensors.

Magnetic resonance imaging based neurosurgical planning on hololens 2: A feasibility study in a paediatric hospital.

The goal of this work is to show how to implement a mixed reality application (app) for neurosurgery planning based on neuroimaging data, highlighting the strengths and weaknesses of its design. Our workflow explains how to handle neuroimaging data, including how to load morphological, functional and diffusion tensor imaging data into a mixed reality environment, thus creating a first guide of this kind. Brain magnetic resonance imaging data from a paediatric patient were acquired using a 3 T Siemens Magnetom Skyra scanner. Initially, this raw data underwent specific software pre-processing and were subsequently transformed to ensure seamless integration with the mixed reality app. After that, we created three-dimensional models of brain structures and the mixed reality environment using Unity™ engine together with Microsoft® HoloLens 2™ device. To get an evaluation of the app we submitted a questionnaire to four neurosurgeons. To collect data concerning the performance of a user session we used Unity Performance Profiler. The use of the interactive features, such as rotating, scaling and moving models and browsing through menus, provided by the app had high scores in the questionnaire, and their use can still be improved as suggested by the performance data collected. The questionnaire's average scores were high, so the overall experiences of using our mixed reality app were positive. We have successfully created a valuable and easy-to-use neuroimaging data mixed reality app, laying the foundation for more future clinical uses, as more models and data derived from various biomedical images can be imported.

Revisiting natural user interaction in virtual world.

Natural user interaction in virtual environment is a prominent factor in any mixed reality applications. In this paper, we revisit the assessment of natural user interaction via a case study of a virtual aquarium. Viewers with the wearable headsets are able to interact with virtual objects via head orientation, gaze, gesture, and visual markers. The virtual environment is operated on both Google Cardboard and HoloLens, the two popular wireless head-mounted displays. Evaluation results reveal the preferences of users over different natural user interaction methods.