Augmented Reality Visualization Research Articles

Development of Augmented Reality Vision for Osteosynthesis Using a 3D Camera

Development of Augmented Reality Vision for Osteosynthesis Using a 3D Camera

Current state of the art and future directions: Augmented reality data visualization to support decision-making

Augmented Reality (AR), as a novel data visualization tool, is advantageous in revealing spatial data patterns and data-context associations. Accordingly, recent research has identified AR data visualization as a promising approach to increasing decision-making efficiency and effectiveness. As a result, AR has been applied in various decision support systems to enhance knowledge conveying and comprehension, in which the different data-reality associations have been constructed to aid decision-making.However, how these AR visualization strategies can enhance different decision support datasets has not been reviewed thoroughly. Especially given the rise of Big Data in the modern world, this support is critical to decision-making in the coming years. Using AR to embed the decision support data and explanation data into the end user’s physical surroundings and focal contexts avoids isolating the human decision-maker from the relevant data. Integrating the decision-maker’s contexts and the DSS support in AR is a difficult challenge. This paper outlines the current state of the Art through a literature review in allowing AR data visualization to support decision-making.To facilitate the publication classification and analysis, the paper proposes one taxonomy to classify different AR data visualization based on the semantic associations between the AR data and physical context. Based on this taxonomy and a decision support system taxonomy, 37 publications have been classified and analyzed from multiple aspects. One of the contributions of this literature review is a resulting AR visualization taxonomy that can be applied to decision support systems. Along with this novel tool, the paper discusses the current state of Art in this field and indicates possible future challenges and directions that AR Data Visualization will bring to support decision-making.

Augmented Reality Visualization and Quantification of COVID-19 Infections in the Lungs

The ongoing COVID-19 pandemic has had a significant impact globally, and the understanding of the disease’s clinical features and impacts remains insufficient. An important metric to evaluate the severity of pneumonia in COVID-19 is the CT Involvement Score (CTIS), which is determined by assessing the proportion of infections in the lung field region using computed tomography (CT) images. Interactive augmented reality visualization and quantification of COVID-19 infection from CT allow us to augment the traditional diagnostic techniques and current COVID-19 treatment strategies. Thus, in this paper, we present a system that combines augmented reality (AR) hardware, specifically the Microsoft HoloLens, with deep learning algorithms in a user-oriented pipeline to provide medical staff with an intuitive 3D augmented reality visualization of COVID-19 infections in the lungs. The proposed system includes a graph-based pyramid global context reasoning module to segment COVID-19-infected lung regions, which can then be visualized using the HoloLens AR headset. Through segmentation, we can quantitatively evaluate and intuitively visualize which part of the lung is infected. In addition, by evaluating the infection status in each lobe quantitatively, it is possible to assess the infection severity. We also implemented Spectator View and Sharing a Scene functions into the proposed system, which enable medical staff to present the AR content to a wider audience, e.g., radiologists. By providing a 3D perception of the complexity of COVID-19, the augmented reality visualization generated by the proposed system offers an immersive experience in an interactive and cooperative 3D approach. We expect that this will facilitate a better understanding of CT-guided COVID-19 diagnosis and treatment, as well as improved patient outcomes.

Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgery.

Depth estimation in robotic surgery is vital in 3D reconstruction, surgical navigation and augmented reality visualization. Although the foundation model exhibits outstanding performance in many vision tasks, including depth estimation (e.g., DINOv2), recent works observed its limitations in medical and surgical domain-specific applications. This work presents a low-ranked adaptation (LoRA) of the foundation model for surgical depth estimation. We design a foundation model-based depth estimation method, referred to as Surgical-DINO, a low-rank adaptation of the DINOv2 for depth estimation in endoscopic surgery. We build LoRA layers and integrate them into DINO to adapt with surgery-specific domain knowledge instead of conventional fine-tuning. During training, we freeze the DINO image encoder, which shows excellent visual representation capacity, and only optimize the LoRA layers and depth decoder to integrate features from the surgical scene. Our model is extensively validated on a MICCAI challenge dataset of SCARED, which is collected from da Vinci Xi endoscope surgery. We empirically show that Surgical-DINO significantly outperforms all the state-of-the-art models in endoscopic depth estimation tasks. The analysis with ablation studies has shown evidence of the remarkable effect of our LoRA layers and adaptation. Surgical-DINO shed some light on the successful adaptation of the foundation models into the surgical domain for depth estimation. There is clear evidence in the results that zero-shot prediction on pre-trained weights in computer vision datasets or naive fine-tuning is not sufficient to use the foundation model in the surgical domain directly.

Detection and analysis of deteriorated areas in solar PV modules using unsupervised sensing algorithms and 3D augmented reality

Solar Photovoltaic (PV) systems are increasingly vital for enhancing energy security worldwide. However, their efficiency and power output can be significantly reduced by hotspots and snail trails, predominantly caused by cracks in PV modules. This article introduces a novel methodology for the automatic segmentation and analysis of such anomalies, utilizing unsupervised sensing algorithms coupled with 3D Augmented Reality (AR) for enhanced visualization. The methodology outperforms existing segmentation techniques, including Weka and the Meta Segment Anything Model (SAM), as demonstrated through computer simulations. These simulations were conducted using the Cali-Thermal Solar Panels and Solar Panel Infrared Image Datasets, with evaluation metrics such as the Jaccard Index, Dice Coefficient, Precision, and Recall, achieving scores of 0.76, 0.82, 0.90, 0.99, and 0.76, respectively. By integrating drone technology, the proposed approach aims to revolutionize PV maintenance by facilitating real-time, automated solar panel detection. This advancement promises substantial cost reductions, heightened energy production, and improved performance of solar PV installations. Furthermore, the innovative integration of unsupervised sensing algorithms with 3D AR visualization opens new avenues for future research and development in the field of solar PV maintenance.

AR EXPLORATION OF THE PERSPECTIVE/ARCHITECTURAL SPACE PAINTED BY PERUGINO

Abstract. The perspective study of one of the panels from the cycle of San Bernardino attributed to Perugino has revealed a contracted space in depth, represented three-dimensionally by the objective model. In contrast, the ideal model eliminates the contraction and maintains the proportions dictated by the architectural interpretation. The contribution presents an advancement in the perspective study aimed at developing an augmented reality (AR) application. It is designed to be experienced at the museum housing the artwork, serving as an in-depth exploration that explains to visitors the nature of the perspective space and the architectural space represented. The case study of the painting by Perugino becomes an opportunity to define and experiment with a workflow aimed at creating AR applications focused on the visualisation of architectural models derived from the perspective restitution of pictorial works. The description of the adopted method focuses on the following aspects: the use of the pictorial image as an AR marker and the relationship between the perspective picture plane and the 3D model; the optimisation of 3D models for AR visualisation purposes; the issue of the landscape; scene lighting; and the user interface for multilayer interaction.

AR-Assisted Guidance for Assembly and Maintenance of Avionics Equipment

The assembly and maintenance of products in the aviation industry constitute a crucial aspect of the product life cycle, with numerous tasks still reliant on manual operations. In order to solve the problem of narrow operation spaces and blind areas in the processes of manual assembly and maintenance, we proposed an augmented reality (AR) assistant guidance method specifically designed for such scenarios. By employing a multi-modality anti-occlusion tracking algorithm, pose data of assembly parts can be obtained, upon which AR guidance information is displayed. Additionally, we proposed an assembly step identification method to alleviate user interaction pressure. We developed an AR visualization assistant guidance system and designed and conducted a user evaluation experiment to measure the learnability, usability, and mental effort required. The results demonstrate that our method significantly enhances training efficiency by 128.77%, as well as improving assembly and maintenance efficiency by 29.53% and 27.27% compared with traditional methods. Moreover, it has significant advantages in learnability, usability, and mental effort, providing a feasible and effective resolution for addressing blind areas during assembly and maintenance within the aviation industry.

HoloBrain: 3D low-cost mobile augmented reality rendering of brain tumour using the GVF snake model segmentation

ABSTRACT Augmented reality (AR) is a dynamic field intersecting various disciplines, with applications spanning medicine and diagnostics. Our focus on advancing medical imaging has led to groundbreaking progress in brain tumor diagnosis, fostering optimism in treatment. In this pursuit, we introduce HoloBrain, an accessible AR application designed to transform medical education. Utilizing smartphones, HoloBrain enables automatic segmentation, AR visualization, and 3D interaction without the need for head-mounted devices. Emphasizing practicality over hyper-realism, our innovation allows medical students to seamlessly integrate a virtual brain tumor model into their surroundings. This immersive approach facilitates exploration of the 3D representation of an afflicted brain, enhancing educational experiences. Tests in a controlled environment using a phantom head yielded promising results, with our solution demonstrating average errors of 2 mm to 4.5 mm on smartphone displays. These outcomes highlight the precision and effectiveness of our contribution, underlining its potential in transforming medical education and training. HoloBrain represents a step forward in making AR accessible and impactful for aspiring medical professionals, offering a valuable tool for enhancing their understanding of complex medical scenarios.

Teaching chemistry in the metaverse: the effectiveness of using virtual and augmented reality for visualization

IntroductionThe metaverse is a promising platform for learning that all participants in the teaching and learning process may find appealing. This study aims to investigate the effectiveness of using metaverse-related visualization technologies, such as Virtual Reality (VR) and Augmented Reality (AR), in teaching chemistry.MethodsThe study has an experimental research design and uses the following tools to collect data: Dombrovskaya’s Motivation for Learning Scale, the SAMOAL Questionnaire, and the Self-management Questionnaire. The study sample includes 236 first-year undergraduate students studying chemistry at the Faculty of Natural Sciences.ResultsResults suggest that the integration of immersive VR/AR technologies in chemistry education signifies the beginning of a metaverse transition; the process, however, is in its early stages.DiscussionThe study concludes that immersive VR/AR technologies have an advantage over traditional programs in terms of motivation and competence development. Future research can investigate the impact of VR/AR technologies on students’ anxiety and build a model for predicting student achievements in the metaverse-based chemistry classroom.

An Augmented Reality Visualization System for Simulated Multirotor Aerial Vehicles

An Augmented Reality Visualization System for Simulated Multirotor Aerial Vehicles

Enhancing Multi-Modal Perception and Interaction: An Augmented Reality Visualization System for Complex Decision Making

Visualization systems play a crucial role in industry, education, and research domains by offering valuable insights and enhancing decision making. These systems enable the representation of complex workflows and data in a visually intuitive manner, facilitating better understanding, analysis, and communication of information. This paper explores the potential of augmented reality (AR) visualization systems that enhance multi-modal perception and interaction for complex decision making. The proposed system combines the physicality and intuitiveness of the real world with the immersive and interactive capabilities of AR systems. By integrating physical objects and virtual elements, users can engage in natural and intuitive interactions, leveraging multiple sensory modalities. Specifically, the system incorporates vision, touch, eye-tracking, and sound as multi-modal interaction methods to further improve the user experience. This multi-modal nature enables users to perceive and interact in a more holistic and immersive manner. The software and hardware engineering of the proposed system are elaborated in detail, and the system’s architecture and preliminary function testing results are also included in the manuscript. The findings aim to aid visualization system designers, researchers, and practitioners in exploring and harnessing the capabilities of this integrated approach, ultimately leading to more engaging and immersive user experiences in various application domains.

Augmented reality guided autonomous assembly system: A novel framework for assembly sequence input validations and creation of virtual content for AR instructions development

In recent years, industries are striving to adapt immersive technologies in the real-time assembly shop floor setting to achieve effective outcomes in terms of enabling the worker to handle the complex cognitive assembly task execution/training circumstances. The manufacturers are presently facing difficulties with the process of Augmented reality (AR) instruction creation for the required product assembly system. The existing AR instruction development process demands highly skilled experts and more time consumption. Hence, the manufacturers are presently relying on third parties to acquire the required AR instructions by providing the necessary assembly particulars as input. Where, the cost of developing AR instruction is another big challenging concern. Predominantly, the significance of providing exact assembly input is vital for the experts to develop the appropriate AR content without any errors. An automated approach is developed in this study to resolve the difficulties with assembly sequence input for the AR instructions development process. The automated approach affords four different provisions that include validation of an assembly sequence input prior to the AR instructions development stage, suggestions for assembly sequence input generation, simulation of validated assembly sequence in the Virtual reality (VR) environment for user perception, and generation of virtual content for AR visualizations. Subsequently, the generated digital contents from the automated approach are imparted into the AR Unity platform to develop the required AR instructions for the product assembly system. Notably, this study explores the AR customized marker technique to visualize the AR content in the user’s view of the physical context. Finally, the AR guidelines are built as apps for the selected 11-part assembly and deployed on the shop floor for the user to perceive the AR content through visualization devices for execution of assembly operations in a real-time context. In addition, the effectiveness of the automated approach is analyzed with the existing approach, and the consequences are explored through the process of receiving feedback from the experts about the error rate, cognitive level, and creation time.

A 3D visualization‐based augmented reality application for brain tumor segmentation

SummaryEvery year on June 8th, the globe observes World Brain Tumor Day to raise awareness and educate people about brain cancer, encompassing both noncancerous (benign) and cancerous (malignant) growths. Research in the field of brain cancer plays a vital role in supporting medical professionals. In this context, augmented reality (AR) technology has emerged as a valuable tool, enabling surgeons to visualize underlying structures and offering a cost‐effective and time‐efficient alternative. Our study focuses on the efficient segmentation of brain tumor classes using Magnetic Resonance Imaging (MRI) and incorporates a three‐stage approach: preprocessing, segmentation, and 3D reconstruction & AR display. In the preprocessing stage, a Gaussian filter is applied to mitigate intensity heterogeneity. Segmentation and detection are achieved using active geometric contour models, complemented by morphological operations. To establish 3D brain tumor reconstruction, a genuine scene is virtually integrated using 3D Slicer software. The proposed methodology was validated using a genuine patient dataset comprising 496 MRI scans obtained from the local Bab El Oued university hospital center. The results demonstrate the effectiveness of our approach in achieving accurate 3D brain tumor reconstruction, efficient tumor extraction, and augmented reality visualization. The obtained segmentation results showcased an impressive accuracy of 98.61%, outperforming existing state‐of‐the‐art methods and affirming the efficacy of our proposed strategy.

Integration of Iot and NDT for real Time Monitoring System

For the findings of non-destructive testing on buildings have only been provided in specific cases. To facilitate all-encompassing quality control throughout the construction's entire lifespan, it might be ideal to integrate the different findings, track their evolution across duration, and apply them when performing on-the-spot maintenance duties. Under the context of building information modelling (BIM), opportunities are established throughout this achievement, enabling the integration of non-destructive testing findings through digital building models. An inventive augmented reality (AR) visualization technique is shown as an illustration of the combined use of planned and measured results. An augmented reality application is being put into place that overlays the camera picture of a tablet looking at a piece of concrete using images captured by radar and ultrasonic sensors of the sample's interior (actual data) and the three-dimensional design layout of the integrated pieces (target data). The geometric relationship between the inner images and the camera image remains intact whenever the device turns or shifts. Therefore, the device's display provides a view inside the concrete structure's interior. The procedures required to prepare and evaluate the findings of non-destructive testing have been collected, and potential markers for linking the physical and digital realms have been explored. The objective is to assess the state, enable model-based examination, and upkeep operations instantly within the framework by enhancing the preparation of data as well as evaluation outcomes with extra data and displaying the findings in their actual situations.

Heads-Up Micronavigation Reliability of Preoperative Transcranial Magnetic Stimulation Maps for the Motor Function: Comparison With Direct Cortical Stimulation.

We aimed to assess the reliability of preoperative navigated transcranial magnetic stimulation (nTMS) maps for motor function as visualized intraoperatively with augmented reality heads-up display and to assess its accuracy via direct point-by-point comparison with the gold-standard direct cortical stimulation (DCS). From January 2022 to January 2023, candidates for surgical removal of lesions involving the motor pathways underwent preoperative nTMS assessment to obtain cortical maps of motor function. Intraoperatively and before tumor removal, nTMS maps were superimposed on the cortical surface, and DCS was performed on positive points with increasing current intensity until obtaining a positive response at 16 mA. The outcome of each stimulation was recorded to obtain discrimination metrics. Twelve patients were enrolled (5 females [42%] vs 7 males [58%], mean age 62.9 ± 12.8 years), for a total of 304 investigated points. Agreement between nTMS and DCS was moderate (κ = 0.43, P < .005), with 0.66 (0.53-0.78) sensitivity, 0.87 (0.82-0.90) specificity, 0.50 (0.39-0.62) positive predictive values, 0.93 (0.89-0.95) negative predictive value, and 0.83 (0.79-0.87) accuracy. A loss of accuracy was observed with higher DCS current intensities. We performed a point-by-point validation of preoperative nTMS maps for motor function using augmented reality visualization. The high negative predictive value and low positive predictive values highlight nTMS reliability to visualize safe cortical zones but not to identify critical functional areas, confirming previous findings of nTMS maps for the language function and suggesting the need for combined use of nTMS maps and DCS for optimal maximal safe resection.

Augmented Reality-Based BIM Data Compatibility Verification Method for FAB Digital Twin implementation

With the advancement of state-of-the-art technologies, the semiconductor industry plays a key role as an essential component in the manufacture of various electronic products. Since the manufacturing of a semiconductor goes through very sophisticated and complex processes, efficient and accurate work and management are essential in the design, construction, and operation stages of the semiconductor fabrication (FAB) plant. Recently, the combined application of building information modeling (BIM) and augmented reality (AR) technology has gained increased attention in this semiconductor FAB industry as an advanced way to improve work efficiency and accuracy while eliminating other related problems, such as human errors. Despite the perceived benefits of combined use of BIM and AR, many technical problems still exist when integrating the target test model and the 3D virtual object model using BIM data and existing AR visualization technology, due to the unique characteristics of the FAB sites. To solve these problems, this study proposed an AR-based real-time BIM data compatibility verification method for future FAB digital twin implementation and demonstrated that it could be converted into a system and applied to actual FAB sites. As a result of the development and verification of this system, the proposed AR-based real-time BIM data compatibility verification system enables the accurate fitting of the AR model and actual object through AR tracking and anchoring technology considering the characteristics of FAB sites. After the fitting, the system was able to maintain compatibility, even when the camera moved and the marker moved away from the screen. By expanding the effective distance of compatibility between the AR model and the actual object, it was possible to increase the AR application range between the 3D virtual object model and the test target model and to improve the compatibility.

Exploring Deep Cervical Compartments in Head and Neck Surgical Oncology through Augmented Reality Vision: A Proof of Concept.

Virtual surgical planning allows surgeons to meticulously define surgical procedures by creating a digital replica of patients' anatomy. This enables precise preoperative assessment, facilitating the selection of optimal surgical approaches and the customization of treatment plans. In neck surgery, virtual planning has been significantly underreported compared to craniofacial surgery, due to a multitude of factors, including the predominance of soft tissues, the unavailability of intraoperative navigation and the complexity of segmenting such areas. Augmented reality represents the most innovative approach to translate virtual planning for real patients, as it merges the digital world with the surgical field in real time. Surgeons can access patient-specific data directly within their field of view, through dedicated visors. In head and neck surgical oncology, augmented reality systems overlay critical anatomical information onto the surgeon's visual field. This aids in locating and preserving vital structures, such as nerves and blood vessels, during complex procedures. In this paper, the authors examine a series of patients undergoing complex neck surgical oncology procedures with prior virtual surgical planning analysis. For each patient, the surgical plan was imported in Hololens headset to allow for intraoperative augmented reality visualization. The authors discuss the results of this preliminary investigation, tracing the conceptual framework for an increasing AR implementation in complex head and neck surgical oncology procedures.

Immersive Unit Visualization with Augmented Reality

Immersive Unit Visualization is an emergent form of visualization that arose from Immersive Analytics where, unlike traditional visualizations, each data point is represented by an individual visual mark in an immersive virtual environment. This practice has focused almost exclusively on virtual reality, excluding augmented reality (AR). This article develops and tests a prototype of an Immersive Unit Visualization (Floating Companies II) with two AR devices: head-mounted display (HMD) and hand-held display (HHD). Results from the testing sessions with 20 users were analyzed through qualitative research analysis and thematic coding indicating that, while the HHD enabled a first contact with AR visualization on a familiar device, HMD improved the perception of hybrid space by supporting greater stability of virtual content, wider field of view, improved spatial perception, increased sense of immersion, and more realistic simulation, which had an impact on information reading and sense-making. The materialization of abstract quantitative values into concrete reality through its simulation in the real environment and the ludic dimension stand out as important opportunities for this type of visualization. This paper investigates the aspects distinguishing two experiences regarding data visualization in hybrid space, and characterizes ways of seeing information with AR, identifying opportunities to advance information design research.

Impact of Virtual Reality (Vr) and Augmented Reality (Ar) in Education

Virtual Reality (VR) and Augmented Reality (AR) technologies have emerged as powerful tools with the potential to revolutionize the field of education. This study provides an overview of the impact of VR and AR in education, drawing from existing research and analytical applications. The impact of VR in education is evident in its ability to create immersive and engaging learning experiences. Students can be transported to virtual environments that facilitate experiential learning, from exploring historical landmarks to simulating complex scientific experiments. The result is enhanced student engagement, improved retention of knowledge, and the development of practical skills. AR, on the other hand, overlays digital information onto the real world, enriching traditional educational materials. Interactive textbooks and guided field trips using AR provide students with dynamic and context-rich learning experiences. Language learners benefit from instant translations and pronunciation guides, while complex data becomes tangible through AR visualizations. Both VR and AR contribute to the promotion of collaborative learning, enabling students and educators to interact in virtual environments. These technologies offer new opportunities for special needs education, as they can be customized to cater to the unique requirements of individual learners. The adoption of VR and AR in education comes with challenges, including cost considerations, the need for suitable technical infrastructure, content development complexities, and the necessity to address safety concerns. Integrating VR and AR into the existing curriculum and ensuring alignment with educational objectives are also ongoing challenges. The study aim is to analyse the impact of VR and AR in education.

Augmented reality aided object mapping for worker assistance/training in an industrial assembly context: Exploration of affordance with existing guidance techniques

In recent years, the affordances of immersive technologies in the industrial context have enabled the onsite worker to have increased potential towards handling the complex procedural and cognitive tasks involved in the strategy of manufacturing an appropriate product. This article describes the potential of Augmented Reality (AR) in terms of upgraded workplace understanding, training, and assisting semi-skilled/new workers in an industrial production environment. The different modes of instruction visualization techniques are investigated for the execution of assembly tasks and explored their impact in accordance to the efficiency of task execution, completion time, error rate perception, cognitive load, and independence. Progress outcomes of 24 participants are explored for real-time execution of a Reverse osmosis (RO) booster pump assembly in terms of different modes of visualization, such as the conventional document method, virtual /desktop mode, and object mapping-based AR. Additionally, the user studies on different visualization devices involving hand-held visualization devices (HHVD) and Head-mounted visualization devices (HMVD) are investigated and validated in terms of adaptability, acceptability, and usability. Furthermore, the effectiveness of the AR-aided object mapping technology versus other methods is explored through the feedback received from both experts and academia participants, which increases the insight of the technology and provides an intelligent mechanism for semi-skilled/new workers to handle complex or challenging operations in the production environment. The studies indicate that object mapping-based AR visualization facilitates increased assembly task efficiency and the minimum number of errors compared to the other guidance modes. Notably, the HMVD affords an increased realism experience, minimized errors, and less completion time compared to the HHVD.