Projector-based Augmented Reality Research Articles

Exploring Embodied Cognition and Brain Dynamics under Multi-tasks Target Detection in Immerse Projector-based Augmented Reality (IPAR) Scenarios.

Embodied cognition explores the intricate interaction between the brain, body, and the surrounding environment. The advancement of mobile devices, such as immersive interactive computing and wireless electroencephalogram (EEG) devices, has presented new challenges and opportunities for studying embodied cognition. To address how mobile technology within immersive hybrid settings affects embodied cognition, we propose a target detection multitask incorporating mixed body movement interference and an environmental distraction light signal. We aim to investigate human embodied cognition in immersive projector-based augmented reality (IPAR) scenarios using wireless EEG technology. We recruited and engaged fifteen participants in four multitasking conditions: standing without distraction (SND), walking without distraction (WND), standing with distraction (SD), and walking with distraction (WD). We pre-processed the EEG data using Independent Component Analysis (ICA) to isolate brain sources and K-means clustering to categorize Independent Components (ICs). Following that, we conducted time-frequency and correlation analyses to identify neural dynamics changes associated with multitasking. Our findings reveal a decline in behavioral performance during multitasking activities. We also observed decreases in alpha and beta power in the frontal and motor cortex during standing target search tasks, decreases in theta power, and increases in alpha power in the occipital lobe during multitasking. We also noted perturbations in theta band power during distraction tasks. Notably, physical movement induced more significant fluctuations in the frontal and motor cortex than distractions from social environment light signals. Particularly in scenarios involving walking and multitasking, there was a noticeable reduction in beta suppression. Our study underscores the importance of brain-body collaboration in multitasking scenarios, where the simultaneous engagement of the body and brain in complex tasks highlights the dynamic nature of cognitive processes within the framework of embodied cognition. Furthermore, integrating immersive augmented reality technology into embodied cognition research enhances our understanding of the interplay between the body, environment, and cognitive functions, with profound implications for advancing human-computer interaction and elucidating cognitive dynamics in multitasking.

Affordable Projector-Based AR Experimental Platform with Fitting Simulation Asset for Exploring Thermal Management

Augmented reality (AR) applied in education provides learners a possible way for better understanding and thorough learning. Although the traditional projector is used to integrate the augmented information with real objects without wearing AR glasses, the projector-based AR system is unlikely to be adopted widely in education due to the cost, heavy weight, and space issues. In this paper, an alternative projector-camera AR platform, utilizing a digital light processing (DLP) module matched with a Beaglebone Black (BB) controller, is proposed for AR physical experiments. After describing the DLP-based AR learning design method, the algorithm of pre-deforming projection content with simulation-based poly fitting is presented to keep the virtual asset consistent with the user action; and then a prototype with the content regarding the thermal management of power devices is illustrated to validate the performance of the AR experimental platform. The result shows that the DLP-based AR platform is an accurate and interactive AR system with a response time of 1 s, and a registration deviation of 3 mm. It is also an affordable AR learning design tool with a bill of materials of about $200, and thus casts light on creating AR-based physical experiments to explore more physical phenomena.

Augmented reality visualization of automated path planning for percutaneous interventions: a phantom study

PurposeInsertion point identification is a major challenge for percutaneous interventions. Planning in 2D slice image data is time-consuming and inefficient. Automated path planning can help to overcome these challenges. However, the setup of the intervention room is difficult to consider. In addition, transferring the insertion point to the skin is often prone to error. Therefore, a visualization for an automated path planning was implemented.MethodsA condition-based automated path planning was calculated with path length, distance to risk structures and insertion angle. The results were displayed on a phantom using projector-based augmented reality (AR) with an access point selection using the insertion needle. Two variants of the insertion visualization and three target displays were evaluated in a user study.ResultsA visualization of insertion points with a representation of the path quality resulted in a choice of safer paths, compared with no insertion point display or no coding of the path quality. A representation of the target was preferred in the final survey, but did not perform better. A target display separate from the insertion point visualization reduced interferences between visualizations.ConclusionA projector-based AR visualization of automated path planning results supports insertion point identification for percutaneous interventions. A display of the path quality enables the choice of safe access paths especially for unexperienced users. Further research is needed to identify clinical benefits and applicability.

AR visualization of automated access path planning for percutaneous interventions

Abstract Minimally invasive interventions, e.g., percutaneous needle interventions, have many advantages compared to traditional surgery. However, they may require complex and time-consuming planning with experience-dependent success. Automated access path planning is faster and more consistent but individual preferences and situational circumstances are not considered. To this end, displaying the path planning results directly on the patient’s skin, using projector-based augmented reality (AR), was investigated. A constraint-based path planning was implemented to evaluate the quality of every path, taking into account risk structures and path length. A visualization was developed to display the results on the skin and to allow for path selection. The choice of the path followed by a navigated insertion was evaluated in a pilot study (n=5), considering four levels of the visualization with different amounts of displayed insertion points. Participants stated that they preferred to have multiple potential puncture points displayed. However, the results for the considered variables show only small differences. Overall, it has been shown that projectorbased AR visualization of automated access path planning is possible and enables individual, situation-adapted insertion point selection. More research is required to further explore optimal display of paths.

A Projector-Based Augmented Reality Navigation System for Computer-Assisted Surgery.

In the medical field, guidance to follow the surgical plan is crucial. Image overlay projection is a solution to link the surgical plan with the patient. It realizes augmented reality (AR) by projecting computer-generated image on the surface of the target through a projector, which can visualize additional information to the scene. By overlaying anatomical information or surgical plans on the surgery area, projection helps to enhance the surgeon’s understanding of the anatomical structure, and intuitively visualizes the surgical target and key structures of the operation, and avoid the surgeon’s sight diversion between monitor and patient. However, it still remains a challenge to project the surgical navigation information on the target precisely and efficiently. In this study, we propose a projector-based surgical navigation system. Through the gray code-based calibration method, the projector can be calibrated with a camera and then be integrated with an optical spatial locator, so that the navigation information of the operation can be accurately projected onto the target area. We validated the projection accuracy of the system through back projection, with average projection error of 3.37 pixels in x direction and 1.51 pixels in y direction, and model projection with an average position error of 1.03 ± 0.43 mm, and carried out puncture experiments using the system with correct rate of 99%, and qualitatively analyzed the system’s performance through the questionnaire. The results demonstrate the efficacy of our proposed AR system.

Aligning Realities: Correlating Content between Projected and Head Worn Displays

Enabling the effective representation of an object’s position and depth in augmented reality (AR) is crucial not just for realism, but also to enable augmented reality’s wider utilization in real world applications. Domains such as architecture and building design cannot leverage AR’s advantages without the effective representation of position. Prior work has examined how the human visual system perceives and interprets such cues in AR. However, it has focused on application systems that only use a single AR modality, i.e., head-mounted display, tablet/handheld, or projection. However, given the respective limitations of each modality regarding shared experience, stereo display, field of view, etc., prior work has ignored the possible benefits of utilizing multiple AR modalities together. By using multiple AR systems together, we can attempt to address the deficiencies of one modality by leveraging the features of other modalities. This work examines methods for representing position in a multi-modal AR system consisting of a stereo head-mounted display and a ceiling mounted projection system. Given that the AR content is now rendered across two separate AR realities, how does the user know which projected object matches the object shown in their head-mounted display? We explore representations to correlate and fuse objects across modalities. In this paper, we review previous work on position and depth in AR, before then describing multiple representations for head-mounted and projector-based AR that can be paired together across modalities. To the authors’ knowledge, this work represents the first step towards utilizing multiple AR modalities in which the AR content is designed directly to compliment deficiencies in the other modality.

Estimating Visibility of Annotations for View Management in Spatial Augmented Reality Based on Machine-Learning Techniques.

Augmented Reality (AR) is a class of “mediated reality” that artificially modifies the human perception by superimposing virtual objects on the real world, which is expected to supplement reality. In visual-based augmentation, text and graphics, i.e., label, are often associated with a physical object or a place to describe it. View management in AR is to maintain the visibility of the associated information and plays an important role on communicating the information. Various view management techniques have been investigated so far; however, most of them have been designed for two dimensional see-through displays, and few have been investigated for projector-based AR called spatial AR. In this article, we propose a view management method for spatial AR, VisLP, that places labels and linkage lines based on the estimation of the visibility. Since the information is directly projected on objects, the nature of optics such as reflection and refraction constrains the visibility in addition to the spatial relationship between the information, the objects, and the user. VisLP employs machine-learning techniques to estimate the visibility that reflects human’s subjective mental workload in reading information and objective measures of reading correctness in various projection conditions. Four classes are defined for a label, while the visibility of a linkage line has three classes. After 88 and 28 classification features for label and linkage line visibility estimators are designed, respectively, subsets of features with 15 and 14 features are chosen to improve the processing speed of feature calculation up to 170%, with slight degradation of classification performance. An online experiment with new users and objects showed that 76.0% of the system’s judgments were matched with the users’ evaluations, while 73% of the linkage line visibility estimations were matched.

One-Fits-All vs. Tailor-Made: User-Centered Workstations for Field Assembly with an Application in Aircraft Parts Manufacturing

Field assembly of large components for aircraft, ships, trains or big machinery is often done in a rather unstructured environment, with little concern for the individual needs of the involved personnel. Confronted with a shortage of qualified personnel and a generally aging workforce, workstation designers need to put more emphasis on improving ergonomics and allowing for individual adjustments and support for a particular worker. Human factor engineering traditionally uses a “percentile-approach” to work-design, preventing an exact adjustment to the ergonomic needs of particular users. New technological advances provide opportunities for major improvements; novel assistance and information systems allow the accommodation of the needs of an increasingly diverse assembly workforce, while minimizing overall physical and mental stress.Based on a recently developed synopsis of options for individualization of assembly workstations, an example of an application in field assembly is being presented and illustrated by a full-scale demonstrator for the assembly of large scale composite parts for the aerospace industry. The pilot-installation combines a projector-based augmented reality environment for dynamic worker-information with automated operator position tracking and field of vision positioning. Finally, a comprehensive concept, that includes other dimensions of individualization, is presented.

Concepts for augmented reality visualisation to support needle guidance inside the MRI.

During MRI-guided interventions, navigation support is often separated from the operating field on displays, which impedes the interpretation of positions and orientations of instruments inside the patient's body as well as hand–eye coordination. To overcome these issues projector-based augmented reality can be used to support needle guidance inside the MRI bore directly in the operating field. The authors present two visualisation concepts for needle navigation aids which were compared in an accuracy and usability study with eight participants, four of whom were experienced radiologists. The results show that both concepts are equally accurate (n}{}2.0 pm 0.6 and n}{}1.7 pm 0.5, {rm mm}), useful and easy to use, with clear visual feedback about the state and success of the needle puncture. For easier clinical applicability, a dynamic projection on moving surfaces and organ movement tracking are needed. For now, tests with patients with respiratory arrest are feasible.

Projector-based augmented reality system for interventional visualization inside MRI scanners.

Navigation support in interventional magnetic resonance imaging (MRI) is separated from the operating field, which makes it difficult to interpret positions and orientations and to coordinate the necessary hand movements. We developed a projector-based augmented reality system to enable visual navigation of tracked instruments on pre-planned paths and visualization of risk structures directly on the patient inside the MRI bore. To assess the accuracy of the system, a user study was carried out with clinicians in a needle navigation test scenario. The targets were reached with an error of 1.7±0.5mm and the entry points with an error of 1.7±0.8mm. The accuracy results are similar to those reached by live image-guided interventions and related work and confirm that this projective augmented reality prototype for the interventional MRI can serve as a platform for current and future research in augmented reality visualization and dynamic registration.

Enhancing User Experiences of Mobile-Based Augmented Reality via Spatial Augmented Reality: Designs and Architectures of Projector-Camera Devices

As smartphones, tablet computers, and other mobile devices have continued to dominate our digital world ecosystem, there are many industries using mobile or wearable devices to perform Augmented Reality (AR) functions in their workplaces in order to increase productivity and decrease unnecessary workloads. Mobile-based AR can basically be divided into three main types: phone-based AR, wearable AR, and projector-based AR. Among these, projector-based AR or Spatial Augmented Reality (SAR) is the most immature and least recognized type of AR for end users. This is because there are a small number of commercial products providing projector-based AR functionalities in a mobile manner. Also, prices of mobile projectors are still relatively high. Moreover, there are still many technical problems regarding projector-based AR that have been left unsolved. Nevertheless, it is projector-based AR that has potential to solve a fundamental problem shared by most mobile-based AR systems. Also the always-visible nature of projector-based AR is one good answer for solving current user experience issues of phone-based AR and wearable AR systems. Hence, in this paper, we analyze what are the user experience issues and technical issues regarding common mobile-based AR systems, recently widespread phone-based AR systems, and rising wearable AR systems. Then for each issue, we propose and explain a new solution of how using projector-based AR can solve the problems and/or help enhance its user experiences. Our proposed framework includes hardware designs and architectures as well as a software computing paradigm towards mobile projector-based AR systems. The proposed design is evaluated by three experts using qualitative and semiquantitative research approaches.

Follow the light: projector-based augmented reality intracorporeal system for laparoscopic surgery

A projector-based augmented reality intracorporeal system (PARIS) is presented that includes a miniature tracked projector, tracked marker, and laparoscopic ultrasound (LUS) transducer. PARIS was developed to improve the efficacy and safety of laparoscopic partial nephrectomy (LPN). In particular, it has been demonstrated to effectively assist in the identification of tumor boundaries during surgery and to improve the surgeon's understanding of the underlying anatomy. PARIS achieves this by displaying the orthographic projection of the cancerous tumor on the kidney's surface. The performance of PARIS was evaluated in a user study with two surgeons who performed 32 simulated robot-assisted partial nephrectomies. They performed 16 simulated partial nephrectomies with PARIS for guidance and 16 simulated partial nephrectomies with only an LUS transducer for guidance. With PARIS, there was a significant reduction [30% ([Formula: see text])] in the amount of healthy tissue excised and a trend toward a more accurate dissection around the tumor and more negative margins. The combined point tracking and reprojection root-mean-square error of PARIS was 0.8mm. PARIS' proven ability to improve key metrics of LPN surgery and qualitative feedback from surgeons about PARIS supports the hypothesis that it is an effective surgical navigation tool.

Projector-based Augmented Reality for Telemaintenance Support

For efficient telemaintenance support, teleoperator and service technician need a common spatial awareness: both need to be able to observe the support environment and to point out to each other individual components of machinery where maintenance steps are to be performed. We propose the use of a projector-based Augmented Reality system that enables this kind of remote bi-directional communication with spatial reference. The system requires only a camera and a projector and allows an operator to precisely place virtual annotations in the remote support area using the projector, highlighting critical machine components and making service steps and instructions easily understandable to the technician. The performance of this Augmented Reality support system was evaluated in a large-scale user study with prospective service technicians.

2-2 Projector-based Augmented Reality for Medicine

2-2 Projector-based Augmented Reality for Medicine

Learning from the Trenches of Embodiment Design

Background The advent of ubiquitous computing requires us to reconsider all aspects of industrial design engineering – to invent, package and optimize such products, services and experiences to society. This project was devised to bridge these in a compelling and magical prototype, called the Kinetic Mirror, a mirror that can not only mimic color but also shape in front of it. It builds upon the efforts performed in the field of projector-based augmented reality and natural design interfaces, and it showcases our ideas of future prototyping of design concepts. Methods This article describes the complexity of engineering when embodying and producing such interactive systems to disseminate design knowledge. Specifically, we reflect on the conceptualization and development of the Kinetic Mirror: a three-dimensional display that mirrors depth and color in 400 “pixels”. Enabled by the introduction of low-cost structured light sensors, we envisioned an instantaneous physical manifestation of the captured scan. Results Challenges included: selecting electronic parts, software architecture, hardware and networking performance, outsourcing of production, power consumption, and overall assembly and construction The final system was put to show on five exhibits to test audience engagement and robustness of the result. This work has implications towards design curricula and provides new focal points of attention for design research and prototyping. Conclusions Demonstration and prototypes are an increasingly important medium to disseminate design knowledge, because experience can only partly conveyed in written text or even in video. However, if products become dynamic, articulated and with behavior, the technology requirements for prototypes become more complex, and as a result harder to maintain. In this paper we shared our lessons learned.

Spatial user interfaces for large-scale projector-based augmented reality.

Spatial augmented reality applies the concepts of spatial user interfaces to large-scale, projector-based augmented reality. Such virtual environments have interesting characteristics. They deal with large physical objects, the projection surfaces are nonplanar, the physical objects provide natural passive haptic feedback, and the systems naturally support collaboration between users. The article describes how these features affect the design of spatial user interfaces for these environments and explores promising research directions and application domains.

Hand gesture guided robot-assisted surgery based on a direct augmented reality interface

Radiofrequency (RF) ablation is a good alternative to hepatic resection for treatment of liver tumors. However, accurate needle insertion requires precise hand-eye coordination and is also affected by the difficulty of RF needle navigation. This paper proposes a cooperative surgical robot system, guided by hand gestures and supported by an augmented reality (AR)-based surgical field, for robot-assisted percutaneous treatment. It establishes a robot-assisted natural AR guidance mechanism that incorporates the advantages of the following three aspects: AR visual guidance information, surgeon's experiences and accuracy of robotic surgery. A projector-based AR environment is directly overlaid on a patient to display preoperative and intraoperative information, while a mobile surgical robot system implements specified RF needle insertion plans. Natural hand gestures are used as an intuitive and robust method to interact with both the AR system and surgical robot. The proposed system was evaluated on a mannequin model. Experimental results demonstrated that hand gesture guidance was able to effectively guide the surgical robot, and the robot-assisted implementation was found to improve the accuracy of needle insertion. This human–robot cooperative mechanism is a promising approach for precise transcutaneous ablation therapy.

Projector-based surgeon–computer interaction on deformable surfaces

Providing intuitive and easy to operate interaction for medical augmented reality is essential for use in the operating room. Commonly, intra-operative navigation information is displayed on an installed monitor, requiring the operating surgeon to change focus from the monitor to the surgical site and vice versa during navigation. Projector-based augmented reality has the potential to alleviate this problem. The aim of our work is to use a projector for visualization and to provide intuitive means for direct interaction with the projected information. A consumer-grade projector is used to visualize preoperatively defined surgical planning data. The projection of the virtual information is possible on any deformable surface, and the surgeon can interact with the presented virtual information. A Microsoft Kinect camera is used to capture both the surgeon interactions and the deformations of the surface over time. After calibration of projector and Kinect camera, the fingertips are localized automatically. A point cloud surface representation is used to determine the surgeon interaction with the projected virtual information. Interaction is detected by estimating the proximity of the surgeon's fingertips to the interaction zone and applying projector-Kinect calibration information. Interaction is performed using multi-touch gestures. In our experimental surgical scenario, the surgeon stands in front of the Microsoft Kinect camera, while relevant medical information is projected on the interaction zone. A hand wave gesture initiates the tracking of the hand. The user can then interact with the projected virtual information according to the defined multi-touch-based gestures. Thus, all information such as preoperative planning data is provided to the surgeon and his/her team intra-operatively in a familiar context. We enabled the projection of the virtual information on an arbitrarily shaped surface and used a Microsoft Kinect camera to capture the interaction zone and the surgeon's actions. The system eliminates the need for the surgeon to alternately view the surgical site and the monitor. The system eliminates unnecessary distractions and may enhance the surgeon's performance.

Projection-based visual guidance for robot-aided RF needle insertion

The use of projector-based augmented reality (AR) in surgery may enable surgeons to directly view anatomical models and surgical data from the patient's surface (skin). It has the advantages of a consistent viewing focus on the patient, an extended field of view and augmented interaction. This paper presents an AR guidance mechanism with a projector-camera system to provide the surgeon with direct visual feedback for supervision of robotic needle insertion in radiofrequency (RF) ablation treatment. The registration of target organ models to specific positions on the patient body is performed using a surface-matching algorithm and point-based registration. An algorithm based on the extended Kalman filter and spatial transformation is used to intraoperatively compute the virtual needle's depth in the patient's body for AR display. Experiments of this AR system on a mannequin were conducted to evaluate AR visualization and accuracy of virtual RF needle insertion. The average accuracy of 1.86mm for virtual needle insertion met the clinical requirement of 2mm or better. The feasibility of augmented interaction with a surgical robot using the proposed open AR interface with active visual feedback was demonstrated. The experimental results demonstrate that this guidance system is effective in assisting a surgeon to perform a robot-assisted radiofrequency ablation procedure. The novelty of the work lies in establishing a navigational procedure for percutaneous surgical augmented intervention integrating a projection-based AR guidance and robotic implementation for surgical needle insertion.

Pi-Tag: a fast image-space marker design based on projective invariants

Visual marker systems have become an ubiquitous tool to supply a reference frame onto otherwise uncontrolled scenes. Throughout the last decades, a wide range of different approaches have emerged, each with dif- ferent strengths and limitations. Some tags are optimized to reach a high accuracy in the recovered camera pose, others are based on designs that aim to maximizing the detection speed or minimizing the effect of occlusion on the detec- tion process. Most of them, however, employ a two-step pro- cedure where an initial homography estimation is used to translate the marker from the image plane to an orthonormal world, where it is validated and recognized. In this paper, we present a general purpose fiducial marker system that performs both steps directly in image-space. Specifically, by exploiting projective invariants such as collinearity and cross-ratios, we introduce a detection and recognition algo- rithm that is fast, accurate and moderately robust to occlu- sion. The overall performance of the system is evaluated in an extensive experimental section, where a comparison with a well-known baseline technique is presented. Additionally, several real-world applications are proposed, ranging from camera calibration to projector-based augmented reality.