Projection-based Augmented Reality Research Articles

Illuminating precise stencils on surgical sites using projection-based augmented reality

In this paper we propose a system that connects surgeons to remote or local experts who provide real-time surgical guidance by illuminating salient markings or stencils (e.g. points, lines and curves) on the physical surgical site using a projector. The projection can be modified in real time by the expert using a GUI and can be seen by all in the operating room (OR) without the use of any wearables. This system overcomes the limitations of AR/VR headsets which can overlay information through a headset, but are obtrusive, not very accurate with movements, and visible only to the surgeon excluding others in the room. Overlaying information, at high precision, directly on the physical surgical site that can be seen by everyone in the OR can become an useful tool for skill transfer, expert consultation and training, especially in telemedicine.In addition to the projector, the system comprises of a RGB-D camera (e.g. Kinect) for feedback, together designated as the PDC (Projector Depth Camera) unit. The PDC is driven by a PC. The RGB-D camera provides depth information in addition to an image at video frame rates. A high resolution mesh of the surgical site is captured using the PDC unit initially. During the surgical planning, training or execution session, this digital model can be marked by appropriate incision markings on a tablet or monitor using touch based or mouse based interface, on the same local machine or after being transmitted to a remote machine. These markings are then communicated back to the PDC unit and illuminated at high precision via the projector on the surgical site in real time. If the surgical site moves during the process, the movement is tracked and updated quickly on the surgical site. Our method specifically overcomes the obtrusive, exclusive, and indirect attributes of headsets and displays while maintaining high accuracy of registration with movements.

Invited Session IV: Extended reality--applications in vision science and beyond: Augmented reality systems for people with low vision.

Low vision is a visual impairment that falls short of blindness but cannot be corrected by eyeglasses or contact lenses. While current low vision aids (e.g., magnifier, CCTV) support basic vision enhancements, such as magnification and contrast enhancement, these enhancements often arbitrarily alter a user's full field of view without considering the user's context, such as their visual abilities, tasks, and environmental factors. As a result, these low vision aids are not sufficient or preferred by low vision users in many important tasks. Augmented reality (AR) technology presents a unique opportunity to enhance low vision people's visual experience by automatically recognizing the surrounding environment and presenting tailored visual augmentations. In this talk, I will talk about how we design and build intelligent AR systems to support low vision people in visual tasks, such as a head-mounted AR system that presents visual cues to orient users' attention in a visual search task, as well as a projection-based AR system that projects visual highlights on the stair edges to support safe stair navigation. I will conclude my talk by discussing our future research direction on AR for low vision accessibility.

Improving the efficacy of exposure therapy using projection-based augmented reality for the treatment of cockroach phobia: a randomised clinical trial protocol

IntroductionIn vivo exposure is the treatment of choice for specific phobia (SP), but this technique presents limitations related to access and acceptability. Augmented reality (AR) offers advantages like maximising strategies...

Effectiveness of a projection-based augmented reality exposure system in treating cockroach phobia: study protocol of a randomised controlled trial

BackgroundDespite being the treatment of choice for phobic disorders, in vivo exposure treatment (IVET) presents some important limitations related mainly to low acceptance and high drop-out rates. Augmented reality (AR)...

A Projection-Based Augmented Reality System for Medical Applications

The aim of this paper was to present the development of an Augmented Reality (AR) system which uses a 2D video projector to project a 3D model of blood vessels, built by combining Computed Tomography (CT) slices of a human brain, onto a model of a human head. The difficulty in building this system is that the human head contains, not flat surfaces, but non-regular curved surfaces. Using a 2D projector to project a 3D model onto non-regular curved 3D surfaces would result in serious distortions of the projection if the image was not uncorrected first. This paper proposed a method of correcting the projection, not only based on the curvatures of the surfaces, but also on the viewing position of the observer. Experimental results of this system showed that an average positional deviation error of 2.065 mm could be achieved under various test conditions.

A UWB-Driven Self-Actuated Projector Platform for Interactive Augmented Reality Applications

With the rapid development of interactive technology, creating systems that allow users to define their interactive envelope freely and provide multi-interactive modalities is important to build up an intuitive interactive space. We present an indoor interactive system where a human can customize and interact through a projected screen utilizing the surrounding surfaces. An ultra-wideband (UWB) wireless sensor network was used to assist human-centered interaction design and navigate the self-actuated projector platform. We developed a UWB-based calibration algorithm to facilitate the interaction with the customized projected screens, where a hand-held input device was designed to perform mid-air interactive functions. Sixteen participants were recruited to evaluate the system performance. A prototype level implementation was tested inside a simulated museum environment, where a self-actuated projector provides interactive explanatory content for the on-display artifacts under the user’s command. Our results depict the applicability to designate the interactive screen efficiently indoors and interact with the augmented content with reasonable accuracy and relatively low workload. Our findings also provide valuable user experience information regarding the design of mobile and projection-based augmented reality systems, with the ability to overcome the limitations of other conventional techniques.

Projection-based augmented reality system for assembly guidance and monitoring

PurposeThis study aims to monitor and guide the assembly process. The operators need to change the assembly process according to the products’ specifications during manual assembly of mass customized production. Traditional information inquiry and display methods, such as manual lookup of assembly drawings or electronic manuals, are inefficient and error-prone.Design/methodology/approachThis paper proposes a projection-based augmented reality system (PBARS) for assembly guidance and monitoring. The system includes a projection method based on viewpoint tracking, in which the position of the operator’s head is tracked and the projection images are changed correspondingly. The assembly monitoring phase applies a method for parts recognition. First, the pixel local binary pattern (PX-LBP) operator is achieved by merging the classical LBP operator with the pixel classification process. Afterward, the PX-LBP features of the depth images are extracted and the randomized decision forests classifier is used to get the pixel classification prediction image (PCPI). Parts recognition and assembly monitoring is performed by PCPI analysis.FindingsThe projection image changes with the viewpoint of the human body, hence the operators always perceive the three-dimensional guiding scene from different viewpoints, improving the human-computer interaction. Part recognition and assembly monitoring were achieved by comparing the PCPIs, in which missing and erroneous assembly can be detected online.Originality/valueThis paper designed the PBARS to monitor and guide the assembly process simultaneously, with potential applications in mass customized production. The parts recognition and assembly monitoring based on pixels classification provides a novel method for assembly monitoring.

El aula como laboratorio de experimentación visual

El presente ensayo visual trata de mostrar algunas soluciones a dos problemáticas relacionadas con la tríada ‘Arte - Educación - Tecnología’. La primera es la desconexión existente entre los contenidos que se imparten en las asignaturas de Educación Artística en relación a las nuevas visualidades o formas de creación artística vinculadas a las nuevas tecnologías. La segunda es la falta de aprovechamiento de los recursos tecnológicos disponibles en las aulas en relación a la enseñanza de las artes visuales.
 El punto de partida para resolver estas carencias se sitúa en los proyectores como recursos tecnológicos eminentemente visuales que se pueden en contrar en la mayoría de los centros educativos y que a su vez son la base de las creaciones contemporáneas que utilizan la Projection-Based Augmented Reality (PBAR) [Realidad Aumentada Basada en la Proyección].
 Tomando como referencia algunas ideas didácticas del cineasta granadino José Val del Omar (1904-1982), se detallan seis ejemplos que proponen la adopción de estrategias educativas basadas en la experimentación con estas tecnologías en la formación inicial del profesorado, transformando el aula en un laboratorio en el que el aprendizaje se acerca a los procesos creativos de las y los artistas profesionales.
 Por último, es importante destacar que las imágenes que poseen un marco con línea discontinua gris contienen un hipervínculo que permiten acceder a once videoensayos que amplían la información visual de cada apartado.

Exploring the use of AR technology for co-creative product and packaging design

Extended Reality technologies, including Virtual Reality (VR) and Augmented Reality (AR), are being applied in a wide variety of industrial applications, but their use within design practice remains very limited, despite some promising research activities in this area over the last 20 years. At the same time, design practice has been evolving to place greater emphasis on the role of the client or end-user in the design process through ‘co-creative design’ activities. Whilst offering many benefits, co-creative design activities also present challenges, notably in the communication between designers and non-designers, which can hinder innovation.In this paper, we investigate the potential of a novel, projection-based AR system for the creation of design representations to support co-creative design sessions. The technology is tested through benchmarking experiments and in-situ trials conducted with two industrial partners. Performance metrics and qualitative feedback are used to evaluate the effectiveness of the new technology in supporting co-creative design sessions. Overall, AR technology allows quick, real-time modifications to the surfaces of a physical prototype to try out new ideas. Consequently, designers perceive the possibility to enhance the collaboration with the end-users participating in the session. Moreover, the quality and novelty of ideas generated whilst using projection-based AR outperform conventional sessions or handheld display AR sessions. Whilst the results of these early trials are not conclusive, the results suggest that projection-based AR design representations provide a promising approach to supporting co-creative design sessions.

Projection-Based Augmented Reality Assistance for Manual Electronic Component Assembly Processes

Personalized production is moving the progress of industrial automation forward, and demanding new tools for improving the decision-making of the operators. This paper presents a new, projection-based augmented reality system for assisting operators during electronic component assembly processes. The paper describes both the hardware and software solutions, and depicts the results obtained during a usability test with the new system.

Evaluation of projection-based augmented reality technique in cerebral catheter procedures

Evaluation of projection-based augmented reality technique in cerebral catheter procedures

Deep-cARe: Projection-Based Home Care Augmented Reality System with Deep Learning for Elderly

Developing innovative and pervasive smart technologies that provide medical support and improve the welfare of the elderly has become increasingly important as populations age. Elderly people frequently experience incidents of discomfort in their daily lives, including the deterioration of cognitive and memory abilities. To provide auxiliary functions and ensure the safety of the elderly in daily living situations, we propose a projection-based augmented reality (PAR) system equipped with a deep-learning module. In this study, we propose three-dimensional space reconstruction of a pervasive PAR space for the elderly. In addition, we propose the application of a deep-learning module to lay the foundation for contextual awareness. Performance experiments were conducted for grafting the deep-learning framework (pose estimation, face recognition, and object detection) onto the PAR technology through the proposed hardware for verification of execution possibility, real-time execution, and applicability. The precision of the face pose is particularly high by pose estimation; it is used to determine an abnormal user state. For face recognition results of whole class, the average detection rate (DR) was 74.84% and the precision was 78.72%. However, for face occlusions, the average DR was 46.83%. It was confirmed that the face recognition can be performed properly if the face occlusion situation is not frequent. By object detection experiment results, the DR increased as the distance from the system decreased for a small object. For a large object, the miss rate increased when the distance between the object and the system decreased. Scenarios for supporting the elderly, who experience degradation in movement and cognitive functions, were designed and realized, constructed using the proposed platform. In addition, several user interfaces (UI) were implemented according to the scenarios regardless of distance between users and the proposed system. In this study, we developed a bidirectional PAR system that provides the relevant information by understanding the user environment and action intentions instead of a unidirectional PAR system for simple information provision. We present a discussion of the possibility of care systems for the elderly through the fusion of PAR and deep-learning frameworks.

Designing and evaluating an intelligent augmented reality system for assisting older adults’ medication management

Intelligent Assistive Systems (IASs) provide personalised support to individuals to increase their autonomy and competence in performing activities. The personalisation of such technology represents a particular challenge, for the individuals abilities typically change over time. The aim of this research was to explore a theory-based, activity-centred framework for the development of IASs that both facilitate the active involvement of caregivers and target users and allow for adaptation to the individual over time.Medication management was focused on as a case study. Three older adults and two caregivers were involved in a co-design process of an IAS prototype system, the MED-AR, which uses projection-based augmented reality as the user interface technology. The theoretical framework for the design and evaluation of the MED-AR was activity theory. For the decision-making process of the MED-AR, formal argumentation theory was used. The major contributions of the study are (i) a formal framework for understanding the level of independence in activities and the way an IAS can tailor support to an individual and a situation, including caregivers; and (ii) a model for involving older adults and caregivers in a co-design process in the first phases of developing IASs.

2D Drawing Visualization Framework for Applying Projection-Based Augmented Reality in a Panelized Construction Manufacturing Facility: Proof of Concept

AbstractProduct quality is recognized as a major benefit in industrialized construction because of the utilization of machines in manufacturing facilities. However, manual work continues to exist i...

Comparing Human Factors for Augmented Reality Supported Single-User and Collaborative Repair Operations of Industrial Robots.

In order to support the decision-making process of industry on how to implement Augmented Reality (AR) in production, this article wants to provide guidance through a set of comparative user studies. The results are obtained from the feedback of 160 participants who performed the same repair task on a switch cabinet of an industrial robot. The studies compare several AR instruction applications on different display devices (head-mounted display, handheld tablet PC and projection-based spatial AR) with baseline conditions (paper instructions and phone support), both in a single-user and a collaborative setting. Next to insights on the performance of the individual device types for the single mode operation, the study is able to show significant indications on AR techniques are being especially helpful in a collaborative setting.

Projection-Based AR: Effective Visual Feedback in Gait Rehabilitation

Effective real-time feedback is crucial to enhance motor learning in physical and cognitive rehabilitation. Although monitor-based (MB) platforms are widely used to provide visual feedback for users, the discrepancy between where the feedback is presented and where the actual movement occurs can negatively affect the use of feedback. In this paper, we propose a novel visual feedback system using projection-based augmented reality (AR) to provide a better understanding of the relationship between body perception and movement kinematics. A set of experiments were performed to assess and compare the ability of unimpaired and impaired participants to detect real-time feedback and make modifications to gait using this feedback in MB and AR platforms. The results showed a significant improvement in participants' ability to better adapt the changes in projection-based AR than MB. This improvement represents the efficacy of this system in enabling users to better synchronize their foot-eye coordination.

Hybrid Augmented Reality for Participatory Learning: The Hidden Efficacy of Multi-User Game-Based Simulation

The goal for this research is to articulate and test a new hybrid Augmented Reality (AR) environment for conceptual understanding. From the theoretical lens of embodied interaction, we have designed a multi-user participatory simulation called ARfrac t where visitors in a science museum can learn about complex scientific concepts on the refraction of light through full-body immersion using optical see-through AR glasses, projection-based AR, and gesture technology. In particular, we developed two different types of simulations for ARfract , namely a game-based simulation and a non-game simulation to explore how the order of different AR simulations influences the perceived usability, user behaviors, learning experiences, and learning outcomes. For the experiment, 10 dyads were randomly assigned to one of the two experimental conditions: 1) the game-to-non-game condition and 2) the non-game-to-game condition. The results indicate that the learners who experienced the game-based simulation before the non-game simulation performed better than did the other group with the reversed experience order. This paper also reports the usability, user behaviors, and learning experience issues regarding the affordances of hybrid AR technologies. The major contribution of this proof-of concept research is that it articulates our understanding of how particular configurations (i.e., order) of the emerging technologies (i.e., hybrid Augmented Reality systems) and its use can lead to different learning outcomes.

2-4 動的対象への投影型拡張現実感(第2部門VR関連技術とその応用)

2-4 動的対象への投影型拡張現実感(第2部門VR関連技術とその応用)

Decoupling of Real and Digital Content in Projection-based Augmented Reality Systems Using Time Multiplexed Image Capture

Decoupling of Real and Digital Content in Projection-based Augmented Reality Systems Using Time Multiplexed Image Capture

Lumipen 2: Dynamic Projection Mapping with Mirror-Based Robust High-Speed Tracking against Illumination Changes

Dynamic projection mapping (DPM) is a type of projection-based augmented reality that aligns projected content with a moving physical object. In order to be able to adjust the projection to fast motions of moving objects, DPM requires high-speed visual feedback. An option to reduce the temporal delay of adjusting the projection to imperceptible levels is to use mirror-based high-speed optical axis controllers. However, using such controllers for capturing visual feedback requires a sufficient amount of illumination of the moving object. This leads to a trade-off between tracking stability and quality of projection content. In this article, we propose a system that combines mirror-based high-speed tracking with using a retroreflective background. The proposed tracking technique observes the silhouette of the target object by episcopic illumination and is robust against illumination changes. It also maintains high-speed, accident-avoidant tracking by performing background subtraction in an active vision system and employing an adaptive windows technique. This allows us to create a DPM with an imperceptible temporal delay, high tracking stability and high visual quality. We analyze the proposed system regarding the visual quality of the retroreflective background, the tracking stability under illumination and disturbance conditions, and the visual consistency relative to delay in the presence of pose estimation. In addition, we demonstrate application scenarios for the proposed DPM system.