Spatial Augmented Reality System Research Articles

Lessons Learned from Human Pose Interaction in an Industrial Spatial Augmented Reality Application

This paper is a technical description of a novel Augmented Reality application in the industrial domain of furniture production. In the presented case, workers suffered from high cognitive load in doing end-of-line quality inspection and individual handling of a high variety of products. The proposed solution consists of a Spatial Augmented Reality system, where a projector directly displays information on the product to assist the worker. At the same time results of an in-line quality inspections are shown which can be interactively modified through human gestures.The main contribution of this work is two-fold: (1) in contrast to other industrial augmented reality applications the described one technically builds upon a deep neural net based pose estimation. This allows a seamless interaction with the system and tracking of human actions rather than deriving them from the current state of the work piece, (2) the paper recapitulates experiences and results of the approach with a focus on lessons learned for using human pose estimation in smart production.

An Interdisciplinary Design of an Interactive Cultural Heritage Visit for In-Situ, Mixed Reality and Affective Experiences

Interactive technologies, such as mixed-reality and natural interactions with avatars, can enhance cultural heritage and the experience of visiting a museum. In this paper, we present the design rationale of an interactive experience for a cultural heritage place in the church of Roncesvalles at the beginning of Camino de Santiago. We followed a participatory design with a multidisciplinary team which resulted in the design of a spatial augmented reality system that employs 3D projection mapping and a conversational agent acting as the storyteller. Multiple features were identified as desirable for an interactive experience: interdisciplinary design team; in-situ; mixed reality; interactive digital storytelling; avatar; tangible objects; gestures; emotions and groups. The findings from a workshop are presented for guiding other interactive cultural heritage experiences.

Effects of spatial augmented reality assistance on the efficiency of prefabricating timber frame walls

ABSTRACT Assistance systems have the potential to support workers in various assembly tasks. Especially in mass customized prefabrication, the correct objects need to be selected and positioned accurately. Using conventional paper plans as assembly guidance, skilled workers need to spend precious time carefully reading plans for the individually designed elements. Augmented reality (AR) systems can help to integrate information for assembly tasks directly into the work environment, which should improve time efficiency and error rates. However, AR systems are often not ergonomic or accurate enough (e.g. smartglasses) or expensive (e.g. laser projectors). In the present study, the effect of a simple projection based spatial augmented reality (SAR) system was studied for building prefabricated wall elements under laboratory conditions, using printed CAD drawings as reference. Each subject built a timber frame wall element with dimensions close to reality. Using the SAR system, the completion times were reduced by nearly 50%, the error rate of subjects with a low experience level decreased and the reported task load measured according to the NASA raw task load index using the SAR system was significantly lower.

Advanced Adaptive Spatial Augmented Reality utilizing Dynamic in-situ Projection in Industrial Site Assembly

An Adaptive Spatial Augmented Reality system with dynamic in-situ projection of work instructions and information guidance already shows benefits in the field of industrial site assembly of aircraft components. Furthermore, the user-adaptive gesture control offers advantages for a human-centered and natural interaction directly in the assembly process. This paper presents results of enhancing dynamic in-situ projection, information guidance and user-adaptive gesture interaction in an advanced assembly process of large-sized toy building bricks. For this purpose, the dynamic projection system displays the location of the bricks in the material provision area as well as the accurate mounting position and dynamically guides the operator through the assembly process. Furthermore, the optimized gesture interaction with YOLO and an optimized synthetic dataset creation process for site assembly, indicate strong intentions for user-adaptive interaction with the projection system. An evaluation in the TU Wien pilot factory demonstrates benefits in terms of time savings in locating parts, usability and cognitive workload.

Exploring Tablet Interfaces for Product Appearance Authoring in Spatial Augmented Reality

Users acceptance of innovative product appearance authoring tools based on Spatial Augmented Reality (SAR) is still limited due to their perception of a high technology complexity and a low performance/functionality of the current interaction systems. The integration of SAR technologies in professional design activities is still marginal, though many studies in this field have already proved their potential as supporting tools. To overcome this barrier, efficient means for interacting with the digital images projected onto the surfaces of real objects are essential. The aim of the current study is to respond to this demand by proposing and validating three UI configurations displayed by an unique and portable device embedded with a touch screen. These interface layouts, designed to cooperate with the output of the SAR system and to not affect the well-known benefits of its augmented environment, provide different types of visual feedback to the user by duplicating, extending or hiding the information already displayed by the projected mock-up. The experimental study reported here, performed with a panel of 41 subjects, revealed that accuracy, efficiency and perceived usability of the proposed solutions are comparable with each other and in comparison to standard desktop setups commonly used for design activities. According to these findings, the research simultaneously demonstrates (i) the high performances achieved by the touch device when coupled with a SAR system during the execution of authoring tasks, (ii) the capability of the projected mock-up to behave as an actual three-dimensional display for the real time rendering of the product appearance and (iii) the possibility to freely select - according to the users preference, the design task or the type of product - one of the three UI configurations without affecting the quality of the result.

Digital Twin Registration Technique of Spatial Augmented Reality for Tangible Interaction

<p indent="0mm">In spatial augmented reality (SAR) systems supporting tangible interaction, user interactions can cause rapid changes in orientation and position of interaction objects. To ensure the efficiency and accuracy of registration during objects movements, a digital twin registration technique for SAR is proposed. Models are made in the digital space and their geometric parameters are entirely consistent with the objects in the physical space. The orientation and position parameters of the physical objects are tracked in real-time during user interaction. Then the digital objects are adjusted according the parameters. Moreover, intrinsic and extrinsic parameters of the projector, which are calibrated in advance, are used to set the virtual camera in the digital space. The projection patterns are rendered in that virtual camera and then projected onto the interaction objects. The efficiency and accuracy are evaluated in the middle school physical experiment learning magnetic induction line based on the projection interaction tabletop, which meet the requirements of tangible interaction in SAR systems.

A Survey on Human-Computer Interaction in Spatial Augmented Reality

<p indent=0mm>Spatial augmented reality (SAR), which typically uses projectors to augment physical spaces with virtual contents, is one of the three main forms of augmented reality. In SAR systems, users can interact with virtual contents in physical spaces, therefore they have good environmental, spatial and collaborative perceptions. SAR systems can also reduce the cognitive load of interaction tasks. SAR display forms are firstly demonstrated, and then SAR architectures are illustrated as well as its technical principles. SAR interaction technologies are classified into 6 categories, which are touch based interaction, gesture or posture based interaction, 3D controller based interaction, handholding projector interaction, tangible interaction, and multimodal interaction. Challenges from 3D registration, virtual-real fusion are analyzed. The application status in the field of sports, medical treatment, education, entertainment and art are discussed. The trends in SAR interaction are finally predicted. Specifically, SAR interaction will become more natural, more intelligent in occlusion processing, more widely applicable, more closely related with the social, cultural and psychological researches, and novel display approaches will also be used by SAR systems.

A socio-cognitive analysis of evaluation and idea generation activities during co-creative design sessions supported by spatial augmented reality

ABSTRACT Co-creation with end-users is gaining an increasing interest in the design industry today. This study investigates the mechanisms by which spatial augmented reality (SAR) technology can affect socio-cognitive processes in groups involved in co-creative design sessions. More precisely, when used in real co-creative design sessions, does a SAR system facilitate the collective creative process, compared to sessions occurring in standard settings? A protocol analysis has been conducted to investigate three different design sessions involving experienced designers and end-users on a product design task: a design session supported by conventional design representations (usual design practices), a design session supported by non-spatial augmented reality (AR), and a session supported by SAR technology. While results do not clearly show that SAR or AR technologies increase end-user’s commitmen t, they illustrate the ability for these technologies to allow browsing through more ideas during a co-creative design session. Furthermore, it tends to reduce time spent on ideas, compared to a traditional session. We also noted that the introduction of this technology does not modify the profiles of the sessions in terms of cognitive activities. This tends to demonstrate that the technology itself does not impair the design activity.

Conveying spatial awareness cues in xR collaborations.

Spatial Augmented Reality (SAR) systems can be suitably combined with other existing Extended Reality (xR) technologies to support collaboration. In existing strategies, users unencumbered by a viewing technology, such as a tablet interface or a head-mounted display, must rely on the transmission of their collaborators' positioning through interpreting a first-person camera view. This design creates a seam between a user's experience of the augmented physical environment in SAR, and their collaborators' experience inside the virtual environment. To assist in development and evaluation of spatial cues to support spatial awareness in SAR environments, an egocentric spatial-communication taxonomy is presented given two determining dimensions, a cue's attachment (physical/virtual) and animation (local/world). We developed four egocentric cues which characterize the four independent dimensions of the matrix: arrow, path, glow, and radial, and a single exocentric world in miniature visualization. Our study shows that virtual attachment cues are preferred, providing the highest accuracy, highest performance when collaborators are occluded, and produce the least mental effort when used with a single virtual collaborator. For multiple collaborators however, the virtual attached, world animated radial cue produces significant increases in mental load and reductions in preference, demonstrating the impact of visual augmentation clutter. The single exocentric visualization produced higher levels of head movement, and poorer accuracy, however the novelty of the visualization produced positive qualitative results.

Collaborative Welding System using BIM for Robotic Reprogramming and Spatial Augmented Reality

Abstract The optimization of the information flow from the initial design and through the several production stages plays a critical role in ensuring product quality while also reducing the manufacturing costs. As such, in this article we present a cooperative welding cell for structural steel fabrication that is capable of leveraging the Building Information Modeling (BIM) standards to automatically orchestrate the necessary tasks to be allocated to a human operator and a welding robot moving on a linear track. We propose a spatial augmented reality system that projects alignment information into the environment for helping the operator tack weld the beam attachments that will be later on seam welded by the industrial robot. This way we ensure maximum flexibility during the beam assembly stage while also improving the overall productivity and product quality since the operator no longer needs to rely on error prone measurement procedures and he receives his tasks through an immersive interface, relieving him from the burden of analyzing complex manufacturing design specifications. Moreover, no expert robotics knowledge is required to operate our welding cell because all the necessary information is extracted from the Industry Foundation Classes (IFC), namely the CAD models and welding sections, allowing our 3D beam perception systems to correct placement errors or beam bending, which coupled with our motion planning and welding pose optimization system ensures that the robot performs its tasks without collisions and as efficiently as possible while maximizing the welding quality.

Design of an Interactive Spatial Augmented Reality System for Stage Performance Based on UWB Positioning and Wireless Triggering Technology

In this research, the authors designed an interactive spatial augmented reality system for stage performance based on the technologies of UWB positioning and Bluetooth® triggering. The position of the actor is obtained through the antenna tag carried by the actor and the signal base station placed on the stage. Special effects can be triggered through the Bluetooth® module according to the actor, and rendered at the relevant location on the screen, which has higher concealment. The system has a higher degree of freedom in practical applications, which can present an interactive spatial augmented reality effect, and therefore provide new possibilities for the application of spatial augmented reality in the stage performance.

Defining requirements for an Augmented Reality system to overcome the challenges of creating and using design representations in co-design sessions

ABSTRACTWithin co-design sessions involving designers and non-designers, the type and characteristics of the design representations employed is known to impact the performance of such sessions in terms of idea generation, idea evaluation and communication. This study captures the challenges practitioners face in creating and using design representations for co-design sessions and goes on to investigate the potential of Spatial Augmented Reality (SAR) to overcome those challenges. The advantages of SAR in this application are that, multiple concepts can be represented using one physical model, concepts can be modified live during the session, and additional equipment (such as head mounted displays or handheld devices) is not required, thus eliminating any possible interference with the natural interactions between participants. Interviews with design practitioners and trials with a prototype SAR system are used to identify the key challenges faced by practitioners in their current use of design representations, and to capture the technology requirements for a SAR system for use in co-design sessions. These findings can inform the work of technology developers and researchers working on systems to support co-design sessions.

A User Study of a Prototype of a Spatial Augmented Reality System for Education and Interaction with Geographic Data

Recent technological advancements in many areas have changed the way that individuals interact with the world. Some daily tasks require visualization skills, especially when in a map-reading context. Augmented Reality systems could provide substantial improvement to geovisualization once it enhances a real scene with virtual information. However, relatively little research has worked on assessing the effective contribution of such systems during map reading. So, this research aims to provide a first look into the usability of an Augmented Reality system prototype for interaction with geoinformation. For this purpose, we have designed an activity with volunteers in order to assess the system prototype usability. We have interviewed 14 users (three experts and 11 non-experts), where experts were subjects with the following characteristics: a professor; with a PhD degree in Cartography, GIS, Geography, or Environmental Sciences/Water Resources; and with experience treating spatial information related to water resources. The activity aimed to detect where the system really helps the user to interpret a hydrographic map and how the users were helped by the Augmented Reality system prototype. We may conclude that the Augmented Reality system was helpful to the users during the map reading, as well as allowing the construction of spatial knowledge within the proposed scenario.

HySAR: Hybrid Material Rendering by an Optical See-Through Head-Mounted Display with Spatial Augmented Reality Projection.

Spatial augmented reality (SAR) pursues realism in rendering materials and objects. To advance this goal, we propose a hybrid SAR (HySAR) that combines a projector with optical see-through head-mounted displays (OST-HMD). In an ordinary SAR scenario with co-located viewers, the viewers perceive the same virtual material on physical surfaces. In general, the material consists of two components: a view-independent (VI) component such as diffuse reflection, and a view-dependent (VD) component such as specular reflection. The VI component is static over viewpoints, whereas the VD should change for each viewpoint even if a projector can simulate only one viewpoint at one time. In HySAR, a projector only renders the static VI components. In addition, the OST-HMD renders the dynamic VD components according to the viewer's current viewpoint. Unlike conventional SAR, the HySAR concept theoretically allows an unlimited number of co-located viewers to see the correct material over different viewpoints. Furthermore, the combination enhances the total dynamic range, the maximum intensity, and the resolution of perceived materials. With proof-of-concept systems, we demonstrate HySAR both qualitatively and quantitatively with real objects. First, we demonstrate HySAR by rendering synthetic material properties on a real object from different viewpoints. Our quantitative evaluation shows that our system increases the dynamic range by 2.24 times and the maximum intensity by 2.12 times compared to an ordinary SAR system. Second, we replicate the material properties of a real object by SAR and HySAR, and show that HySAR outperforms SAR in rendering VD specular components.

Non-Contact Thermo-Visual Augmentation by IR-RGB Projection.

This paper presents an approach for non-contact haptic augmentation with spatial augmented reality (SAR). We construct a thermo-visual projection system by combining a standard RGB projector and a fabricated infrared (IR) projector. The primary contribution of this paper is that we conduct thorough psychophysical experiments to investigate a design guideline for spatiotemporal projection patterns for both RGB and IR projectors to render a warm object with high presence. We develop application systems to evaluate the validity of the proposed system and design guideline. The evaluation results demonstrate that the proposed system can render warm objects with significantly higher presence than a standard SAR system.

Robust Multithreaded Object Tracker through Occlusions for Spatial Augmented Reality

A spatial augmented reality (SAR) system enables a virtual image to be projected onto the surface of a real-world object and the user to intuitively control the image using a tangible interface. However, occlusions frequently occur, such as a sudden change in the lighting environment or the generation of obstacles. We propose a robust object tracker based on a multithreaded system, which can track an object robustly through occlusions. Our multithreaded tracker is divided into two threads: the detection thread detects distinctive features in a frame-to-frame manner, and the tracking thread tracks features periodically using an optical-flow-based tracking method. Consequently, although the speed of the detection thread is considerably slow, we achieve real-time performance owing to the multithreaded configuration. Moreover, the proposed outlier filtering automatically updates a random sample consensus distance threshold for eliminating outliers according to environmental changes. Experimental results show that our approach tracks an object robustly in real-time in an SAR environment where there are frequent occlusions occurring from augmented projection images.

Use of projector based augmented reality to improve manual spot-welding precision and accuracy for automotive manufacturing

This paper presents the use of a projector-based spatial augmented reality system in an industrial quality assurance setting to highlight spot-weld locations on vehicle panels for manual welding operators. The aim of this work is to improve the precision and accuracy of manual spot-weld placements with the aid of visual cues as a proactive step by the automotive manufacturer to enhance product quality. The prototype system was deployed at General Motors (GM) Holden plant in Elizabeth, Australia on the production line building Holden Cruze vehicles. Production trials were conducted and techniques developed to analyse and validate the precision and accuracy of spot-welds both with and without the visual cues. A reduction of 52 % of the standard deviation of manual spot-weld placement was observed when using augmented reality visual cues. The average standard deviation with-AR assistance (19 panels and 114 spot-welds) was calculated at 1.94 mm compared to without-AR (45 panels and 270 spot-welds) at 4.08 mm. All welds were within the required specification and panels evaluated in this study were used as the final product made available to consumers. The visual cues enabled operators to spot-weld at a higher degree of precision and accuracy.

Data-Driven Kinematic Control for Robotic Spatial Augmented Reality System with Loose Kinematic Specifications

We propose a data-driven kinematic control method for a robotic spatial augmented reality (RSAR) system. We assume a scenario where a robotic device and a projector-camera unit (PCU) are assembled in an ad hoc manner with loose kinematic specifications, which hinders the application of a conventional kinematic control method based on the exact link and joint specifications. In the proposed method, the kinematic relation between a PCU and joints is represented as a set of B-spline surfaces based on sample data rather than analytic or differential equations. The sampling process, which automatically records the values of joint angles and the corresponding external parameters of a PCU, is performed as an off-line process when an RSAR system is installed. In an on-line process, an external parameter of a PCU at a certain joint configuration, which is directly readable from motors, can be computed by evaluating the pre-built B-spline surfaces. We provide details of the proposed method and validate the model through a comparison with an analytic RSAR model with synthetic noises to simulate assembly errors.