Augmented Reality Scene Research Articles

A dataset of eye gaze images for calibration-free eye tracking augmented reality headset

Eye tracking is a widely used technique. To enhance eye gaze estimation in different contexts, many eye tracking datasets have been proposed. However, these datasets depend on calibrations in data capture and its applications. We seek to construct a dataset that enables the design of a calibration-free eye tracking device irrespective of users and scenes. To reach this goal, we present ARGaze, a dataset with 1,321,968 pairs of eye gaze images at 32 × 32 pixel resolution and 50 corresponding videos of world views based on a replicable augmented reality headset. The dataset was captured from 25 participants who completed eye gaze tasks for 30 min in both real-world and augmented reality scenes. To validate the dataset, we compared it against state-of-the-art eye gaze datasets in terms of effectiveness and accuracy and report that the ARGaze dataset achieved record low gaze estimation error by 3.70 degrees on average and 1.56 degrees on specific participants without calibrations to the two scenes. Implications for generalising the use of the dataset are discussed.

Augmented reality navigation with real-time tracking for facial repair surgery.

Facial repair surgeries (FRS) require accuracy for navigating the critical anatomy safely and quickly. The purpose of this paper is to develop a method to directly track the position of the patient using video data acquired from the single camera, which can achieve noninvasive, real time, and high positioning accuracy in FRS. Our method first performs camera calibration and registers the surface segmented from computed tomography to the patient. Then, a two-step constraint algorithm, which includes the feature local constraint and the distance standard deviation constraint, is used to find the optimal feature matching pair quickly. Finally, the movements of the camera and the patient decomposed from the image motion matrix are used to track the camera and the patient, respectively. The proposed method achieved fusion error RMS of 1.44 ± 0.35, 1.50 ± 0.15, 1.63 ± 0.03mm in skull phantom, cadaver mandible, and human experiments, respectively. The above errors of the proposed method were lower than those of the optical tracking system-based method. Additionally, the proposed method could process video streams up to 24 frames per second, which can meet the real-time requirements of FRS. The proposed method does not rely on tracking markers attached to the patient; it could be executed automatically to maintain the correct augmented reality scene and overcome the decrease in positioning accuracy caused by patient movement during surgery.

Human-machine interface for a master-slave hydraulic manipulator with vision enhancement and auditory feedback

This paper proposes a novel human-machine interface (HMI) for addressing the operational inefficiency problem owing to the asymmetrical workspace mapping in master-slave hydraulic manipulators. An augmented reality-based head-mounted display system combining binocular vision with real-time virtual modeling was designed for demonstrating the commanded motion, and a hidden-appearance visualization rule was established for reducing visual interference, caused by the occlusion effect. An auditory feedback interface was designed for providing an additional cue for the master-slave position error. Eighteen participants were required for executing two typical tasks for validation. The test results indicated that the task completion time was reduced using the proposed HMI, and the number of collisions with surrounding items was reduced. Although the subjective feeling evaluation showed no statistical improvement owing to the visual fatigue and limited definition of the augmented reality scene, the proposed HMI is a practical method for improving the operational efficiency of hydraulic manipulators.

Multimodal Art Pose Recognition and Interaction With Human Intelligence Enhancement.

This paper provides an in-depth study and analysis of human artistic poses through intelligently enhanced multimodal artistic pose recognition. A complementary network model architecture of multimodal information based on motion energy proposed. The network exploits both the rich information of appearance features provided by RGB data and the depth information provided by depth data as well as the characteristics of robustness to luminance and observation angle. The multimodal fusion is accomplished by the complementary information characteristics of the two modalities. Moreover, to better model the long-range temporal structure while considering action classes with sub-action sharing phenomena, an energy-guided video segmentation method is employed. And in the feature fusion stage, a cross-modal cross-fusion approach is proposed, which enables the convolutional network to share local features of two modalities not only in the shallow layer but also to obtain the fusion of global features in the deep convolutional layer by connecting the feature maps of multiple convolutional layers. Firstly, the Kinect camera is used to acquire the color image data of the human body, the depth image data, and the 3D coordinate data of the skeletal points using the Open pose open-source framework. Then, the action automatically extracted from keyframes based on the distance between the hand and the head, and the relative distance features are extracted from the keyframes to describe the action, the local occupancy pattern features and HSV color space features are extracted to describe the object, and finally, the feature fusion is performed and the complex action recognition task is completed. To solve the consistency problem of virtual-reality fusion, the mapping relationship between hand joint point coordinates and the virtual scene is determined in the augmented reality scene, and the coordinate consistency model of natural hand and virtual model is established; finally, the real-time interaction between hand gesture and virtual model is realized, and the average correct rate of its hand gesture reaches 99.04%, which improves the robustness and real-time interaction of hand gesture recognition.

Application of Virtual Reality Technology in the Maintenance of Steam Turbine

Traditional steam turbine maintenance work has many shortcomings, such as poor effectiveness, high cost, and long cycle. In order to solve this problem and improve the professional skills and maintenance efficiency of steam turbine maintenance personnel, we first collect steam turbine equipment drawings and 3D dimension data of parts, and use 3D modeling software to establish augmented reality scene model. Then combined with the virtual reality technology and image fusion technology to realize the maintenance procedure training and maintenance auxiliary guidance of the steam turbine unit. The practical application results show that this technology can realize the virtual and real operation of the model by users in augmented reality scenes, bring interactive and immersive experience to operators, and improve the work efficiency of maintenance personnel.

CIRO: The Effects of Visually Diminished Real Objects on Human Perception in Handheld Augmented Reality

Augmented reality (AR) scenes often inadvertently contain real world objects that are not relevant to the main AR content, such as arbitrary passersby on the street. We refer to these real-world objects as content-irrelevant real objects (CIROs). CIROs may distract users from focusing on the AR content and bring about perceptual issues (e.g., depth distortion or physicality conflict). In a prior work, we carried out a comparative experiment investigating the effects on user perception of the AR content by the degree of the visual diminishment of such a CIRO. Our findings revealed that the diminished representation had positive impacts on human perception, such as reducing the distraction and increasing the presence of the AR objects in the real environment. However, in that work, the ground truth test was staged with perfect and artifact-free diminishment. In this work, we applied an actual real-time object diminishment algorithm on the handheld AR platform, which cannot be completely artifact-free in practice, and evaluated its performance both objectively and subjectively. We found that the imperfect diminishment and visual artifacts can negatively affect the subjective user experience.

Development of Augmented Reality System for Productivity Enhancement in Offshore Plant Construction

As the scale of offshore plants has gradually increased, the amount of management points has significantly increased. Therefore, there are needs for innovative process control, quality management, and an installation support system to improve productivity and efficiency for timely construction. In this paper, we introduce a novel approach to deal with these issues using augmented reality (AR) technology. The core of successful AR implementation is up to scene matching through accurate pose (position and alignment) estimation using an AR camera. To achieve this, this paper first introduces an accurate marker registration technique that can be used in huge structures. In order to improve the precision of marker registration, we propose a method that utilizes the natural feature points and the marker corner points in the optimization step simultaneously. Subsequently, a method of precisely generating AR scenes by utilizing these registered markers is described. Finally, to validate the proposed method, the best practices and its effects are introduced. Based on the proposed AR system, construction workers are now able to quickly navigate to onboard destinations by themselves. In addition, they are able to intuitively install and inspect outfitting parts without paper drawings. Through field tests and surveys, we confirm that AR-based inspection has a significant time-saving effect compared to conventional drawing-based inspection.

Construction of routes in augmented reality using GIS technologies

At present, one of the promising directions of development in the field of information technologies is augmented reality, which offers access to real-world data through the device's camera using an interactive data display approach. Augmented reality is widely applied in various areas of life: beginning from game applications, finishing with professional applications for briefing, training for personnel in a manufacturing sector, heavy mechanical engineering and the high-tech sector, thereby providing an opportunity to use virtual instructions for implementation of the qualified operations. Also, the main way to apply augmented reality is location service - the possibility to track the position of objects on the geographic information map in real time. The article discusses ways of constructing routes in augmented reality on the field of location service data and explores the peculiarities of application. Problems identified when applying the above method are analyzed. An example of route construction is given on the map with reference to location service data and on the stage of augmented reality with calculated coordinates of route points. Methods of converting geographical coordinates into coordinates of augmented reality scene are presented.

Augmented Reality for Interactive Visualization of 3D Maxillofacial Prosthetic Data

To introduce an augmented reality (AR) application for students, patients, and practitioners to use to visualize a maxillectomy defect, the associated obturator prosthesis, and the eventual effects on the patient's facial appearance. Various 3D virtual models related to maxillectomy defects and their prosthetic rehabilitations were used to illustrate the handheld AR application. The AR "scene" was created using Vuforia and Unity3D software. Within this scene, 3D virtual models were linked to the target image by detecting a symbol track marker. The final file was saved as an installable application and exported for Android smart devices. The target image containing the maxillectomy defect and the prosthetic rehabilitation were successfully visualized interactively in 3D mode using the handheld AR maxillofacial prosthetic application. An AR application for visualizing maxillofacial prosthetic data could interestingly be developed as a pedagogic tool to explain prosthetic treatments.

Detection of real-time augmented reality scene light sources and construction of photorealis tic rendering framework

In this paper, the main network of multi-channel light sources is improved, so that multi-channel pictures can be fused for joint training. Secondly, for high-resolution detection pictures, the huge memory consumption leads to a reduction in batches and then affects the model distribution. Group regularization is adopted. We can still train the model normally in small batches; then, combined with the method of the regional candidate network, the final detection accuracy and the accuracy of the candidate frame regression are improved. Finally, through in-depth analysis, based on image lighting technology and physical-based rendering theory, the requirements for lighting effects and performance limitations, combined with a variety of image enhancement technologies, such as gamma correction, HDR, and these technologies used in Java. Real-time lighting algorithms that currently run efficiently on mainstream PCs. The algorithm can be well integrated into the existing rasterization rendering pipeline, while into account better lighting effects and higher operating efficiency. Finally, the lighting effects achieved by the algorithm are tested and compared through experiments. This algorithm not only achieves a very good light and shadow effect when rendering virtual objects with a real scene as the background but also can meet the realistic rendering of picture frames in more complex scenes. Rate requirements. The experimental results show that the virtual light source automatically generated by this algorithm can approximate the lighting of the real scene, and the virtual object and the real object can produce approximately consistent lighting effects in an augmented reality environment with one or more real light sources.

Auto-fit two movable projections into a single 3D display screen for virtual and augmented reality scenes

Auto-fit two movable projections into a single 3D display screen for virtual and augmented reality scenes

Real-time Illumination and Visual Coherence for Photorealistic Augmented/Mixed Reality

A realistically inserted virtual object in the real-time physical environment is a desirable feature in augmented reality (AR) applications and mixed reality (MR) in general. This problem is considered a vital research area in computer graphics, a field that is experiencing ongoing discovery. The algorithms and methods used to obtain dynamic and real-time illumination measurement, estimating, and rendering of augmented reality scenes are utilized in many applications to achieve a realistic perception by humans. We cannot deny the powerful impact of the continuous development of computer vision and machine learning techniques accompanied by the original computer graphics and image processing methods to provide a significant range of novel AR/MR techniques. These techniques include methods for light source acquisition through image-based lighting or sampling, registering and estimating the lighting conditions, and composition of global illumination. In this review, we discussed the pipeline stages with the details elaborated about the methods and techniques that contributed to the development of providing a photo-realistic rendering, visual coherence, and interactive real-time illumination results in AR/MR.

Presentation and interaction of Internet of Things data based on augmented reality

In recent years, AR (Augmented Reality) technology has gradually entered people’s horizons and has become a research hotspot in the field of image processing. At the same time, the mobility and interactivity of AR have also been valued. This paper studies the interaction technology in the AR scene, and proposes a scheme to detect the user’s gaze at the cone perspective. The Leapmotion body sensor was used for gesture extension, and the hardware connection and data transmission scheme of Leapmotion and Holo Lens was designed. Based on the three input modes of sight, gesture and voice, the interaction logic between the user and the system in the AR scene is designed. The network of the Internet of Things is dominated by wireless sensor networks (Wireless Sensor Networks). This paper studies the problem of congestion in wireless sensor network sensing data transmission. WSNs nodes need to transmit the sensed data or preprocessed sensed data to sink nodes. When the data packets (or packets) received by a sensor node exceed its upper limit for forwarding or processing data packets, the excess data packets need to be buffered. When the memory of cached packets is full, excess packets are discarded, causing buffer overflows and causing node-level congestion. This paper proposes an energy balance strategy based on feedback control. The feedback control algorithm is used to adjust the node’s transmit power to realize the redistribution of node load in the network, solve the problem of node energy imbalance in wireless sensor networks, and further extend the sensor network.

Factored Occlusion: Single Spatial Light Modulator Occlusion-capable Optical See-through Augmented Reality Display.

Occlusion is a powerful visual cue that is crucial for depth perception and realism in optical see-through augmented reality (OST-AR). However, existing OST-AR systems additively overlay physical and digital content with beam combiners - an approach that does not easily support mutual occlusion, resulting in virtual objects that appear semi-transparent and unrealistic. In this work, we propose a new type of occlusion-capable OST-AR system. Rather than additively combining the real and virtual worlds, we employ a single digital micromirror device (DMD) to merge the respective light paths in a multiplicative manner. This unique approach allows us to simultaneously block light incident from the physical scene on a pixel-by-pixel basis while also modulating the light emitted by a light-emitting diode (LED) to display digital content. Our technique builds on mixed binary/continuous factorization algorithms to optimize time-multiplexed binary DMD patterns and their corresponding LED colors to approximate a target augmented reality (AR) scene. In simulations and with a prototype benchtop display, we demonstrate hard-edge occlusions, plausible shadows, and also gaze-contingent optimization of this novel display mode, which only requires a single spatial light modulator.

Development of a tracheal model for an endotracheal suction simulator using augmented reality scene with haptics

Development of a tracheal model for an endotracheal suction simulator using augmented reality scene with haptics

Bare-Hand Depth Perception Used in Augmented Reality Assembly Supporting

Augmented Reality (AR) had been used in mechanical assembly supporting. The current reported researches of AR assembly supporting system (ARASS) less concerned about the interaction between the operators' hands and virtual objects, especially on the occlusion between hands and virtual objects. The correct occlusion results would make ARASS more immersive and then the operators' interaction experiences were increased. To address this issue, this paper presented a bared-hand depth perception method that was designed for improving interaction experience in ARASS. The method was based on an interaction scene that was designed for AR mechanical assembly supporting. The method consisted of two components: hand segmentation and hand mesh generation. Hand segmentation method segmented operator's hand areas from depth image of scene and divided hand area into several sub areas that provided hand information. Hand surface mesh generation method generated hand surface meshes in 3D space based on the results of hand segmentation. The meshes were used to solve the occlusion problem of hand area in AR scene. The results verified that the bared-hand depth perception method could handle the occlusion between operator's hands and virtual objects correctly in real-time and recognize hand information in a limited space. The method could increase operators' depth perception of bared hand and make ARASS more immersive.

The Use of Augmented Reality to Raise Awareness of the Differences Between Osteoarthritis and Rheumatoid Arthritis.

Arthritis is one of the most common disease states worldwide but is still publicly misunderstood and lacks engaging public awareness materials. Within the UK, the most prevalent types of arthritis are osteoarthritis (OA) and rheumatoid arthritis (RA). The two are commonly mistaken as the same disease but, in fact, have very different pathogenesis, symptoms and treatments. This chapter describes a study which aimed to assess whether an augmented reality (AR) application could be used to raise awareness about the difference between OA and RA.An application was created for Android tablets that included labelled 3D models, animations and AR scenes triggered from a poster. In total 11 adult participants tested the application taking part in a pretest and posttest which aim to measure the usability of the application and the acquisition of knowledge on OA and RA. A T-test was performed to assess the effectiveness of the application from the pretest and posttest questionnaire outcomes. Overall results were encouraging reporting a very significant acquisition of knowledge and a highly satisfactory user experience.

ARTS, an AR Tourism System, for the Integration of 3D Scanning and Smartphone AR in Cultural Heritage Tourism and Pedagogy.

Interactions between cultural heritage, tourism, and pedagogy deserve investigation in an as-built environment under a macro- or micro-perspective of urban fabric. The heritage site of Shih Yih Hall, Lukang, was explored. An Augmented Reality Tourism System (ARTS) was developed on a smartphone-based platform for a novel application scenario using 3D scans converted from a point cloud to a portable interaction size. ARTS comprises a real-time environment viewing module, a space-switching module, and an Augmented Reality (AR) guide graphic module. The system facilitates scenario initiations, projection and superimposition, annotation, and interface customization, with software tools developed using ARKit® on the iPhone XS Max®. The three-way interaction between urban fabric, cultural heritage tourism, and pedagogy was made possible through background block-outs and an additive or selective display. The illustration of the full-scale experience of the smartphone app was made feasible for co-relating the cultural dependence of urban fabric on tourism. The great fidelity of 3D scans and AR scenes act as a pedagogical aid for students or tourists. A Post-Study System Usability Questionnaire (PSSUQ) evaluation verified the usefulness of ARTS.

Influence of virtual objects' shadows and lighting coherence on distance perception in optical see‐through augmented reality

AbstractThis paper focuses on how virtual objects' shadows as well as differences in alignment between virtual and real lighting influence distance perception in optical see‐through (OST) augmented reality (AR). Four hypotheses are proposed: (H1) Participants underestimate distances in OST AR; (H2) Virtual objects' shadows improve distance judgment accuracy in OST AR; (H3) Shadows with different realism levels have different influence on distance perception in OST AR; (H4) Different levels of lighting misalignment between real and virtual lights have different influence on distance perception in OST AR scenes. Two experiments were designed with an OST head mounted display (HMD), the Microsoft HoloLens. Participants had to match the position of a virtual object displayed in the OST‐HMD with a real target. Distance judgment accuracy was recorded under the different shadows and lighting conditions. The results validate hypotheses H2 and H4 but surprisingly showed no impact of the shape of virtual shadows on distance judgment accuracy thus rejecting hypothesis H3. Regarding hypothesis H1, we detected a trend toward underestimation; given the high variance of the data, more experiments are needed to confirm this result. Moreover, the study also reveals that perceived distance errors and completion time of trials increase along with targets' distance.

Advanced Volumetric Capture and Processing

Volumetric video is regarded worldwide as the next important development in media production, especially in the context of rapidly evolving virtual and augmented reality markets where volumetric video is becoming a key technology. Fraunhofer Heinrich Hertz Institute (HHI) has developed a novel technology for volumetric video. The 3D Human Body Reconstruction (3DHBR) technology captures real persons with our novel volumetric capture system and creates naturally moving dynamic 3D models, which can then be observed from arbitrary viewpoints in a virtual or augmented reality scene. The capture system consists of an integrated multicamera and lighting system for a full 360° acquisition. A cylindrical studio has been developed with a diameter of 6 m and consists of 32 20-MPixel cameras and 120 light-emitting-diode (LED) panels that allow for an arbitrary lit background. Hence, diffuse lighting and automatic keying are supported. The avoidance of green screen and provision of diffuse lighting offers the best possible conditions for relighting of the dynamic 3D models afterward at the design stage of the virtual reality (VR) experience. In contrast to classical character animation, facial expressions and moving clothes are reconstructed at high geometrical detail and texture quality. The complete workflow is fully automatic, requires about 12 hr/min of mesh sequence, and provides a high level of quality for immediate integration in virtual scenes. Meanwhile, a second, professional studio has been built up on the film campus of Potsdam Babelsberg. This studio is operated by VoluCap GmbH, a joint venture between Studio Babelsberg, ARRI, UFA, Interlake, and Fraunhofer Heinrich Hertz Institute (HHI).