Proposed Augmented Reality System Research Articles

Supporting Visualization Analysis in Industrial Process Tomography by Using Augmented Reality-A Case Study of an Industrial Microwave Drying System.

Industrial process tomography (IPT) based process control is an advisable approach in industrial heating processes for improving system efficiency and quality. When using it, appropriate dataflow pipelines and visualizations are key for domain users to implement precise data acquisition and analysis. In this article, we propose a complete data processing and visualizing workflow regarding a specific case—microwave tomography (MWT) controlled industrial microwave drying system. Furthermore, we present the up-to-date augmented reality (AR) technique to support the corresponding data visualization and on-site analysis. As a pioneering study of using AR to benefit IPT systems, the proposed AR module provides straightforward and comprehensible visualizations pertaining to the process data to the related users. Inside the dataflow of the case, a time reversal imaging algorithm, a post-imaging segmentation, and a volumetric visualization module are included. For the time reversal algorithm, we exhaustively introduce each step for MWT image reconstruction and then present the simulated results. For the post-imaging segmentation, an automatic tomographic segmentation algorithm is utilized to reveal the significant information contained in the reconstructed images. For volumetric visualization, the 3D generated information is displayed. Finally, the proposed AR system is integrated with the on-going process data, including reconstructed, segmented, and volumetric images, which are used for facilitating interactive on-site data analysis for domain users. The central part of the AR system is implemented by a mobile app that is currently supported on iOS/Android platforms.

A Projector-Based Augmented Reality Navigation System for Computer-Assisted Surgery.

In the medical field, guidance to follow the surgical plan is crucial. Image overlay projection is a solution to link the surgical plan with the patient. It realizes augmented reality (AR) by projecting computer-generated image on the surface of the target through a projector, which can visualize additional information to the scene. By overlaying anatomical information or surgical plans on the surgery area, projection helps to enhance the surgeon’s understanding of the anatomical structure, and intuitively visualizes the surgical target and key structures of the operation, and avoid the surgeon’s sight diversion between monitor and patient. However, it still remains a challenge to project the surgical navigation information on the target precisely and efficiently. In this study, we propose a projector-based surgical navigation system. Through the gray code-based calibration method, the projector can be calibrated with a camera and then be integrated with an optical spatial locator, so that the navigation information of the operation can be accurately projected onto the target area. We validated the projection accuracy of the system through back projection, with average projection error of 3.37 pixels in x direction and 1.51 pixels in y direction, and model projection with an average position error of 1.03 ± 0.43 mm, and carried out puncture experiments using the system with correct rate of 99%, and qualitatively analyzed the system’s performance through the questionnaire. The results demonstrate the efficacy of our proposed AR system.

Augmented Reality as a Tool in the Teaching-Learning Process of Natural Sciences in a Primary Institution

Recent challenges for strategic educational planning in Ecuador request implement Information and Communication Technologies (ICT) in different levels of educational institutions. However, there´s a notable lack of successful implementations of ICTs on national educational institutions, which causes several problems in student's learning cycle such as loss of attention, motivation and innovation. These problems are further aggravated at the level of primary education institutions. Nevertheless, this paper demonstrates the feasibility of implementing digital tools with augmented reality (AR) for teaching and learning of Natural Sciences (NC) in the Cesar Mosquera Primary School located at the northern zone of Ecuador. To this end, we began by identifying digital strategies that facilitate the teaching of NC as an element of feedback and development of spatial imagination. Next, a diagnosis was made of the weaknesses of both teachers and students in the teaching of NC at primary school level. A mixed research method was applied, using quantitative techniques such as survey analysis and using qualitative techniques such as direct observations, to make a content analysis applied to teacher planning. Then, cutting-edge technologies such as augmented reality deployed on easily accessible devices such as Smartphones or Tablets were implemented using Curiscope-Virtuali-Tee software and web 3.0 virtual classroom tools that were applied during the COVID-19 pandemic. Once implemented, the proposed AR system was compared with classic teaching methods. Finally, the feasibility of applying AR in the mentioned institution was demonstrated due to all students approved the final test of acquired knowledge in each class taken.

A novel visualization system of using augmented reality in knee replacement surgery: Enhanced bidirectional maximum correntropy algorithm.

Image registration and alignment are the main limitations of augmented reality (AR)-based knee replacement surgery. This research aims to decrease the registration error, eliminate outcomes that are trapped in local minima to improve the alignment problems, handle the occlusion and maximize the overlapping parts. Markerless image registration method was used for AR-based knee replacement surgery to guide and visualize the surgical operation. While weight least square algorithm was used to enhance stereo camera-based tracking by filling border occlusion in right-to-left direction and non-border occlusion from left-to-right direction. This study has improved video precision to 0.57-0.61mm alignment error. Furthermore, with the use of bidirectional points, that is, forward and backward directional cloud point, the iteration on image registration was decreased. This has led to improve the processing time as well. The processing time of video frames was improved to 7.4-11.74 frames per second. It seems clear that this proposed system has focused on overcoming the misalignment difficulty caused by the movement of patient and enhancing the AR visualization during knee replacement surgery. The proposed system was reliable and favourable which helps in eliminating alignment error by ascertaining the optimal rigid transformation between two cloud points and removing the outliers and non-Gaussian noise. The proposed AR system helps in accurate visualization and navigation of anatomy of knee such as femur, tibia, cartilage, blood vessels and so forth.

Developing the Next Generation of Augmented Reality Games for Pediatric Healthcare: An Open-Source Collaborative Framework Based on ARCore for Implementing Teaching, Training and Monitoring Applications.

Augmented Reality (AR) provides an alternative to the traditional forms of interaction between humans and machines, and facilitates the access to certain technologies to groups of people with special needs like children. For instance, in pediatric healthcare, it is important to help children to feel comfortable during medical procedures and tests that may be performed on them. To tackle such an issue with the help of AR-based solutions, this article presents the design, implementation and evaluation of a novel open-source collaborative framework that enables to develop teaching, training, and monitoring pediatric healthcare applications. Specifically, such a framework allows for building collaborative applications and shared experiences for AR devices, providing functionalities for connecting with other AR devices and enabling real-time visualization and simultaneous interaction with virtual objects. Since all the communications involved in AR interactions are handled by AR devices, the proposed collaborative framework is able to operate autonomously through a Local Area Network (LAN), thus requiring no cloud or external servers. In order to demonstrate the potential of the proposed framework, a practical use case application is presented. Such an application has been designed to motivate pediatric patients and to encourage them to increase their physical activity through AR games. The presented games do not require any previous configuration, as they use ARCore automatic surface detection technology. Moreover, the AR mobile gaming framework allows multiple players to engage in the same AR experience, so children can interact and collaborate among them sharing the same AR content. In addition, the proposed AR system provides a remote web application that is able to collect and to visualize data on patient use, aiming to provide healthcare professionals with qualified data about the mobility and mood of their patients through an intuitive and user-friendly web tool. Finally, to determine the performance of the proposed AR system, this article presents its evaluation in terms of latency and processing time. The results show that both times are low enough to provide a good user experience.

Development of Augmented Reality System for Productivity Enhancement in Offshore Plant Construction

As the scale of offshore plants has gradually increased, the amount of management points has significantly increased. Therefore, there are needs for innovative process control, quality management, and an installation support system to improve productivity and efficiency for timely construction. In this paper, we introduce a novel approach to deal with these issues using augmented reality (AR) technology. The core of successful AR implementation is up to scene matching through accurate pose (position and alignment) estimation using an AR camera. To achieve this, this paper first introduces an accurate marker registration technique that can be used in huge structures. In order to improve the precision of marker registration, we propose a method that utilizes the natural feature points and the marker corner points in the optimization step simultaneously. Subsequently, a method of precisely generating AR scenes by utilizing these registered markers is described. Finally, to validate the proposed method, the best practices and its effects are introduced. Based on the proposed AR system, construction workers are now able to quickly navigate to onboard destinations by themselves. In addition, they are able to intuitively install and inspect outfitting parts without paper drawings. Through field tests and surveys, we confirm that AR-based inspection has a significant time-saving effect compared to conventional drawing-based inspection.

Immersive augmented reality system for the training of pattern classification control with a myoelectric prosthesis

BackgroundHand amputation can have a truly debilitating impact on the life of the affected person. A multifunctional myoelectric prosthesis controlled using pattern classification can be used to restore some of the lost motor abilities. However, learning to control an advanced prosthesis can be a challenging task, but virtual and augmented reality (AR) provide means to create an engaging and motivating training.MethodsIn this study, we present a novel training framework that integrates virtual elements within a real scene (AR) while allowing the view from the first-person perspective. The framework was evaluated in 13 able-bodied subjects and a limb-deficient person divided into intervention (IG) and control (CG) groups. The IG received training by performing simulated clothespin task and both groups conducted a pre- and posttest with a real prosthesis. When training with the AR, the subjects received visual feedback on the generated grasping force. The main outcome measure was the number of pins that were successfully transferred within 20 min (task duration), while the number of dropped and broken pins were also registered. The participants were asked to score the difficulty of the real task (posttest), fun-factor and motivation, as well as the utility of the feedback.ResultsThe performance (median/interquartile range) consistently increased during the training sessions (4/3 to 22/4). While the results were similar for the two groups in the pretest, the performance improved in the posttest only in IG. In addition, the subjects in IG transferred significantly more pins (28/10.5 versus 14.5/11), and dropped (1/2.5 versus 3.5/2) and broke (5/3.8 versus 14.5/9) significantly fewer pins in the posttest compared to CG. The participants in IG assigned (mean ± std) significantly lower scores to the difficulty compared to CG (5.2 ± 1.9 versus 7.1 ± 0.9), and they highly rated the fun factor (8.7 ± 1.3) and usefulness of feedback (8.5 ± 1.7).ConclusionThe results demonstrated that the proposed AR system allows for the transfer of skills from the simulated to the real task while providing a positive user experience. The present study demonstrates the effectiveness and flexibility of the proposed AR framework. Importantly, the developed system is open source and available for download and further development.

Augmented reality display system using modulated moiré imaging technique.

To enhance the depth rendering ability of augmented reality (AR) display systems, a modulated moiré imaging technique is used to render the true three-dimensional (3D) images for AR display systems. 3D images with continuous depth information and large depth of field are rendered and superimposed on the real scene. The proposed AR system consists of a modulated moiré imaging subsystem and an optical combiner. The modulated moiré imaging subsystem employs modulated point light sources, a display device, and a microlens array to generate 3D images. A defocussing equal period moiré imaging structure is used, which gives a chance for the point light sources to modulate the depth position of 3D images continuously. The principles of the imaging system are deduced analytically. A custom-designed transparent off-axis spherical reflective lens is used as an optical combiner to project the 3D images into the real world. An experimental AR system that provides continuous 3D images with depth information ranging from 0.5 to 2.5 m is made to verify the feasibility of the proposed technique.

A novel interactive assembly teaching aid using multi-template augmented reality

Augmented Reality (AR) technology is a fast growing field in the academics and industry. This paper presents a novel design of an interactive assembly teaching aid based on a multi-template AR system, which consists of three units: a multi-template AR unit, an online three-dimension (3D) model assembly unit, a hand-gesture interaction unit. The design of the multi-template AR unit employs an efficient multi-template pose tracking method to detect and track multiple template images simultaneously. The online 3D model assembly unit is enabled, when the pose of each target template is tracked and computed by the multi-template pose tracking method. This method measures the distance between the two templates to determine the 3D rendering mode of the virtual object. The third unit aims to realize a vision-based human-computer interactive system, which combines the AR rendering system with the real-time hand-gesture recognition method to decide the status of AR rendering of a 3D dynamic animation according to the user’s hand gesture. Based on the feedback of twenty-one users of the AR-based interactive teaching-aid, it can be concluded that the system creates an interactive experiences that engages the users and facilitates them to increase their learning interest significantly. In future, we plan to port the proposed AR system to Android and iOS mobile devices while improving the functionality, interactivity, and entertaining qualities of the teaching aid system.

Varifocal augmented reality adopting electrically tunable uniaxial plane-parallel plates.

Vergence-accommodation conflict (VAC) is a major challenge in optical-see through augmented reality (AR) system. To resolve this conflict, many approaches are proposed, for instance, by means of adjustment of the projected virtual image to coincide with the surroundings, called image registration, which is more often referred to as varifocal function. In this paper, a varifocal AR system is demonstrated by adopting electrically tunable liquid crystal (LC) plane-parallel plates to solve VAC problem. The LC plates provide electrically tunable optical paths when the directors of LC molecules are re-orientated with applied voltages, which leads to a corresponding change of light speed for an extraordinary wave. To provide a sufficient tunable optical path, three pieces of multiple-layered LC structures are used with the total thickness of the active LC layers (∼510 μm). In experiments, the projected virtual image can be adjusted from 1.4 m to 2.1 m away from the AR system, while the thickness of LC plane-parallel plates are only less than 3 mm without any mechanical moving part. When light propagates in the uniaxial LC layers, the wave vector and the Poynting vector are different. The longitudinal displacement of the image plane is determined by Poynting vectors rather than wave vectors. As a result, the analysis of the AR system should be based on Poynting vectors during geometrical optical analysis. Surprisingly, the tunable range of the longitudinal displacement of Poynting vectors is 2-fold larger than the tunable range of the wave vectors. Moreover, the virtual image shifts in opposite directions with respect to the Poynting vectors and wave vectors. The proposed AR system is not only simple but also thin, and it exhibits a large clear aperture. The investigation here paves the way to a simple solution of the VAC problem for augmented reality systems.

Real-time human segmentation by BowtieNet and a SLAM-based human AR system

BackgroundGenerally, it is difficult to obtain accurate pose and depth for a non-rigid moving object from a single RGB camera to create augmented reality (AR). In this study, we build an augmented reality system from a single RGB camera for a non-rigid moving human by accurately computing pose and depth, for which two key tasks are segmentation and monocular Simultaneous Localization and Mapping (SLAM). Most existing monocular SLAM systems are designed for static scenes, while in this AR system, the human body is always moving and non-rigid. MethodsIn order to make the SLAM system suitable for a moving human, we first segment the rigid part of the human in each frame. A segmented moving body part can be regarded as a static object, and the relative motions between each moving body part and the camera can be considered the motion of the camera. Typical SLAM systems designed for static scenes can then be applied. In the segmentation step of this AR system, we first employ the proposed BowtieNet, which adds the atrous spatial pyramid pooling (ASPP) of DeepLab between the encoder and decoder of SegNet to segment the human in the original frame, and then we use color information to extract the face from the segmented human area. ResultsBased on the human segmentation results and a monocular SLAM, this system can change the video background and add a virtual object to humans. ConclusionsThe experiments on the human image segmentation datasets show that BowtieNet obtains state-of-the-art human image segmentation performance and enough speed for real-time segmentation. The experiments on videos show that the proposed AR system can robustly add a virtual object to humans and can accurately change the video background.

Augmented reality near-eye display using Pancharatnam-Berry phase lenses

An augmented reality (AR) near-eye display using Pancharatnam-Berry (PB) phase lenses is proposed. PB phase lenses provide different optical effects depending on the polarization state of the incident light. By exploiting this characteristic, it is possible to manufacture an AR combiner with a small form factor and a large numerical aperture value. The AR combiner adopted in the proposed system operates as a convex lens for right-handed circularly polarized light and operates as transparent glass for left-handed circularly polarized light. By merging this combiner with a transparent screen, such as diffuser-holographic optical elements (DHOEs), it is possible to make an AR near-eye display with a small form factor and a wide field of view. In addition, the proposed AR system compensates the chromatic aberration that occurs in PB phase lens by adopting three-layered DHOEs. The operating principle of the proposed system is covered, and its feasibility is verified with experiments and analysis.

Using Augmented Reality to Elicit Pretend Play for Children with Autism.

Children with autism spectrum condition (ASC) suffer from deficits or developmental delays in symbolic thinking. In particular, they are often found lacking in pretend play during early childhood. Researchers believe that they encounter difficulty in generating and maintaining mental representation of pretense coupled with the immediate reality. We have developed an interactive system that explores the potential of Augmented Reality (AR) technology to visually conceptualize the representation of pretense within an open-ended play environment. Results from an empirical study involving children with ASC aged 4 to 7 demonstrated a significant improvement of pretend play in terms of frequency, duration and relevance using the AR system in comparison to a non computer-assisted situation. We investigated individual differences, skill transfer, system usability and limitations of the proposed AR system. We discuss design guidelines for future AR systems for children with ASC and other pervasive developmental disorders.

Mobile augmented reality for computer-assisted percutaneous nephrolithotomy

Percutaneous nephrolithotomy (PCNL) plays an integral role in treatment of renal stones. Creating percutaneous renal access is the most important and challenging step in the procedure. To facilitate this step, we evaluated our novel mobile augmented reality (AR) system for its feasibility of use for PCNL. A tablet computer, such as an iPad[Formula: see text], is positioned above the patient with its camera pointing toward the field of intervention. The images of the tablet camera are registered with the CT image by means of fiducial markers. Structures of interest can be superimposed semi-transparently on the video images. We present a systematic evaluation by means of a phantom study. An urological trainee and two experts conducted 53 punctures on kidney phantoms. The trainee performed best with the proposed AR system in terms of puncturing time (mean: 99 s), whereas the experts performed best with fluoroscopy (mean: 59 s). iPad assistance lowered radiation exposure by a factor of 3 for the inexperienced physician and by a factor of 1.8 for the experts in comparison with fluoroscopy usage. We achieve a mean visualization accuracy of 2.5 mm. The proposed tablet computer-based AR system has proven helpful in assisting percutaneous interventions such as PCNL and shows benefits compared to other state-of-the-art assistance systems. A drawback of the system in its current state is the lack of depth information. Despite that, the simple integration into the clinical workflow highlights the potential impact of this approach to such interventions.

Real-time markerless Augmented Reality for Remote Handling system in bad viewing conditions

Remote Handling (RH) in harsh environments usually has to tackle the lack of sufficient visual feedback for the human operator due to the limited number of on-site cameras, the not optimized position of the cameras, the poor viewing angles, occlusion, failure, etc. Augmented Reality (AR) enables the user to perceive virtual computer-generated objects in a real scene. The most common goals usually include visibility enhancement and provision of extra information, such as positional data of various objects. The proposed AR system first recognizes and locates the markerless object by using a template based matching algorithm, and then augments the virtual model on top of the recognized item. The tracking algorithm is exploited for locating the object in a continuous sequence of frames. Conceptually, the template is found by computing the similarity between the template and the image frame, for all the relevant template poses (rotation and translation). As a case study, AR interface was displaying measured orientation and transformation of the Water Hydraulic Manipulator (WHMAN) Divertor preloading tool, in near real-time tracking. The bad viewing condition implies on the case when the view angle is such that the interesting features of the object are not in the field of view. The method in this paper was validated in concrete operational context at DTP2. The developed method proved to deliver robust positional and orientation information while augmenting and tracking the moving tool object.

Augmented reality for assembly guidance using a virtual interactive tool

The application of augmented reality (AR) technology for assembly guidance is a novel approach in the traditional manufacturing domain. In this paper, we propose an AR approach for assembly guidance using a virtual interactive tool that is intuitive and easy to use. The virtual interactive tool, termed the virtual interaction panel (VirIP), is an easy-to-use tool that can be used to interactively control AR systems. The VirIP is composed of virtual buttons, which have meaningful assembly information that can be activated by an interaction pen during the assembly process. The interaction pen can be any general pen-like object with a certain colour distribution. It is tracked using a restricted coulomb energy (RCE) network in real-time and used to trigger the relevant buttons in the VirIPs for assembly guidance. Meanwhile, a visual assembly tree structure (VATS) is used for information management and assembly instructions retrieval in this AR environment. VATS is a hierarchical tree structure that can be easily maintained via a visual interface. It can be directly integrated into the AR system or it can alternatively act as an independent central control station on a remote computer to control the data flow of the assembly information. This paper describes a typical scenario for assembly guidance using VirIP and VATS. The main characteristic of the proposed AR system is the intuitive way in which an assembly operator can easily step through a pre-defined assembly plan/sequence without the need of any sensor schemes or markers attached on the assembly components. Several experiments were conducted to validate the performance of the proposed AR-based method using a monitor and a head-mounted display. The results show that the AR-based method can provide an efficient way for assembly guidance.