Vision-based Augmented Reality Research Articles

Can A Vision-Based Augmented Reality System Be Used for Middle Ear Surgery?

Can A Vision-Based Augmented Reality System Be Used for Middle Ear Surgery?

Vision-Based Augmented Reality System for Middle Ear Surgery: Evaluation in Operating Room Environment.

Augmented reality (AR) solely based on image features is achievable in operating room conditions and its precision is compatible with otological surgery. The objective of this work was to evaluate the performance of a vision-based AR system for middle ear surgery in the operating room conditions. Nine adult patients undergoing ossicular procedures were included in this prospective study. AR was obtained by combining real-time video from the operating microscope with the virtual image obtained from the preoperative computed tomography (CT)-scan. Initial registration between the video and the virtual CT image was achieved using manual selection of six points on the tympanic sulcus. Patient-microscope movements during the procedure were tracked using image-feature matching algorithm. The microscope was randomly moved at an approximated speed of 5 mm/s in the three axes of space and rotation for 180 seconds. The accuracy of the system was assessed by calculating the distance between each fiducial point selected on the video image and its corresponding point on the scanner. AR could be obtained for at least 3 minutes in seven out of nine patients. The overlay fiducial and target registration errors were 0.38 ± 0.23 mm (n = 7) and 0.36 ± 0.15 mm (n = 5) respectively, with a drift error of 1.2 ± 0.5 μm/s. The system was stable throughout the procedure and achieved a refresh rate of 12 fps. Moderate bleeding and introduction of surgical instruments did not compromise the performance of the system. The AR system yielded sub-millimetric accuracy and remained stable throughout the experimental study despite patient-microscope movements and field of view obtrusions.

Use of vision-based augmented reality to improve student learning of the spine and spinal deformities. An exploratory study.

BackgroundKnowledge of anatomy and pathology of the spine together with spinal deformities is integral to several healthcare disciplines. This knowledge is crucial for graduates for assessment and management of patients with spinal problems. Physiotherapy students generally find it difficult to conceptualise the integrity of the structure and function of the spine that affects their acquisition of related physiotherapy skills.ObjectiveOur first objective was to introduce and evaluate the use of a Vision-Based Augmented Reality (VBAR) mobile application to teach students the anatomy and accessory movements of the spine. A further objective was to explore student experiences of and engagement with VBAR by conducting a post-lecture survey comparing VBAR to traditional teaching.MethodsThis post-intervention crossover design study included two groups: final year physiotherapy students (n = 74) and mean age of 23 (±1.8). The computing department at Teesside University developed the VBAR mobile application. Moreover, a survey adapted from a previously published article was disseminated to students to evaluate their level of understanding following the use of the VBAR application.ResultsThe results demonstrated that the median questionnaire scores in students’ perceived level of understanding for the VBAR group were significantly higher than for the traditional teaching group (p < 0.05).ConclusionThe results of this post-intervention survey suggest that the integration of VBAR learning activities results in gains relating to students’ understanding of spinal anatomy, function, pathology and deformities. These findings suggest that VBAR could be an additional teaching tool to support student learning.Clinical implicationsGreater understanding is expected to increase the quality of clinical practice.

Augmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System

IntroductionIntraoperative navigation during liver resection remains difficult and requires high radiologic skills because liver anatomy is complex and patient-specific. Augmented reality (AR) during open liver surgery could be helpful to guide hepatectomies and optimize resection margins but faces many challenges when large parenchymal deformations take place. We aimed to experiment a new vision-based AR to assess its clinical feasibility and anatomical accuracy. Patients and MethodsBased on preoperative CT scan 3-D segmentations, we applied a non-rigid registration method, integrating a physics-based elastic model of the liver, computed in real time using an efficient finite element method. To fit the actual deformations, the model was driven by data provided by a single RGB-D camera. Five livers were considered in this experiment. In vivo AR was performed during hepatectomy (n = 4), with manual handling of the livers resulting in large realistic deformations. Ex vivo experiment (n = 1) consisted in repeated CT scans of explanted whole organ carrying internal metallic landmarks, in fixed deformations, and allowed us to analyze our estimated deformations and quantify spatial errors. ResultsIn vivo AR tests were successfully achieved in all patients with a fast and agile setup installation (< 10 min) and real-time overlay of the virtual anatomy onto the surgical field displayed on an external screen. In addition, an ex vivo quantification demonstrated a 7.9mm root mean square error for the registration of internal landmarks. ConclusionThese first experiments of a markerless AR provided promising results, requiring very little equipment and setup time, yet providing real-time AR with satisfactory 3D accuracy. These results must be confirmed in a larger prospective study to definitively assess the impact of such minimally invasive technology on pathological margins and oncological outcomes.

Development of pass-through augmented reality interface for human robot interaction

In this technology driven world, human machine interaction is indispensable. Conventional human machine interface techniques pose the challenges of extensive training and increased cognitive load. Usage of hand gestures, face and speech recognition, eye ball tracking, etc. are in the upward trend to overcome these limitations and prove to be more intuitive and user friendly. In this paper, a vision-based pass-through augmented reality system is developed that enables the user to command the mobile robot for various mobility and navigation tasks. A pass-through augmented reality setup is achieved by a combination of a typical virtual reality headset and a stereo camera with depth perception. The stereo camera and a Leap Motion sensor mounted on a wearable VR headset acts as the vision system for the user and the gesture recognition system respectively. Necessary graphical user interface with the list of functions and information will appear on the VR headset display from which the user can select the option/command to be given to the mobile robot. The gesture made by the user in the GUI is detected and communicated to the robot through wireless means to perform the corresponding task.

Intelligent visual enhancement system

Vision-based augmented reality is a new kind of visual application technology, which transfers synthetic sensory information into a user’s perception of a real environment. It is perceived by human senses. However, the existing visual enhancement equipment has single function, limited processing capacity and poor interaction. To overcome these shortcomings, we designed a visual enhancement system that integrates cloud computing, AR technology and deep learning. First, small, remote and wireless cameras are used to obtain image data, which need to be uploaded to a cloud. Then, the method of deep learning and feature matching is adopted to carry out facial consistency analysis, which improves the robustness of target detection. And stable target tracking is achieved by time sequence state filtering. Finally, the information of image analysis and processing is transmitted back to the AR device, so the prompts of target text and voice are given for intelligent auxiliary decision-making in time.

Atom Tracker: Designing a Mobile Augmented Reality Experience to Support Instruction About Cycles and Conservation of Matter in Outdoor Learning Environments

We describe a mobile augmented reality (AR) experience called Atom Tracker designed to help middle school students better understand the cycling of matter in ecosystems with a focus on the concept of conservation of matter and the processes of photosynthesis and respiration. Location-based AR allows students to locate virtual "hotspots," where they interact with multiple representations including vision-based AR animations of virtual atoms during ecological processes such as photosynthesis and physical LEGO® -based representations of molecules. This design case describes the design rationale, the iterative design process, the context for implementation, and reflections on the success and limitations of the Atom Tracker AR experience. An augmented reality interface was chosen due to theoretical support for its utility in supporting interaction with multiple representations (both physical and virtual) of atoms and molecules, the ability to condense and expand temporal and spatial scales associated with ecological processes, and its ability to explicitly situate these representations in real-world contexts that could support learning. Two significant design challenges that we recognized were (a) appropriately leveraging narrative, student engagement and agency when designing around the topic of atoms and molecules, which are inanimate and invisible; and (b) designing for engagement with both virtual and physical resources available during the experience.

Constructing liminal blends in a collaborative augmented-reality learning environment

In vision-based augmented-reality (AR) environments, users view the physical world through a video feed or device that augments the display with a graphical or informational overlay. Our goal in this manuscript is to ask how and why these new technologies create opportunities for learning. We suggest that AR is uniquely positioned to support learning through its ability to support students in developing “conceptual blends”—which we propose extend beyond cognitive spaces to include the layering of multiple ideas and physical materials, often supplied by different conversation participants. We document one case study and trace how the narrative structure of a board game, the physical floor materials (e.g. linoleum), a student’s first-person embodied experiences, the third-person live camera feed, and the augmented-reality symbols become integrated in the activity. As a result, students’ conceptualization of force and friction become fused with a diverse set of intellectual resources. We conclude by suggesting that the framework of liminal blends may inform the design of future AR learning environments and in particular help generate predictions about the ways in which the juxtaposition of certain resources may otherwise produce unexpected results.

Tideland Animal AR: Superimposing 3D Animal Models to User Defined Targets for Augmented Reality Game

With the development of computer vision, mobile platform, and information technologies, there have been many mobile augmented reality applications, contents, and services. The technology of augmented reality has been developed to many application areas such as books, games, advertisements, and tour guide systems. However, most vision-based augmented reality applications have to train some physical objects patterns or paper markers before using it. If the physical objects for the applications are destroyed or damaged, users have to buy new objects or print them out again. To reduce the inconvenience, there has been developed technology that users take pictures of 2D makers what users want and superimpose content on it in real-time. In this paper, we present an application that superimposes 3D animal models living in tideland sequentially whenever users make image targets in real time. We implemented 3D models of 10 animals in tideland, and a mobile application to display the animals on smart phone.

Full paper: Markerless vision-based augmented reality for enhanced project visualization

Full paper: Markerless vision-based augmented reality for enhanced project visualization

Markerless vision-based augmented reality for enhanced project visualization

Purpose This work aims to develop a system that enables improved visualization of Architectural Engineering and Construction (AEC) 3D-information for application in design, construction, and management of the built environment. Method A novel Augmented Reality (AR) system is presented that uses a single standard digital camera and which, contrary to other investigated approaches, does not rely on any markers inserted in the scene, nor on any positioning and inertial technologies. The system is solely image-based and consists of two stages. In a first offline stage, a 3D-map of the scene is automatically constructed from a set of digital images, and the augmenting information (e.g. the 3D-model of the building asset) is subsequently registered with this map. The 3D-map reconstruction employs structure-from-motion techniques with SURF features (the resulting map consisting of a set 3D-referenced SURF-features) followed by a Poisson mesh reconstruction procedure. The next step consists of online operations. The positions of target digital images (e.g. from video stream or head-mounted camera) are automatically calculated, using a robust SURF feature matching procedure that is optimized for three different situations (initialization, tracking, and resetting) implementing octrees for efficient 3D-pruning, and kd-trees for efficient feature matching. Once each input image is positioned within the map, the view is augmented. A notable feature of dense mesh scene reconstruction conducted in the present work is that it enables static occlusions of the scene on the augmenting data to be taken into account. Results & Discussion Several experiments validate the pr posed system and demonstrate its overall performance: a near real-time processing speed, very accurate and stable positioning. The limitations of the current system are also discussed including: the currently limited processing speed and the need for adequately textured scenes.

Structure and motion in urban environments using upright panoramas

Image-based modeling of urban environments is a key component of enabling outdoor, vision-based augmented reality applications. The images used for modeling may come from off-line efforts, or online user contributions. Panoramas have been used extensively in mapping cities and can be captured quickly by an end-user with a mobile phone. In this paper, we describe and evaluate a reconstruction pipeline for upright panoramas taken in an urban environment. We first describe how panoramas can be aligned to a common vertical orientation using vertical vanishing point detection, which we show to be robust for a range of inputs. The orientation sensors in modern cameras can also be used to correct the vertical orientation. Secondly, we introduce a pose estimation algorithm, which uses knowledge of a common vertical orientation as a simplifying constraint. This procedure is shown to reduce pose estimation error in comparison with the state of the art. Finally, we evaluate our reconstruction pipeline with several real-world examples.

Editorial: special issue on augmented reality

It gives us great pleasure to introduce this special issue of the Springer Virtual Reality Journal, dedicated to research in augmented reality (AR) in light of the growing interest in the field. Over 40 years ago, Ivan Sutherland developed the first AR application that allowed virtual images to be seamlessly overlaid onto the real world. Since then, much of the research in the field has been focused on the enabling technology for providing AR experiences, such as tracking, display devices, and software systems. However, in the call for papers, we asked for work with a focus on augmented reality experiences, such as new techniques for interacting with AR, methods for evaluating the quality of the experience, better tools for creating AR experiences for nonprogrammers and other related topics. We received more than thirty submissions, most of which were of a very high quality, and we are delighted to present the final selection of eleven papers. Each of these papers were reviewed by a least four reviewers and have been revised to address the reviewers feedback before appearing here. The papers can be grouped into three categories: Tracking and Projection Methods, Interaction Techniques, and Interface Design and Systems. In the Tracking group, (Lieberknecht et al. 2011) present methods for benchmarking template-based computer vision tracking algorithms for AR. Template-based tracking has been the most popular form of vision-based AR tracking for the past 10 years and their work gives valuable guidance on how to benchmark the performance of such systems. Uchiyama (2011) discusses how to do camera tracking of online learning of key points using a Locally Likely Arrangement Hashing technique which looks promising. Finally, (Nagase et al. 2011) shows how model-based optimal projector selection in multiprojection environment can be used to compensate for dynamic defocus and occlusion in projected imagery. The Interaction papers presented a range of different ways of interacting with AR content. Lee et al.’s work (2011) explores tangible interaction methods and, in particular, how two-handed tangible techniques can be used for arranging AR blocks. In their paper, they show how very natural block arranging methods with real blocks can be used for intuitive manipulation of virtual content. Iwai and Sato’s paper on the Limpid Desk (2011) presents an interesting AR projectionbased interface for seeing virtual imagery projected into real documents. They also use gesture-based interaction to enable touch sensing and browsing through the document stack. Finally, the (Ajanki et al. 2011) paper explores how to develop context-sensitive interaction techniques. Using position and gaze sensing, their system can automatically infer user’s interest in people and topics, and display AR content reflecting their interest and so support implicit rather than explicit interaction. The largest selection of papers is in the Interface Design and AR Systems group. Livingston’s paper (2011) presents general user interface guidelines for military AR applications and, in particular, for presenting outdoor AR content. Kim (2011) describes a related system which allows virtual video to be overlaid on aerial maps. The In-Place AR paper of (Hagbi et al. 2011) also shows an interesting map-based M. Billinghurst (&) HIT Lab NZ, University of Canterbury, Christchurch, New Zealand e-mail: mark.billinghurst@hitlabnz.org

Computer Vision-Based Augmented Reality System for Assembly Interaction

A research effort aimed at creating a computer vision-based augmented reality assembly system is presented. The architecture of this system for assembly interaction is described. To realize intuitive human-computer interaction, a computer vision-based hand tracking and gesture recognition approach is applied. This approach uses the color-based method to segment the hand from background, and locate the hand position by the marker attached on hand, and then recognizes the gesture according to the geometry constraint of human hand. In comparison with the traditional approaches, this new method does not require a stationary camera, and is not sensitive with intensity difference. So, it provides a real time performance and is easy to realize. Moreover, occlusion identification is studied in this paper to raise the real and virtual objects combination effect. Finally, a prototype system is provided to demonstrate the effectiveness and robustness of the presented approaches.

Visual Enhancement for Sports Entertainment by Vision-Based Augmented Reality

This paper presents visually enhanced sports entertainment applications: AR Baseball Presentation System and Interactive AR Bowling System. We utilize vision-based augmented reality for getting immersive feeling. First application is an observation system of a virtual baseball game on the tabletop. 3D virtual players are playing a game on a real baseball field model, so that users can observe the game from favorite view points through a handheld monitor with a web camera. Second application is a bowling system which allows users to roll a real ball down a real bowling lane model on the tabletop and knock down virtual pins. The users watch the virtual pins through the monitor. The lane and the ball are also tracked by vision-based tracking. In those applications, we utilize multiple 2D markers distributed at arbitrary positions and directions. Even though the geometrical relationship among the markers is unknown, we can track the camera in very wide area.

Computer Vision-Based Augmented Reality for Guiding Manual Assembly

Augmented reality (AR) has the goal of enhancing a person's perception of the surrounding world, unlike virtual reality (VR) that aims at replacing the perception of the world with an artificial one. An important issue in AR is making the virtual world sensitive to the current state of the surrounding real world as the user interacts with it. For providing the appropriate augmentation stimulus at the right position and time, the system needs some sensor to interpret the surrounding scene. Computer vision holds great potential in providing the necessary interpretation of the scene. While a computer vision-based general interpretation of a scene is extremely difficult, the constraints from the assembly domain and specific marker-based coding scheme are used to develop an efficient and practical solution. We consider the problem of scene augmentation in the context of a human engaged in assembling a mechanical object from its components. Concepts from robot assembly planning are used to develop a systematic framework for presenting augmentation stimuli for this assembly domain. An experimental prototype system, VEGAS (Visual Enhancement for Guiding Assembly Sequences), is described, that implements some of the AR concepts for guiding assembly using computer vision.