Markerless Augmented Reality System Research Articles

Novel method for augmented reality guided endodontics: An in vitro study

ObjectiveThe aim of this study is to evaluate the accuracy in endodontics of a novel augmented reality (AR) method for guided access cavity preparation in 3D-printed jaws. MethodsTwo operators with different levels of experience in endodontics performed pre-planned virtually guided access cavities through a novel markerless AR system on three sets of 3D-printed jaw models (Objet Connex 350, Stratasys) mounted on a phantom. After the treatment, a post-operative high-resolution CBCT scan (NewTom VGI Evo, Cefla) was taken for each model and registered to the pre-operative model. All the access cavities were then digitally reconstructed by filling the cavity area using 3D medical software (3-Matic 15.0, materialize). For the anterior teeth and the premolars, the deviation at the coronal and apical entry points as well as the angular deviation of the access cavity were compared to the virtual plan. For the molars, the deviation at the coronal entry point was compared to the virtual plan. Additionally, the surface area of all access cavities at the entry point was measured and compared to the virtual plan. Descriptive statistics for each parameter were performed. A 95% confidence interval was calculated. ResultsA total of 90 access cavities were drilled up to a depth of 4 mm inside the tooth. The mean deviation in the frontal teeth and in the premolars at the entry point was 0.51 mm and 0.77 mm at the apical point, with a mean angular deviation of 8.5° and a mean surface overlap of 57%. The mean deviation for the molars at the entry point was 0.63 mm, with a mean surface overlap of 82%. ConclusionThe use of AR as a digital guide for endodontic access cavity drilling on different teeth showed promising results and might have potential for clinical use. However, further development and research might be needed before in vivo validation.

Visual SLAM Based Spatial Recognition and Visualization Method for Mobile AR Systems

The simultaneous localization and mapping (SLAM) market is growing rapidly with advances in Machine Learning, Drones, and Augmented Reality (AR) technologies. However, due to the absence of an open source-based SLAM library for developing AR content, most SLAM researchers are required to conduct their own research and development to customize SLAM. In this paper, we propose an open source-based Mobile Markerless AR System by building our own pipeline based on Visual SLAM. To implement the Mobile AR System of this paper, we use ORB-SLAM3 and Unity Engine and experiment with running our system in a real environment and confirming it in the Unity Engine’s Mobile Viewer. Through this experimentation, we can verify that the Unity Engine and the SLAM System are tightly integrated and communicate smoothly. In addition, we expect to accelerate the growth of SLAM technology through this research.

Marker-less augmented reality based on monocular vision for falx meningioma localization.

The existing augmented reality (AR) based neuronavigation systems typically require markers and additional tracking devices for model registration, which causes excessive preparatory steps. For fast and accurate intraoperative navigation, this work proposes a marker-less AR system that tracks the head features with the monocular camera. After the semi-automatic initialization process, the feature points between the captured image and the pre-loaded keyframes are matched for obtaining correspondences. The camera pose is estimated by solving the Perspective-n-Point problem. The localization error of AR visualization on scalp and falx meningioma is 0.417±0.057 and 1.413±0.282mm, respectively. The maximum localization error is less than 2mm. The AR system is robust to occlusions and changes in viewpoint and scale. We demonstrate that the developed system can successfully display the augmented falx meningioma with enough accuracy and provide guidance for neurosurgeons to locate the tumour in brain.

A proof-of-concept augmented reality system in oral and maxillofacial surgery

BackgroundThe advent of digital medical imaging, medical image analysis and computer vision has opened the surgeon horizons with the possibility to add virtual information to the real operative field. For oral and maxillofacial surgeons, overlaying anatomical structures to protect (such as teeth, sinus floors, inferior and superior alveolar nerves) or to remove (such as cysts, tumours, impacted teeth) presents a real clinical interest. Material and methodsThrough this work, we propose a proof-of-concept markerless augmented reality system for oral and maxillofacial surgery, where a virtual scene is generated preoperatively and mixed with reality to reveal the location of hidden anatomical structures intraoperatively. We devised a computer software to process still video frames of the operating field and to display them on the operating room screens. ResultsFirstly, we give a description of the proposed system, where virtuality aligns with reality without artificial markers. The dental occlusion plan analysis and cusps detection allow us to initialise the alignment process. Secondly, we validate the feasibility with an experimental approach on a 3D printed jaw phantom and an ex-vivo pig jaw. Thirdly, we evaluate the potential clinical benefit on a patient. Conclusionthis proof-of-concept highlights the feasibility and the interest of augmented reality for hidden anatomical structures visualisation without artificial markers.

Optimized registration based on an ant colony for markerless augmented reality systems

Accurate registration in augmented reality systems is essential to guarantee the visual consistency of the augmented environment. Although error in the virtual-real alignment is almost unavoidable, different approaches have been proposed to quantify and reduce such errors. However, many of the existing solutions require a lot of a priori information, or they only focus on camera calibration to guarantee good results in the registration. This article presents a heuristic method that aims to reduce registration errors in markerless augmented reality systems. The proposed solution sees error reduction as a mono-objective optimization problem, which is addressed by means of the Ant Colony Optimization (ACO) algorithm. Experimental results reveal the validity of the proposed method, reaching an average error of 1.49 pixels for long video sequences.

A Task-level Pipelined Many-SIMD Augmented Reality Processor with Congestion-aware Network-on-Chip Scheduler

For all day long operable markerless augmented reality system, low-power BONE-AR processor is implemented to execute object recognition, camera pose estimation, and 3D graphics rendering in real-time for a HD resolution video input. BONE-AR adopts 6 clusters of heterogeneous SIMD processors distributed on the mesh topology network-on-chip (NoC) to exploit data-level parallelism and task-level parallelism. Visual attention algorithm reduces overall workload by removing background clutters from the input video frames, but also incurs NoC congestion due to dynamically fluctuating workload. We propose a congestion-aware scheduler (CAS) that detects and resolves the NoC congestion to prevent throughput degradation of task-level pipeline.

An Augmented Reality Processor with a Congestion-Aware Network-on-Chip Scheduler

For a markerless augmented reality system that can operate all day, the authors implemented a low-power Basic On-Chip Network-Augmented Reality (BONE-AR) processor to execute object recognition, camera pose estimation, and 3D graphics rendering in real time for an HD resolution video input. BONE-AR employs six clusters of heterogeneous SIMD processors distributed on the mesh topology network on a chip (NoC) to exploit data- and task-level parallelism. A visual attention algorithm reduces overall workload by removing background clutters from the input video frames, but also incurs NoC congestion because of a dynamically fluctuating workload. The authors propose a congestion-aware scheduler that detects and resolves the NoC congestion to prevent throughput degradation of a task-level pipeline.

Implementation of Augmented Reality System for Smartphone Advertisements

The recent development and popularization of personal mobile devices has brought a lot of changes into information search and offering. In particular, the newly emerged augmented reality system made possible an innovative way of information acquisition by providing additional information of the virtual world to the real world. This work used the markerless augmented reality system on smartphones to design and implement the smartphone application service aimed at efficiently conveying information on advertisements to users. The conventional advertising applications simply introduce and explain goods by inducing consumers to have an interest, whereas the markerless application service developed in this work interacts with the augmented reality system and database management system to quickly provide more accurate and diversified information to users.

Mobile Camera as a Human Vision in Augmented Reality

The real world objects can be recognized by using marker based and marker-less augmented reality systems. Mostly, the previous developers used markers based augmented reality systems. However, those systems actually hide the reality and it was also difficult to keep the markers everywhere. Furthermore, the previous marker-less approaches use client-server architecture, which is drastically affected by network latency. Smartphone camera is matured enough that it can recognize real world objects without markers. It can guide users about their location and the direction in a convenient way. The use of Smartphone is best suited for outdoor mobile augmented-reality applications. Therefore, a marker-less natural features based tracking system in mobile augmented reality was formulated. In the adapted framework, the state-of-the-art algorithm (speed up robust features) was modified for computing image features from live mobile camera image and compares with locally stored images features for recognition. Moreover, the local static database of location tagged image features using SQLite was implemented to bypass the server. The proposed system was tested in a mobile AR-prototype application using iPhone called UNIMAS Guide. It was found from the results that the adapted marker-less system could recognize the real world objects in speedy, easy and convenient way. This technology can be applied in tourism industry, surgery and educational fields. DOI: http://dx.doi.org/10.11591/telkomnika.v11i7.2798 Full Text: PDF

Hand Recognition for Augmented Reality

In recent years, Augmented Reality (AR) becomes the focus of attention as a technology for obtaining some information. Most AR systems have used some markers to display any information. However, since markerless AR systems can be used intuitively, they are researched actively.In this paper, we study about the AR system based on the hand recognition as a markerless system. The reason why we used a hand is that we don't need to prepare or carry on any tool, and can easy to watch any information on a hand. Taehee Lee et al. introduced HandyAR which employ the hand recognition. In this method, there are some problems. At first, when the method learns skin model, it needs many learning data. This is because it uses RGB color model. Secondly, it is difficult to get hand because hand images which are extracted by the skin model have noise. Thirdly, the method uses finger positions to estimate the hand coordinate. Therefore it is sensitive to finger state. We improved HandyAR to solve these problems. Experimental result showed that our method had higher performance than conventional methods.

Dynamic Manipulation of a Virtual Object in Marker-less AR system Based on Both Human Hands

This paper presents a novel approach to control the augmented reality (AR) objects robustly in a marker-less AR system by fingertip tracking and hand pattern recognition. It is known that one of the promising ways to develop a marker-less AR system is using human’s body such as hand or face for replacing traditional fiducial markers. This paper introduces a real-time method to manipulate the overlaid virtual objects dynamically in a marker-less AR system using both hands with a single camera. The left bare hand is considered as a virtual marker in the marker-less AR system and the right hand is used as a hand mouse. To build the marker-less system, we utilize a skin-color model for hand shape detection and curvature-based fingertip detection from an input video image. Using the detected fingertips the camera pose are estimated to overlay virtual objects on the hand coordinate system. In order to manipulate the virtual objects rendered on the marker-less AR system dynamically, a vision-based hand control interface, which exploits the fingertip tracking for the movement of the objects and pattern matching for the hand command initiation, is developed. From the experiments, we can prove that the proposed and developed system can control the objects dynamically in a convenient fashion.

Marker-less registration based on template tracking for augmented reality

Accurate 3D registration is a key issue in the Augmented Reality (AR) applications, particularly where are no markers placed manually. In this paper, an efficient markerless registration algorithm is presented for both outdoor and indoor AR system. This algorithm first calculates the correspondences among frames using fixed region tracking, and then estimates the motion parameters on projective transformation following the homography of the tracked region. To achieve the illumination insensitive tracking, the illumination parameters are solved jointly with motion parameters in each step. Based on the perspective motion parameters of the tracked region, the 3D registration, the camera's pose and position, can be calculated with calibrated intrinsic parameters. A marker-less AR system is described using this algorithm, and the system architecture and working flow are also proposed. Experimental results with comparison quantitatively demonstrate the correctness of the theoretical analysis and the robustness of the registration algorithm.