3D Surface Reconstruction Research Articles

Novel Method for Virtual Restoration of Cultural Relics with Complex Geometric Structure Based on Multiscale Spatial Geometry

Because of the age of relics and the lack of historical data, the geometric forms of missing parts can only be judged by the subjective experience of repair personnel, which leads to varying restoration effects when the geometric structure of the complex relic is reconstructed. Therefore, virtual repair effects cannot fully reflect the historical appearance of cultural relics. In order to solve this problem, this paper presents a virtual restoration method based on the multiscale spatial geometric features of cultural relics in the case of complex construction where the geometric shape of the damaged area is unknown, using the Dazu Thousand-Hand Bodhisattva statue in China as an example. In this study, the global geometric features of the three-dimensional (3D) model are analyzed in space to determine the geometric shape of the damaged parts of cultural relics. The local geometric features are represented by skeleton lines based on regression analysis, and a geometric size prediction model of the defective parts is established, which is used to calculate the geometric dimensions of the missing parts. Finally, 3D surface reconstruction technology is used to quantitate virtual restoration of the defective parts. This method not only provides a new idea for the virtual restoration of artifacts with complex geometric structure, but also may play a vital role in the protection of cultural relics.

Advanced 3D Photogrammetric Surface Reconstruction of Extensive Objects by UAV Camera Image Acquisition.

This paper proposes a replicable methodology to enhance the accuracy of the photogrammetric reconstruction of large-scale objects based on the optimization of the procedures for Unmanned Aerial Vehicle (UAV) camera image acquisition. The relationships between the acquisition grid shapes, the acquisition grid geometric parameters (pitches, image rates, camera framing, flight heights), and the 3D photogrammetric surface reconstruction accuracy were studied. Ground Sampling Distance (GSD), the necessary number of photos to assure the desired overlapping, and the surface reconstruction accuracy were related to grid shapes, image rate, and camera framing at different flight heights. The established relationships allow to choose the best combination of grid shapes and acquisition grid geometric parameters to obtain the desired accuracy for the required GSD. This outcome was assessed by means of a case study related to the ancient arched brick Bridge of the Saracens in Adrano (Sicily, Italy). The reconstruction of the three-dimensional surfaces of this structure, obtained by the efficient Structure-From-Motion (SfM) algorithms of the commercial software Pix4Mapper, supported the study by validating it with experimental data. A comparison between the surface reconstruction with different acquisition grids at different flight heights and the measurements obtained with a 3D terrestrial laser and total station-theodolites allowed to evaluate the accuracy in terms of Euclidean distances.

Real-time intra-fraction motion management in breast cancer radiotherapy: analysis of 2028 treatment sessions

BackgroundIntra-fraction motion represents a crucial issue in the era of precise radiotherapy in several settings, including breast irradiation. To date, only few data exist on real-time measured intra-fraction motion in breast cancer patients. Continuous surface imaging using visible light offers the capability to monitor patient movements in three-dimensional space without any additional radiation exposure. The aim of the present study was to quantify the uncertainties of possible intra-fractional motion during breast radiotherapy.Material and methodsOne hundred and four consecutive patients that underwent postoperative radiotherapy following breast conserving surgery or mastectomy were prospectively evaluated during 2028 treatment sessions. During each treatment session the patients’ motion was continuously measured using the Catalyst™ optical surface scanner (C-RAD AB, Sweden) and compared to a reference scan acquired at the beginning of each session. The Catalyst system works through an optical surface imaging with light emitting diode (LED) light and reprojection captured by a charge coupled device (CCD) camera, which provide target position control during treatment delivery with a motion detection accuracy of 0.5 mm. For 3D surface reconstruction, the system uses a non-rigid body algorithm to calculate the distance between the surface and the isocentre and using the principle of optical triangulation. Three-dimensional deviations and relative position differences during the whole treatment fraction were calculated by the system and analyzed statistically.ResultsOverall, the maximum magnitude of the deviation vector showed a mean change of 1.93 mm ± 1.14 mm (standard deviation [SD]) (95%-confidence interval: [0.48–4.65] mm) and a median change of 1.63 mm during dose application (beam-on time only). Along the lateral and longitudinal axis changes were quite similar (0.18 mm ± 1.06 mm vs. 0.17 mm ± 1.32 mm), on the vertical axis the mean change was 0.68 mm ± 1.53 mm. The mean treatment session time was 154 ± 53 (SD) seconds and the mean beam-on time only was 55 ± 16 s. According to Friedman’s test differences in the distributions of the three possible directions (lateral, longitudinal and vertical) were significant (p < 0.01), in post-hoc analysis there were no similarities between any two of the three directions.ConclusionThe optical surface imaging system is an accurate and easy tool for real-time motion management in breast cancer radiotherapy. Intra-fraction motion was reported within five millimeters in all directions. Thus, intra-fraction motion in our series of 2028 treatment sessions seems to be of minor clinical relevance in postoperative radiotherapy of breast cancer.

The Human Cochlear Aqueduct and Accessory Canals: a Micro-CT Analysis Using a 3D Reconstruction Paradigm.

We sought to study the anatomic variations of the cochlear aqueduct and its accessory canals in human temporal bones using micro-CT and a 3D reconstruction paradigm. More knowledge about the anatomic variations of these structures, particularly at the basal turn of the cochlea and round window niche, may be important to better preserve residual hearing as well as the neural supply during cochlear implant surgery. An archival collection of 30 human temporal bones underwent micro-CT and 3D reconstruction. A surface enhancement paradigm was applied. The application displays reconstructed slices as a 3D object with realistic 3D visualization of scanned objects. Virtual sectioning or "cropping" of the petrous bone presented subsequent areas. Thereby, the bony canals could be followed from inside the basal turn of cochlea and middle ear to the jugular foramen. The cochlear aqueduct was always paralleled by an accessory canal containing the inferior cochlear vein. It ran from the basal turn of the cochlea and exited laterally in the jugular foramen. In 70% of the cases, a secondary accessory canal was observed and it derived mostly from a depression or infundibulum located in the floor of the round window niche. This canal also exited in the jugular foramen. The secondary accessory canal occasionally anastomosed with the primary accessory canal suggesting that it contains a vein that drains middle ear blood to the cranial sinus. Micro-CT with 3D surface reconstruction paradigm offers new possibilities to study the topographic anatomy of minor details in the human inner ear. The technique creates simulated transparent "castings" of the labyrinth with a coinciding surface view through enhancement of contrast between boundaries. Accessory canals that drain blood from the cochlea, spiral ganglion, and middle ear could be characterized three-dimensionally.

Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks.

Surgical guidance and decision making could be improved with accurate and real-time measurement of intra-operative data including shape and spectral information of the tissue surface. In this work, a dual-modality endoscopic system has been proposed to enable tissue surface shape reconstruction and hyperspectral imaging (HSI). This system centers around a probe comprised of an incoherent fiber bundle, whose fiber arrangement is different at the two ends, and miniature imaging optics. For 3D reconstruction with structured light (SL), a light pattern formed of randomly distributed spots with different colors is projected onto the tissue surface, creating artificial texture. Pattern decoding with a Convolutional Neural Network (CNN) model and a customized feature descriptor enables real-time 3D surface reconstruction at approximately 12 frames per second (FPS). In HSI mode, spatially sparse hyperspectral signals from the tissue surface can be captured with a slit hyperspectral imager in a single snapshot. A CNN based super-resolution model, namely "super-spectral-resolution" network (SSRNet), has also been developed to estimate pixel-level dense hypercubes from the endoscope cameras standard RGB images and the sparse hyperspectral signals, at approximately 2 FPS. The probe, with a 2.1 mm diameter, enables the system to be used with endoscope working channels. Furthermore, since data acquisition in both modes can be accomplished in one snapshot, operation of this system in clinical applications is minimally affected by tissue surface movement and deformation. The whole apparatus has been validated on phantoms and tissue (ex vivo and in vivo), while initial measurements on patients during laryngeal surgery show its potential in real-world clinical applications.

POINT CLOUD AND DIGITAL SURFACE MODEL GENERATION FROM HIGH RESOLUTION MULTIPLE VIEW STEREO SATELLITE IMAGERY

Abstract. Nowadays, multiple-view stereo satellite imagery has become a valuable data source for digital surface model generation and 3D reconstruction. In 2016, a well-organized multiple view stereo publicly benchmark for commercial satellite imagery has been released by the John Hopkins University Applied Physics Laboratory, USA. This benchmark motivates us to explore the method that can generate accurate digital surface models from a large number of high resolution satellite images. In this paper, we propose a pipeline for processing the benchmark data to digital surface models. As a pre-procedure, we filter all the possible image pairs according to the incidence angle and capture date. With the selected image pairs, the relative bias-compensated model is applied for relative orientation. After the epipolar image pairs’ generation, dense image matching and triangulation, the 3D point clouds and DSMs are acquired. The DSMs are aligned to a quasi-ground plane by the relative bias-compensated model. We apply the median filter to generate the fused point cloud and DSM. By comparing with the reference LiDAR DSM, the accuracy, the completeness and the robustness are evaluated. The results show, that the point cloud reconstructs the surface with small structures and the fused DSM generated by our pipeline is accurate and robust.

LARGE SCALE TEXTURED MESH RECONSTRUCTION FROM MOBILE MAPPING IMAGES AND LIDAR SCANS

Abstract. The representation of 3D geometric and photometric information of the real world is one of the most challenging and extensively studied research topics in the photogrammetry and robotics communities. In this paper, we present a fully automatic framework for 3D high quality large scale urban texture mapping using oriented images and LiDAR scans acquired by a terrestrial Mobile Mapping System (MMS). First, the acquired points and images are sliced into temporal chunks ensuring a reasonable size and time consistency between geometry (points) and photometry (images). Then, a simple, fast and scalable 3D surface reconstruction relying on the sensor space topology is performed on each chunk after an isotropic sampling of the point cloud obtained from the raw LiDAR scans. Finally, the algorithm proposed in (Waechter et al., 2014) is adapted to texture the reconstructed surface with the images acquired simultaneously, ensuring a high quality texture with no seams and global color adjustment. We evaluate our full pipeline on a dataset of 17 km of acquisition in Rouen, France resulting in nearly 2 billion points and 40000 full HD images. We are able to reconstruct and texture the whole acquisition in less than 30 computing hours, the entire process being highly parallel as each chunk can be processed independently in a separate thread or computer.

Template-Based 3D Reconstruction of Non-rigid Deformable Object from Monocular Video

In this paper, we propose a template-based 3D surface reconstruction system of non-rigid deformable objects from monocular video sequence. Firstly, we generate a semi-dense template of the target object with structure from motion method using a subsequence video. This video can be captured by rigid moving camera orienting the static target object or by a static camera observing the rigid moving target object. Then, with the reference template mesh as input and based on the framework of classical template-based methods, we solve an energy minimization problem to get the correspondence between the template and every frame to get the time-varying mesh to present the deformation of objects. The energy terms combine photometric cost, temporal and spatial smoothness cost as well as as-rigid-as-possible cost which can enable elastic deformation. In this paper, an easy and controllable solution to generate the semi-dense template for complex objects is presented. Besides, we use an effective iterative Schur based linear solver for the energy minimization problem. The experimental evaluation presents qualitative deformation objects reconstruction results with real sequences. Compare against the results with other templates as input, the reconstructions based on our template have more accurate and detailed results for certain regions. The experimental results show that the linear solver we used performs better efficiency compared to traditional conjugate gradient based solver.

A New Oren–Nayar Shape-from-Shading Approach for 3D Reconstruction Using High-Order Godunov-Based Scheme

3D shape reconstruction from images has been an important topic in the field of robot vision. Shape-From-Shading (SFS) is a classical method for determining the shape of a 3D surface from a one intensity image. The Lambertian reflectance is a fundamental assumption in conventional SFS approaches. Unfortunately, when applied to characterize the reflection attribute of the diffuse reflection, the Lambertian model is tested to be inexact. In this paper, we present a new SFS approach for 3D reconstruction of diffuse surfaces whose reflection attribute is approximated by the Oren–Nayar reflection model. The partial differential Image Irradiance Equation (IIR) is set up with this model under a single distant point light source and an orthographic camera projection whose direction coincides with the light source. Then, the IIR is converted into an eikonal equation by solving a quadratic equation that includes the 3D surface shape. The viscosity solution of the resulting eikonal equation is approximated by using the high-order Godunov-based scheme that is accelerated by means of an alternating sweeping strategy. We conduct the experiments on synthetic and real-world images, and the experimental results illustrate the effectiveness of the presented approach.

Reconstruction of 3D curves and surfaces using new variants of gravitational search algorithm

The problem of reconstruction of 3D curves and surfaces is modelled as a nonlinear optimization problem in which the objective function to be minimized is the error function between given data points and data points on the generated curve/surface.Although any optimization algorithm can be used to solve this problem, but due to the inherent complexities in the optimization model, heuristic algorithms are best suited. In this paper, a recently proposed nature motivated optimization technique, namely Gravitational Search Algorithm and its three variants are used to solve reconstruction problem. The approach is illustrated with the help of two examples- a helix and a dumbbell. The results are demonstrated and it is concluded that the approach is very promising in the area.

3D skin surface reconstruction from a single image by merging global curvature and local texture using the guided filtering for 3D haptic palpation.

Skin surface evaluation has been studied using various imaging techniques. However, all these studies had limited impact because they were performed using visual exam only. To improve on this scenario with haptic feedback, we propose 3D reconstruction of the skin surface using a single image. Unlike extant 3D skin surface reconstruction algorithms, we utilize the local texture and global curvature regions, combining the results for reconstruction. The first entails the reconstruction of global curvature, achieved by bilateral filtering that removes noise on the surface while maintaining the edge (ie, furrow) to obtain the overall curvature. The second entails the reconstruction of local texture, representing the fine wrinkles of the skin, using an advanced form of bilateral filtering. The final image is then composed by merging the two reconstructed images. We tested the curvature reconstruction part by comparing the resulting curvatures with measured values from real phantom objects while local texture reconstruction was verified by measuring skin surface roughness. Then, we showed the reconstructed result of our proposed algorithm via the reconstruction of various real skin surfaces. The experimental results demonstrate that our approach is a promising technology to reconstruct an accurate skin surface with a single skin image. We proposed 3D skin surface reconstruction using only a single camera. We highlighted the utility of global curvature, which has not been considered important in the past. Thus, we proposed a new method for 3D reconstruction that can be used for 3D haptic palpation, dividing the concepts of local and global regions.

Improvement of stereo matching algorithm for 3D surface reconstruction

The stereo matching algorithm is one of the important methods for 3D surface reconstruction. A stereo matching process produces a disparity map which provides the depth of information required in 3D reconstruction. This map consists of disparity values of two corresponding points. Furthermore, the accuracy of 3D reconstruction depends on how precise the disparity being estimated on each pixel location. To get a good 3D reconstruction result, the propose stereo matching algorithm must be strong against the radiometric differences and edge distortions. Hence, this article proposes a new stereo matching algorithm with high accuracy for 3D surface reconstruction. First stage, Sum of Gradient Matching (SG) is proposed which uses magnitude differences with fixed window size. The gradient matching is strong against the radiometric distortions due to different characteristics of the input stereo cameras. Second stage, the Adaptive Support Weight (ASW) with iterative Guided Filter (ASW iGF) is proposed to improve the edges of object matching. The last stage, Joint Weighted Guided Filter (JWGF) is suggested to reduce the remaining noise on the disparity map. Based on the standard quantitative benchmarking stereo dataset, the proposed work in this article produces good results and performs much better compared with before the proposed framework. This new algorithm is also competitive with some established methods in the literature.

Axial plane dissimilarities of two identical Lenke-type 6C scoliosis cases visualized and analyzed by vertebral vectors

The global appearance of scoliosis in the horizontal plane is not really known. Therefore, the aims of this study were to analyze scoliosis in the horizontal plane using vertebral vectors in two patients classified with the same Lenke group, and to highlight the importance of the information obtained from these vertebral vector-based top-view images in clinical practice. Two identical cases of scoliosis were selected, based on preoperative full-body standing anteroposterior and lateral radiographs obtained by the EOS™ 2D/3D system. Three-dimensional (3D) surface reconstructions of the spinal curves were performed by using sterEOS™ 3D software before and after surgery. In both patients, we also determined the vertebral vectors and horizontal plane coordinates for analyzing the curves mathematically before and after surgery. Despite the identical appearance of spinal curves in the frontal and sagittal planes, the horizontal views seemed to be significantly different. The vertebral vectors in the horizontal plane provided different types of parameters regarding scoliosis and the impact of surgical treatment: reducing lateral deviations, achieving harmony of the curves in the sagittal plane, and reducing rotations in the horizontal plane. Vertebral vectors allow the evolution of scoliosis curve projections in the horizontal plane before and after surgical treatment, along with representation of the entire spine. The top view in the horizontal plane is essential to completely evaluate the scoliosis curves, because, despite the similar representations in the frontal and sagittal planes, the occurrence of scoliosis in the horizontal plane can be completely different. These slides can be retrieved under Electronic Supplementary Material.

3D sensing of back symmetry curve suited for dynamic analysis of spinal deformities

ABSTRACTThis paper is focused on meeting the demands for better detection and evaluation of the spinal deformities that go beyond a basic static 3D analysis. It is expected that by observing the spine while it moves an improved 3D deformity analysis and measurements can be achieved. In this paper, we present a novel approach for the automatic back symmetry curve sensing. Our approach can be used in the back shape analysis of many different positions, i.e. segments of the dynamic motion, and is not limited only to the upright standing position. The proposed method is based on the 3D surface reconstruction, surface curvature analysis and graph theoretic approach for the (semi-)automatic detection of the symmetry curve. In addition, we introduce a 3D scanning system which was used in our experiment to successfully generate 3D back surface reconstructions for each frame of the captured forward bending motion. We also tested the proposed method on the data collected using commercial 3D spine analysis system and the results were comparable. The additional experiment focusing on the dynamic analysis demonstrated that the proposed method can enable further advances in the automatic 3D back surface analysis by tracking the spine position throughout performed movements.

A New Approach for Photorealistic Visualization of Rendered Computed TomographyImages.

Classical single-colored or multicolored 3-dimensional (3D) visualization of sectional images lacked in being realistic and revealed limited anatomical discrimination. Recently, a new technique called cinematic volume rendering for 3D reconstruction of computed tomography has been developed. The aim of this study was to analyze this new visualization algorithm from a technical perspective and to investigate potential benefits for neurosurgical applications. A standard test in computer graphics called Cornell Box was adapted and applied for reproducibility of light effects in cinematic rendering opposed to classic rendering methods. Simulation of distinct camera effects such as variable apertures, exposition time, optics, and surface refinements are presented in a human skull, respectively. Postprocessing capabilities allow for immediate clinical use. This volume-rendering technique generates cadaver-like 3D reconstructions. By considering complex interactions between a scanned object and dynamic light patterns, a cinematic illumination of a 3D surface reconstruction can be achieved. A spinal tumor case and a complex intracranial carotid artery aneurysm are presented, comparing all available rendering techniques. Cinematic rendering results in greater spatial discrimination of neighboring anatomical structures. This technical and clinical description focuses on the neurosurgical relevance of a new rendering technique. Considering the improved image impression of cinematic rendering and viewers' perception, it seems likely that the technique will gain wide acceptance in the clinical routine.

Computer-assisted 3D bowel length measurement for quantitative laparoscopy.

This study aimed at developing and evaluating a tool for computer-assisted 3D bowel length measurement (BMS) to improve objective measurement in minimally invasive surgery. Standardization and quality of surgery as well as its documentation are currently limited by lack of objective intraoperative measurements. To solve this problem, we developed BMS as a clinical application of Quantitative Laparoscopy (QL). BMS processes images from a conventional 3D laparoscope. Computer vision algorithms are used to measure the distance between laparoscopic instruments along a 3D reconstruction of the bowel surface. Preclinical evaluation was performed in phantom, ex vivo porcine, and in vivo porcine models. A bowel length of 70cm was measured with BMS and compared to a manually obtained ground truth. Afterwards 70cm of bowel (ground truth) was measured and compared to BMS. Ground truth was 66.1 ± 2.7cm (relative error + 5.8%) in phantom, 65.8 ± 2.5cm (relative error + 6.4%) in ex vivo, and 67.5 ± 6.6cm (relative error + 3.7%) in in vivo porcine evaluation when 70cm was measured with BMS. Using 70cm of bowel, BMS measured 75.0 ± 2.9cm (relative error + 7.2%) in phantom and 74.4 ± 2.8cm (relative error + 6.3%) in ex vivo porcine evaluation. After thorough preclinical evaluation, BMS was successfully used in a patient undergoing laparoscopic Roux-en-Y gastric bypass for morbid obesity. QL using BMS was shown to be feasible and was successfully translated from studies on phantom, ex vivo, and in vivo porcine bowel to a clinical feasibility study.

Integrated Image Matching and Segmentation for 3D Surface Reconstruction in Urban Areas

Integrated Image Matching and Segmentation for 3D Surface Reconstruction in Urban Areas

Assessing the Impact of Color Adjustments on Photogrammetry-Based 3D Surface Reconstruction

International Journal of Computer Sciences and Engineering (A UGC Approved and indexed with DOI, ICI and Approved, DPI Digital Library) is one of the leading and growing open access, peer-reviewed, monthly, and scientific research journal for scientists, engineers, research scholars, and academicians, which gains a foothold in Asia and opens to the world, aims to publish original, theoretical and practical advances in Computer Science,Information Technology, Engineering (Software, Mechanical, Civil, Electronics & Electrical), and all interdisciplinary streams of Computing Sciences. It intends to disseminate original, scientific, theoretical or applied research in the field of Computer Sciences and allied fields. It provides a platform for publishing results and research with a strong empirical component. It aims to bridge the significant gap between research and practice by promoting the publication of original, novel, industry-relevant research.

Virtual reality in advanced medical immersive imaging: a workflow for introducing virtual reality as a supporting tool in medical imaging

Radiologic evaluation of images from computed tomography (CT) or magnetic resonance imaging for diagnostic purposes is based on the analysis of single slices, occasionally supplementing this information with 3D reconstructions as well as surface or volume rendered images. However, due to the complexity of anatomical or pathological structures in biomedical imaging, innovative visualization techniques are required to display morphological characteristics three dimensionally. Virtual reality is a modern tool of representing visual data, The observer has the impression of being “inside” a virtual surrounding, which is referred to as immersive imaging. Such techniques are currently being used in technical applications, e.g. in the automobile industry. Our aim is to introduce a workflow realized within one simple program which processes common image stacks from CT, produces 3D volume and surface reconstruction and rendering, and finally includes the data into a virtual reality device equipped with a motion head tracking cave automatic virtual environment system. Such techniques have the potential to augment the possibilities in non-invasive medical imaging, e.g. for surgical planning or educational purposes to add another dimension for advanced understanding of complex anatomical and pathological structures. To this end, the reconstructions are based on advanced mathematical techniques and the corresponding grids which we can export are intended to form the basis for simulations of mathematical models of the pathogenesis of different diseases.

SLAM-based dense surface reconstruction in monocular Minimally Invasive Surgery and its application to Augmented Reality

Background and objective: While Minimally Invasive Surgery (MIS) offers considerable benefits to patients, it also imposes big challenges on a surgeon’s performance due to well-known issues and restrictions associated with the field of view (FOV), hand-eye misalignment and disorientation, as well as the lack of stereoscopic depth perception in monocular endoscopy. Augmented Reality (AR) technology can help to overcome these limitations by augmenting the real scene with annotations, labels, tumour measurements or even a 3D reconstruction of anatomy structures at the target surgical locations. However, previous research attempts of using AR technology in monocular MIS surgical scenes have been mainly focused on the information overlay without addressing correct spatial calibrations, which could lead to incorrect localization of annotations and labels, and inaccurate depth cues and tumour measurements. In this paper, we present a novel intra-operative dense surface reconstruction framework that is capable of providing geometry information from only monocular MIS videos for geometry-aware AR applications such as site measurements and depth cues. We address a number of compelling issues in augmenting a scene for a monocular MIS environment, such as drifting and inaccurate planar mapping.Methods: A state-of-the-art Simultaneous Localization And Mapping (SLAM) algorithm used in robotics has been extended to deal with monocular MIS surgical scenes for reliable endoscopic camera tracking and salient point mapping. A robust global 3D surface reconstruction framework has been developed for building a dense surface using only unorganized sparse point clouds extracted from the SLAM. The 3D surface reconstruction framework employs the Moving Least Squares (MLS) smoothing algorithm and the Poisson surface reconstruction framework for real time processing of the point clouds data set. Finally, the 3D geometric information of the surgical scene allows better understanding and accurate placement AR augmentations based on a robust 3D calibration.Results: We demonstrate the clinical relevance of our proposed system through two examples: (a) measurement of the surface; (b) depth cues in monocular endoscopy. The performance and accuracy evaluations of the proposed framework consist of two steps. First, we have created a computer-generated endoscopy simulation video to quantify the accuracy of the camera tracking by comparing the results of the video camera tracking with the recorded ground-truth camera trajectories. The accuracy of the surface reconstruction is assessed by evaluating the Root Mean Square Distance (RMSD) of surface vertices of the reconstructed mesh with that of the ground truth 3D models. An error of 1.24 mm for the camera trajectories has been obtained and the RMSD for surface reconstruction is 2.54 mm, which compare favourably with previous approaches. Second, in vivo laparoscopic videos are used to examine the quality of accurate AR based annotation and measurement, and the creation of depth cues. These results show the potential promise of our geometry-aware AR technology to be used in MIS surgical scenes.Conclusions: The results show that the new framework is robust and accurate in dealing with challenging situations such as the rapid endoscopy camera movements in monocular MIS scenes. Both camera tracking and surface reconstruction based on a sparse point cloud are effective and operated in real-time. This demonstrates the potential of our algorithm for accurate AR localization and depth augmentation with geometric cues and correct surface measurements in MIS with monocular endoscopes.