2D-3D Matching Research Articles

Absolute Pose Estimation With a Known Direction by Motion Decoupling

This paper develops an extremely robust solution for absolute pose estimation with known prior gravity direction by motion decoupling. Absolute pose estimation is a fundamental problem in computer vision, and recently the prior known vertical direction is commonly applied to help solve the pose estimation problem. In this paper, we explore the geometrical constraints of the absolute pose estimation with a known direction. We find that the rigid pose can be decoupled with the help of the known direction. Thereby, absolute pose estimation algorithms, which decouple rigid motion, are proposed. Notably, in real applications, there may be imperfect inputs, i.e., outliers, due to incorrect 2D-3D matches. Unfortunately, these outliers may lead to unacceptable results. To suppress the outliers, the decoupled absolute pose estimation problem is solved by branch-and-bound algorithm and globally voting, which can provide the optimal solution with provable guarantees. Moreover, in extreme case, the proposed method can solve absolute pose estimation problem without knowing the 2D-3D correspondences, which is also known as simultaneous camera pose correspondence estimation. To demonstrate the feasibility and the superiority of the proposed methods, comprehensive comparison experiment are conduced. The source code is available at https://github.com/Liu-Yinlong/algorithm-for-PnP-with-known-vertical-direction.

Cross domain 2D-3D descriptor matching for unconstrained 6-DOF pose estimation

This paper presents a novel approach for cross-domain descriptor matching between 2D and 3D modalities. The 2D-3D matching is applied to localize 2D images in 3D point clouds. Direct cross-domain matching allows our technique to localize images in any type of 3D point cloud without any constraints on the nature or mechanism by which it is obtained. We propose a learning based framework, called Desc-Matcher, to directly match features between the two modalities. A dataset of 2D and 3D features with corresponding locations in images and point clouds is generated to train the Desc-Matcher. To estimate the pose of an image in any 3D cloud, keypoints and feature descriptors are extracted from the query image and the point cloud. The trained Desc-Matcher is then used to match the features from the image and the point cloud. A robust pose estimator is used to predict the location and orientation of the query image from the corresponding positions of the matched 2D and 3D features. We carried out an extensive evaluation of the proposed method for indoor and outdoor scenarios and with different types of point clouds to verify the feasibility of our approach. Experimental results show that the proposed approach can reliably estimate the 6-DOF poses of query cameras in any type of 3D point cloud with high precision. We achieved average median errors of 1.09cm/0.27∘ and 19cm/0.39∘ on the Stanford and Cambridge datasets, respectively.

DLALoc: Deep-Learning Accelerated Visual Localization Based on Mesh Representation

Visual localization, i.e., the camera pose localization within a known three-dimensional (3D) model, is a basic component for numerous applications such as autonomous driving cars and augmented reality systems. The most widely used methods from the literature are based on local feature matching between a query image that needs to be localized and database images with known camera poses and local features. However, this method still struggles with different illumination conditions and seasonal changes. Additionally, the scene is normally presented by a sparse structure-from-motion point cloud that has corresponding local features to match. This scene representation depends heavily on different local feature types, and changing the different local feature types requires an expensive feature-matching step to generate the 3D model. Moreover, the state-of-the-art matching strategies are too resource intensive for some real-time applications. Therefore, in this paper, we introduce a novel framework called deep-learning accelerated visual localization (DLALoc) based on mesh representation. In detail, we employ a dense 3D model, i.e., mesh, to represent a scene that can provide more robust 2D-3D matches than 3D point clouds and database images. We can obtain their corresponding 3D points from the depth map rendered from the mesh. Under this scene representation, we use a pretrained multilayer perceptron combined with homotopy continuation to calculate the relative pose of the query and database images. We also use the scale consistency of 2D-3D matches to perform the efficient random sample consensus to find the best 2D inlier set for the subsequential perspective-n-point localization step. Furthermore, we evaluate the proposed visual localization pipeline experimentally on Aachen DayNight v1.1 and RobotCar Seasons datasets. The results show that the proposed approach can achieve state-of-the-art accuracy and shorten the localization time about five times.

Dual Branch PnP Based Network for Monocular 6D Pose Estimation

Monocular 6D pose estimation is a functional task in the field of computer vision and robotics. In recent years, 2D-3D correspondence-based methods have achieved improved performance in multiview and depth data-based scenes. However, for monocular 6D pose estimation, these methods are affected by the prediction results of the 2D-3D correspondences and the robustness of the perspective-n-point (PnP) algorithm. There is still a difference in the distance from the expected estimation effect. To obtain a more effective feature representation result, edge enhancement is proposed to increase the shape information of the object by analyzing the influence of inaccurate 2D-3D matching on 6D pose regression and comparing the effectiveness of the intermediate representation. Furthermore, although the transformation matrix is composed of rotation and translation matrices from 3D model points to 2D pixel points, the two variables are essentially different and the same network cannot be used for both variables in the regression process. Therefore, to improve the effectiveness of the PnP algorithm, this paper designs a dual-branch PnP network to predict rotation and translation information. Finally, the proposed method is verified on the public LM, LM-O and YCB-Video datasets. The ADD(S) values of the proposed method are 94.2 and 62.84 on the LM and LM-O datasets, respectively. The AUC of ADD(-S) value on YCB-Video is 81.1. These experimental results show that the performance of the proposed method is superior to that of similar methods.

2D3D-MVPNet: Learning cross-domain feature descriptors for 2D-3D matching based on multi-view projections of point clouds

2D3D-MVPNet: Learning cross-domain feature descriptors for 2D-3D matching based on multi-view projections of point clouds

A Visual and VAE Based Hierarchical Indoor Localization Method.

Precise localization and pose estimation in indoor environments are commonly employed in a wide range of applications, including robotics, augmented reality, and navigation and positioning services. Such applications can be solved via visual-based localization using a pre-built 3D model. The increase in searching space associated with large scenes can be overcome by retrieving images in advance and subsequently estimating the pose. The majority of current deep learning-based image retrieval methods require labeled data, which increase data annotation costs and complicate the acquisition of data. In this paper, we propose an unsupervised hierarchical indoor localization framework that integrates an unsupervised network variational autoencoder (VAE) with a visual-based Structure-from-Motion (SfM) approach in order to extract global and local features. During the localization process, global features are applied for the image retrieval at the level of the scene map in order to obtain candidate images, and are subsequently used to estimate the pose from 2D-3D matches between query and candidate images. RGB images only are used as the input of the proposed localization system, which is both convenient and challenging. Experimental results reveal that the proposed method can localize images within 0.16 m and 4° in the 7-Scenes data sets and 32.8% within 5 m and 20° in the Baidu data set. Furthermore, our proposed method achieves a higher precision compared to advanced methods.

Voxel-Based Scene Representation for Camera Pose Estimation of a Single RGB Image

Deep learning has been utilized in end-to-end camera pose estimation. To improve the performance, we introduce a camera pose estimation method based on a 2D-3D matching scheme with two convolutional neural networks (CNNs). The scene is divided into voxels, whose size and number are computed according to the scene volume and the number of 3D points. We extract inlier points from the 3D point set in a voxel using random sample consensus (RANSAC)-based plane fitting to obtain a set of interest points consisting of a major plane. These points are subsequently reprojected onto the image using the ground truth camera pose, following which a polygonal region is identified in each voxel using the convex hull. We designed a training dataset for 2D–3D matching, consisting of inlier 3D points, correspondence across image pairs, and the voxel regions in the image. We trained the hierarchical learning structure with two CNNs on the dataset architecture to detect the voxel regions and obtain the location/description of the interest points. Following successful 2D–3D matching, the camera pose was estimated using n-point pose solver in RANSAC. The experiment results show that our method can estimate the camera pose more precisely than previous end-to-end estimators.

LCD: Learned Cross-Domain Descriptors for 2D-3D Matching

In this work, we present a novel method to learn a local cross-domain descriptor for 2D image and 3D point cloud matching. Our proposed method is a dual auto-encoder neural network that maps 2D and 3D input into a shared latent space representation. We show that such local cross-domain descriptors in the shared embedding are more discriminative than those obtained from individual training in 2D and 3D domains. To facilitate the training process, we built a new dataset by collecting ≈ 1.4 millions of 2D-3D correspondences with various lighting conditions and settings from publicly available RGB-D scenes. Our descriptor is evaluated in three main experiments: 2D-3D matching, cross-domain retrieval, and sparse-to-dense depth estimation. Experimental results confirm the robustness of our approach as well as its competitive performance not only in solving cross-domain tasks but also in being able to generalize to solve sole 2D and 3D tasks. Our dataset and code are released publicly at https://hkust-vgd.github.io/lcd.

Divided-volume matching technique for volume displacement estimation of patient positioning in radiation therapy

PurposeWe propose the Divided-Volume Matching (DVM) technique to visualize and estimate three-dimensional (3D) displacements of internal structures to enable more accurate patient positioning for radiation therapy. MethodsA CT volume is divided into a volume of interest (VOI) and a base volume (BV); 2D-3D matching is achieved using digital radiography (DR) images and digitally reconstructed radiographs (DRRs), where the DRRs are iteratively generated by changing the 3D positions and rotation angles of the separate volumes independently to identify the best match with the DR images. We demonstrate this technique with two phantom and two clinical cases. Results3D displacements of the VOIs could be estimated independently and simultaneously with those of the BVs, with accuracies comparable to those of the conventional 2D-3D matching. The proposed technique yielded more suitable matching results when internal displacements occurred in the regions of interest (ROIs). The best matches were found when the ROI was confined to the focused structure, initial displacement values were coarsely adjusted, one volume was matched while the other was fixed, or any combination thereof. ConclusionsThe proposed technique can be used effectively for independent displacement estimations of VOIs and BVs for patient positioning in radiation therapy.

Analysis of forearm rotational motion using biplane fluoroscopic intensity-based 2D–3D matching

Measuring three-dimensional (3D) forearm rotational motion is difficult. We aimed to develop and validate a new method for analyzing 3D forearm rotational motion. We proposed biplane fluoroscopic intensity-based 2D–3D matching, which employs automatic registration processing using the evolutionary optimization strategy. Biplane fluoroscopy was conducted for forearm rotation at 12.5 frames per second along with computed tomography (CT) at one static position. An arm phantom was embedded with eight stainless steel spheres (diameter, 1.5 mm), and forearm rotational motion measurements using the proposed method were compared with those using radiostereometric analysis, which is considered the ground truth. As for the time resolution analysis, we measured radiohumeral joint motion in a patient with posterolateral rotatory instability and compared the 2D–3D matching method with the simulated multiple CT method, which uses CTs at multiple positions and interpolates between the positions. Rotation errors of the radius and ulna between these two methods were 0.31 ± 0.35° and 0.32 ± 0.33°, respectively, translation errors were 0.43 ± 0.35 mm and 0.29 ± 0.25 mm, respectively. Although the 2D–3D method could detect joint dislocation, the multiple CT method could not detect quick motion during joint dislocation. The proposed method enabled high temporal- and spatial-resolution motion analyses with low radiation exposure. Moreover, it enabled the detection of a sudden motion, such as joint dislocation, and may contribute to 3D motion analysis, including joint dislocation, which currently cannot be analyzed using conventional methods.

38. Application of lean-six-sigma methodology in radiotherapy: Breast daily repositioning case

Purpose The concept of quality assurance in radiotherapy encompasses a comprehensive approach to all activities in the department. When hypo-fractionation regiment is adopted, the need of a continuous quality procedures updating is self-evident. At this aim, the Lean Six Sigma Methodology (LSSM) has been applied. LSSM was introduced in industry for providing near-perfect services to large processes, by reducing improbable occurrence. It consists of the synergetic adoption of two methods: (i) create a continuous process flow eliminating waste (Lean); (ii) reduce process variation (Six sigma). LSSM has been prospectively applied to breast patient repositioning. This is the first time the LSSM has been rigorously applied to a RT process. Methods Breast patients treated by hypo-fractionated RT in SIB delivered through VMAT technology with daily 2D-2D matching were considered. The five DMAIC (Define, Measure, Analyze, Improve, Control) LSSM steps were applied by an interdisciplinary team. The process was retrospective measured over 30 months (Jul2014–Dec2016) by querying the RT RecordV distribution symmetry improvement (Skewness moved from 1.4 to 1.1) and outlier eduction, verified by Kurtosis diminution, demonstrated better “normalization” of the procedure after the LSSM application. Conclusions LSSM was successfully applied in a RT department, allowing the breast repositioning matching procedure to be redesigned.

Applying Lean-Six-Sigma Methodology in radiotherapy: Lessons learned by the breast daily repositioning case

Background & purposeLean Six Sigma Methodology (LSSM) was introduced in industry to provide near-perfect services to large processes, by reducing improbable occurrence. LSSM has been applied to redesign the 2D-2D breast repositioning process (Lean) by the retrospective analysis of the database (Six Sigma). Materials & methodsBreast patients with daily 2D-2D matching before RT were considered. The five DMAIC (define, measure, analyze, improve, and control) LSSM steps were applied. The process was retrospectively measured over 30 months (7/2014–12/2016) by querying the RT Record&Verify database. Two Lean instruments (Poka-Yoke and Visual Management) were considered for advancing the process. The new procedure was checked over 6 months (1–6/2017). Results14,931 consecutive shifts from 1342 patients were analyzed. Only 0.8% of patients presented median shifts >1 cm. The major observed discrepancy was the monthly percentage of fractions with almost zero shifts (AZS = 13.2% ± 6.1%). Ishikawa fishbone diagram helped in defining the main discrepancy con-causes. Procedure harmonization involving a multidisciplinary team to increase confidence in matching procedure was defined. AZS was reduced to 4.8% ± 0.6%. Furthermore, distribution symmetry improvement (Skewness moved from 1.4 to 1.1) and outlier reduction, verified by Kurtosis diminution, demonstrated a better “normalization” of the procedure after the LSSM application. ConclusionsLSSM was implemented in a RT department, allowing to redesign the breast repositioning matching procedure.

Video registration in egocentric vision under day and night illumination changes

With the spread of wearable devices and head mounted cameras, a wide range of application requiring precise user localization is now possible. In this paper we propose to treat the problem of obtaining the user position with respect to a known environment as a video registration problem. Video registration, i.e. the task of aligning an input video sequence to a pre-built 3D model, relies on a matching process of local keypoints extracted on the query sequence to a 3D point cloud. The overall registration performance is strictly tied to the actual quality of this 2D-3D matching, and can degrade if environmental conditions such as steep changes in lighting like the ones between day and night occur. To effectively register an egocentric video sequence under these conditions, we propose to tackle the source of the problem: the matching process. To overcome the shortcomings of standard matching techniques, we introduce a novel embedding space that allows us to obtain robust matches by jointly taking into account local descriptors, their spatial arrangement and their temporal robustness. The proposal is evaluated using unconstrained egocentric video sequences both in terms of matching quality and resulting registration performance using different 3D models of historical landmarks. The results show that the proposed method can outperform state of the art registration algorithms, in particular when dealing with the challenges of night and day sequences.

Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localization

Accurately determining the position and orientation from which an image was taken, i.e., computing the camera pose, is a fundamental step in many Computer Vision applications. The pose can be recovered from 2D-3D matches between 2D image positions and points in a 3D model of the scene. Recent advances in Structure-from-Motion allow us to reconstruct large scenes and thus create the need for image-based localization methods that efficiently handle large-scale 3D models while still being effective, i.e., while localizing as many images as possible. This paper presents an approach for large scale image-based localization that is both efficient and effective. At the core of our approach is a novel prioritized matching step that enables us to first consider features more likely to yield 2D-to-3D matches and to terminate the correspondence search as soon as enough matches have been found. Matches initially lost due to quantization are efficiently recovered by integrating 3D-to-2D search. We show how visibility information from the reconstruction process can be used to improve the efficiency of our approach. We evaluate the performance of our method through extensive experiments and demonstrate that it offers the best combination of efficiency and effectiveness among current state-of-the-art approaches for localization.

Does Pelvic Sagittal Inclination in the Supine and Standing Positions Change Over 10 Years of Follow-Up After Total Hip Arthroplasty?

BackgroundFunctional anteversion and inclination of the cup change as the pelvic sagittal inclination (PSI) changes. The purposes of this study were to investigate the chronological changes of PSI during a 10-year follow-up period after total hip arthroplasty (THA) and to report the characteristics of patients who showed a greater than 10° change in the PSI from the supine to the standing position. MethodsThe subjects were 70 patients who were followed up for 10 years after THA. PSI values in the supine and standing positions were measured by 2D-3D matching using computed tomography images and pelvic radiographs. PSI values before THA and 1, 5, and 10 years after THA were compared in both the supine and standing positions. ResultsSupine PSI showed less than 5° of change, whereas standing PSI showed a significant decrease with time over the 10-year period. Although 43% of patients with less than 10° of difference in the PSI between the supine and standing positions before THA increased PSI posteriorly (reclining) more than 10° in standing from the supine position at 10 years, no late dislocation was observed. ConclusionSupine PSI showed no significant change, but standing PSI showed a significant increase posteriorly with time over a 10-year period. However, this PSI change did not reach the level that it caused negative consequences such as late dislocation. The pelvic position in the supine position might still be a good functional reference position of the pelvis for aiming to achieve proper cup alignment at 10 years.

X-Stop® Implantation Effectively Limits Segmental Lumbar Extension in-vivo without Altering the Kinematics of the Adjacent Levels.

To evaluate the in-vivo three-dimensional (3D) vertebral kinematics of the implanted and adjacent segments after implantation of the interspinous process distracting (ISP) device during various functional activities. Eight patients with lumbar spinal stenosis (LSS) scheduled for X-Stop® surgery were recruited. Prior to surgery, patients were scanned with computed tomography/ magnetic resonance imaging (CT/MRI) in order to construct 3D L2 to S1 vertebral models. The lumbar spines of the patients were then imaged using two fluoroscopes while they performed seven functional activities before and after X-Stop® surgeries. The in-vivo 3D vertebral positions were determined in the dual fluoroscopic images using an established 2D-3D matching method. The vertebral 3D ranges of motion (ROM) of the implanted and cranial-caudal adjacent levels were then measured. Primary ROMs of the implanted segments were significantly decreased (p < 0.05) by 50.2% only at torso extension, from preoperative 2.5 ± 1.4° to postoperative 1.1 ± 0.5°, but not significantly (p > 0.05) at flexion, twisting and lateral bending. Primary ROM and the coupled translations and rotations of the implanted and the adjacent levels were not significantly changed during each posture. X-Stop® implantation reduced the in-vivo range of extension by 50.2% at the implanted segment without disturbing 3D kinematics at the adjacent segments.

Evaluation of automated image registration algorithm for image-guided radiotherapy (IGRT)

The performance of an image registration (IR) software was evaluated for automatically detecting known errors simulated through the movement of ExactCouch using an onboard imager. Twenty-seven set-up errors (11 translations, 10 rotations, 6 translation and rotation) were simulated by introducing offset up to ± 15 mm in three principal axes and 0° to ± 1° in yaw. For every simulated error, orthogonal kV radiograph and cone beam CT were acquired in half-fan (CBCT_HF) and full-fan (CBCT_FF) mode. The orthogonal radiographs and CBCTs were automatically co-registered to reference digitally reconstructed radiographs (DRRs) and planning CT using 2D-2D and 3D-3D matching software based on mutual information transformation. A total of 79 image sets (ten pairs of kV X-rays and 69 session of CBCT) were analyzed to determine the (a) reproducibility of IR outcome and (b) residual error, defined as the deviation between the known and IR software detected displacement in translation and rotation. The reproducibility of automatic IR of planning CT and repeat CBCTs taken with and without kilovoltage detector and kilovoltage X-ray source arm movement was excellent with mean SD of 0.1 mm in the translation and 0.0° in rotation. The average residual errors in translation and rotation were within ± 0.5 mm and ± 0.2°, ± 0.9 mm and ± 0.3°, and ± 0.4 mm and ± 0.2° for setup simulated only in translation, rotation, and both translation and rotation. The mean (SD) 3D vector was largest when only translational error was simulated and was 1.7 (1.1) mm for 2D-2D match of reference DRR with radiograph, 1.4 (0.6) and 1.3 (0.5) mm for 3D-3D match of reference CT and CBCT with full fan and half fan, respectively. In conclusion, the image-guided radiation therapy (IGRT) system is accurate within 1.8 mm and 0.4° and reproducible under control condition. Inherent error from any IGRT process should be taken into account while setting clinical IGRT protocol.

Commissioning and early experience with a new-generation low-energy linear accelerator with advanced delivery and imaging functionalities

BackgroundA new-generation low-energy linear accelerator (UNIQUE) was introduced in the clinical arena during 2009 by Varian Medical Systems. The world's first UNIQUE was installed at Oncology Institute of Southern Switzerland and put into clinical operation in June 2010. The aim of the present contribution was to report experience about its commissioning and first year results from clinical operation.MethodsCommissioning data, beam characteristics and the modeling into the treatment planning system were summarized. Imaging system of UNIQUE included a 2D-2D matching capability and tests were performed to identify system repositioning capability. Finally, since the system is capable of delivering volumetric modulated arc therapy with RapidArc, a summary of the tests performed for such modality to assess its performance in preclinical settings and during clinical usage was included.ResultsIsocenter virtual diameter was measured as less than 0.2 mm. Observed accuracy of isocenter determination and repositioning for 2D-2D matching procedures in image guidance was <1.2 mm. Concerning reproducibility and stability over a period of 1 year, deviations from reference were found <0.3 ± 0.2% for linac output, <0.1% for homogeneity, similarly to symmetry. Rotational accuracy of the entire gantry-portal imager system showed a maximum deviation from nominal 0.0 of <1.2 mm. Pre treatment quality assurance of RapidArc plans resulted with a Gamma Agreement Index (fraction of points passing the gamma criteria) of 97.0 ± 1.6% on the first 182 arcs verified.ConclusionsThe results of the commissioning tests and of the first period of clinical operation, resulted meeting specifications and having good margins respect to tolerances. UNIQUE was put into operation for all delivery techniques; in particular, as shown by the pre-treatment quality assurance results, it enabled accurate and safe delivery of RapidArc plans.

A Remote Diagnosis System for Rotating Machinery Using Virtual Reality

It is important to look for alternative forms of physical movement of people and equipments in order to assure diagnosis and maintenance tasks, especially in an environment where workers are subject to danger. An evident and classical solution is the use of the tele-operation and tele-robotics. If the tele-operation helps to solve a lot of real and technical problems, it still remains insufficient to assure an appropriate remote diagnosis and maintenance. The use of virtual reality techniques with the tele-operation can be the solution for an effective remote maintenance and diagnosis. In this paper we show the inefficiency occurred with the use of only tele-operation in the remote maintenance, we introduce our original new system where we use virtual reality techniques and 2D-3D matching (2D camera image-3D virtual objects) with tele-operation to remotely collect machinery vibration data. We explain its structure, implementation and its advantages. We finished by experimenting the system, measuring the different operating times and precision and discuss the results.

PERFORM: a fast object recognition method using intersection of projection

This paper describes an object recognition methodology called PERFORM that finds matches by establishing correspondences between model and image features using this formulation. PERFORM evaluates correspondences by intersecting error regions in the image space. The algorithm is analyzed with respect to theoretical complexity as well as actual running times. When a single solution to the matching problem is sought, the time complexity of the sequential matching algorithm for 2D-2D matching using point features is of the order O(l/sup 3/ N/sup 2/), where N is the number of model features and l is the number of image features. When line features are used, the sequential complexity is of the order O(l/sup 2/ N/sup 2/). When a single solution is sought, PERFORM runs faster than the fastest known algorithm to solve the bounded-error matching problem. The PERFORM method is shown to be easily realizable on both SIMD and MIMD architectures.