Simultaneous Pose Research Articles

Global BIM-point cloud registration and association for construction progress monitoring

Traditional manual and semi-automatic approaches rely heavily on surveying control points and manually picking equivalent point pairs, which is time-consuming and labor-intensive. This paper proposes an automatic algorithm for automatic global BIM-point registration and association to support construction progress monitoring. A representation using distance fields is proposed to efficiently integrate BIM in registration tasks. By leveraging a coarse-to-fine strategy, a primitive-level coarse algorithm is developed to achieve rough alignment between BIM and point cloud. This approach is then complemented by a point-level fine registration approach, which enables simultaneous pose refinement and BIM-point association. Extensive experiments are conducted on the data from simulation and real-world construction sites. The results demonstrate the promising registration and association performance of the proposed algorithm.

SimPS-Net: Simultaneous Pose and Segmentation Network of Surgical Tools

SimPS-Net: Simultaneous Pose and Segmentation Network of Surgical Tools

Fast and deterministic (3+1)DOF point set registration with gravity prior

Point set registration is the technology used to estimate the spatial transformation between two LiDAR scans, which is challenging in the presence of outlier correspondences and noise. Our focus is on 4 degrees of freedom (DOF) point set registration, in which 1DOF rotation and 3DOF translation need to be estimated. It is commonly found in practical scenarios, such as arbitrarily mobile robots equipped with an inertial measurement unit (IMU), terrestrial LiDAR scanners, or planar moving vehicles in urban environments. Recently, many solutions have leveraged branch and bound (BnB) in global and deterministic approaches to solve the registration problem with performance guarantees. However, BnB-based methods are usually time-consuming since their convergence speed is exponential to the dimensionality of the solution domain, and existing methods estimate these 4DOF simultaneously. Our key idea is to speed up BnB-based methods by decoupling the joint pose into separate translation and rotation with the aid of known gravity directions. This effectively reduces the search domain to 3DOF+1DOF, thereby enhancing algorithm efficiency. Specifically, we propose a novel BnB-based consensus maximization method for a fast 3DOF translation search and derive the specific lower and upper bound functions. We then propose an efficient global voting method for estimating the rotation with 1DOF. To demonstrate the superiority of our proposed method, we conduct extensive experiments on both synthetic and real-world datasets. The experimental results show that (1) our proposed method is more robust against outliers and noise than several existing methods and far faster than the existing BnB-based 4DOF method by almost an order of magnitude, (2) our proposed method is robust against the biases in gravity directions, such that the general error of the IMU is acceptable, and (3) thanks to its significant robustness, our proposed method can solve the challenging problem of simultaneous pose and correspondence registration (SPCR). Moreover, the proposed approach is also more robust and accurate than several SPCR benchmark methods. Code is available at https://github.com/Xinyi-tum/Fast-and-Deterministic-Registration.

Efficient and Outlier-Robust Simultaneous Pose and Correspondence Determination by Branch-and-Bound and Transformation Decomposition.

Estimating the pose of a calibrated camera relative to a 3D point set from one image is an important task in computer vision. Perspective-n-Point algorithms are often used if perfect 2D-3D correspondences are known. However, it is difficult to determine 2D-3D correspondences perfectly, and then the simultaneous pose and correspondence determination problem is needed to be solved. Early methods aimed to solve this problem by local optimization. Recently, several new methods are proposed to globally solve this problem by using branch-and-bound (BnB) method, but they tend to be slow because the time complexity of the BnB-based methods is exponential to the dimensionality of the parameter space, and they directly search the 6D parameter space. In this paper, we propose to decompose the joint searching into two separate searching processes by introducing a rotation invariant feature (RIF). Specifically, we construct RIFs from the original 3D and 2D point sets and search for the globally optimal translation to match these two RIFs first. Then, the original 3D point set is translated and matched with the 2D point set to find a globally optimal rotation. Experiments on challenging data show that the proposed method outperforms state-of-the-art methods in terms of both speed and accuracy.

Mobile3DScanner: An Online 3D Scanner for High-quality Object Reconstruction with a Mobile Device.

We present a novel online 3D scanning system for high-quality object reconstruction with a mobile device, called Mobile3DScanner. Using a mobile device equipped with an embedded RGBD camera, our system provides online 3D object reconstruction capability for users to acquire high-quality textured 3D object models. Starting with a simultaneous pose tracking and TSDF fusion module, our system allows users to scan an object with a mobile device to get a 3D model for real-time preview. After the real-time scanning process is completed, the scanned 3D model is globally optimized and mapped with multi-view textures as an efficient postprocess to get the final textured 3D model on the mobile device. Unlike most existing state-of-the-art systems which can only scan homeware objects such as toys with small dimensions due to the limited computation and memory resources of mobile platforms, our system can reconstruct objects with large dimensions such as statues. We propose a novel visual-inertial ICP approach to achieve real-time accurate 6DoF pose tracking of each incoming frame on the front end, while maintaining a keyframe pool on the back end where the keyframe poses are optimized by local BA. Simultaneously, the keyframe depth maps are fused by the optimized poses to a TSDF model in real-time. Especially, we propose a novel adaptive voxel resizing strategy to solve the out-of-memory problem of large dimension TSDF fusion on mobile platforms. In the post-process, the keyframe poses are globally optimized and the keyframe depth maps are optimized and fused to obtain a final object model with more accurate geometry. The experiments with quantitative and qualitative evaluation demonstrate the effectiveness of the proposed 3D scanning system based on a mobile device, which can successfully achieve online high-quality 3D reconstruction of natural objects with larger dimensions for efficient AR content creation.

An Integrated GNSS/LiDAR-SLAM Pose Estimation Framework for Large-Scale Map Building in Partially GNSS-Denied Environments

This article presents an integrated global navigation satellite system/light detection and ranging (GNSS/LiDAR)-based simultaneous localization and mapping (SLAM) pose estimation framework to perform large-scale 3-D map building in partially GNSS-denied outdoor environments. The framework takes the advantage of the complementarity between GNSS positioning and LiDAR-SLAM to decompose the map building task according to the GNSS real-time kinematic (RTK) status. When mapping in GNSS-denied scenes, a 3-D LiDAR-SLAM algorithm is adopted to estimate poses and a correction algorithm is presented to correct drift errors. On the other hand, when mapping in open scenes, a GNSS-initialized LiDAR mapping algorithm (GL-mapping) is proposed to loosely couple GNSS positioning and LiDAR data registration. It can perform the orientation estimation without the use of either the high-cost inertial sensing device or the GNSS dual-antenna. Experiments are conducted in large-scale outdoor environments to demonstrate that the proposed framework can accomplish simultaneous pose estimation and map building with high precision in both open scenes and GNSS-denied scenes.

Boosting Multi-Vehicle Tracking with a Joint Object Detection and Viewpoint Estimation Sensor.

In this work, we address the problem of multi-vehicle detection and tracking for traffic monitoring applications. We preset a novel intelligent visual sensor for tracking-by-detection with simultaneous pose estimation. Essentially, we adapt an Extended Kalman Filter (EKF) to work not only with the detections of the vehicles but also with their estimated coarse viewpoints, directly obtained with the vision sensor. We show that enhancing the tracking with observations of the vehicle pose, results in a better estimation of the vehicles trajectories. For the simultaneous object detection and viewpoint estimation task, we present and evaluate two independent solutions. One is based on a fast GPU implementation of a Histogram of Oriented Gradients (HOG) detector with Support Vector Machines (SVMs). For the second, we adequately modify and train the Faster R-CNN deep learning model, in order to recover from it not only the object localization but also an estimation of its pose. Finally, we publicly release a challenging dataset, the GRAM Road Traffic Monitoring (GRAM-RTM), which has been especially designed for evaluating multi-vehicle tracking approaches within the context of traffic monitoring applications. It comprises more than 700 unique vehicles annotated across more than 40.300 frames of three videos. We expect the GRAM-RTM becomes a benchmark in vehicle detection and tracking, providing the computer vision and intelligent transportation systems communities with a standard set of images, annotations and evaluation procedures for multi-vehicle tracking. We present a thorough experimental evaluation of our approaches with the GRAM-RTM, which will be useful for establishing further comparisons. The results obtained confirm that the simultaneous integration of vehicle localizations and pose estimations as observations in an EKF, improves the tracking results.

Simultaneous Pose and Correspondence Determination Combining Softassign and Orthogonal Iteration

Pose estimation with unknown correspondences between 3D object points and 2D image points is known as the simultaneous pose and correspondence determination problem in the field of computer vision. It currently is still diffcutlt to solve particularly with the appearance of occlusion and cluster. In this paper, we present a new iterative algorithm for the pose estimation of an 3D object without any additional 3D-2D corresondence infromation. Our method combines SoftAssign algorithm for derterming the correspondence and OI (orthogonal iteration) algorithm for computing the pose. An assignment matrix which describes the correspondence is first introduced to the objective function of OI algorithm, and the simultaneous pose and correspondence determination problem is formulated as that of minimizing the weighted object space collinearity error. The pose and correspondence are evolved from an initial pose to an optimum value by nesting the two algorithms into one deterministic annealing process. Simulation and experimental results demonstrate that the proposed method is computationally more efficient and more accurate than the state-of-art methods.

Simultaneous pose and reliability estimation using convolutional neural network and Rao–Blackwellized particle filter

ABSTRACTIn this study, we propose a novel localization approach that simultaneously estimates the reliability of estimation results. In the approach, a convolutional neural network (CNN) is used to make decision whether the localization process has failed or not. We train the CNN using a dataset that includes successful localization results and faults. However, the decision will contain some noise and many misdetection results may occur when the decision made by the CNN is used directly to detect faults. Therefore, we estimate both a robot's pose and reliability of the localization results based on the decision. To simultaneously estimate the robot's pose and reliability, we propose a new graphical model and implement a Rao–Blackwellized particle filter based on the model. We evaluated the proposed approach based on simulations and actual environments, which showed that the reliability estimated by the proposed approach can be used as an exact criterion for detecting localization faults. In addition, we show that the proposed approach can be applied in actual environments even when a dataset created from a simulation is used to train the CNN.

Globally-Optimal Inlier Set Maximisation for Camera Pose and Correspondence Estimation

Estimating the 6-DoF pose of a camera from a single image relative to a 3D point-set is an important task for many computer vision applications. Perspective-n-point solvers are routinely used for camera pose estimation, but are contingent on the provision of good quality 2D-3D correspondences. However, finding cross-modality correspondences between 2D image points and a 3D point-set is non-trivial, particularly when only geometric information is known. Existing approaches to the simultaneous pose and correspondence problem use local optimisation, and are therefore unlikely to find the optimal solution without a good pose initialisation, or introduce restrictive assumptions. Since a large proportion of outliers and many local optima are common for this problem, we instead propose a robust and globally-optimal inlier set maximisation approach that jointly estimates the optimal camera pose and correspondences. Our approach employs branch-and-bound to search the 6D space of camera poses, guaranteeing global optimality without requiring a pose prior. The geometry of SE(3) is used to find novel upper and lower bounds on the number of inliers and local optimisation is integrated to accelerate convergence. The algorithm outperforms existing approaches on challenging synthetic and real datasets, reliably finding the global optimum, with a GPU implementation greatly reducing runtime.

Emotion Recognition from Facial Expressions using Images with Pose, Illumination and Age Variation for Human-Computer/Robot Interaction

A technique for emotion recognition from facial expressions in images with simultaneous pose, illumination and age variation in real time is proposed in this paper. The basic emotions considered are anger, disgust, happy, surprise, and neutral. Feature vectors that were formed from images from the CMU-MultiPIE database for pose and illumination were used for training the classifier. For real-time implementation, Raspberry Pi II was used, which can be placed on a robot to recognize emotions in interactive real-time applications. The proposed method includes face detection using Viola Jones Haar cascade, Active Shape Model (ASM) for feature extraction, and AdaBoost for classification in real- time. Performance of the proposed method was validated in real time by testing with subjects from different age groups expressing basic emotions with varying pose and illumination. 96% recognition accuracy at an average time of 120 ms was obtained. The results are encouraging, as the proposed method gives better accuracy with higher speed compared to existing methods from the literature. The major contribution and strength of the proposed method lie in marking suitable feature points on the face, its speed and invariance to pose, illumination and age in real time.

Joint camera blur and pose estimation from aliased data.

A joint-estimation algorithm is presented that enables simultaneous camera blur and pose estimation from a known calibration target in the presence of aliasing. Specifically, a parametric maximum-likelihood (ML) point-spread function estimate is derived for characterizing a camera's optical imperfections through the use of a calibration target in an otherwise loosely controlled environment. The imaging perspective, ambient-light levels, target reflectance, detector gain and offset, quantum efficiency, and read-noise levels are all treated as nuisance parameters. The Cramér-Rao bound is derived, and simulations demonstrate that the proposed estimator achieves near optimal mean squared error performance. The proposed method is applied to experimental data to validate the fidelity of the forward models as well as to establish the utility of the resulting ML estimates for both system identification and subsequent image restoration.

Real-Time Simultaneous Pose and Shape Estimation for Articulated Objects Using a Single Depth Camera.

In this paper we present a novel real-time algorithm for simultaneous pose and shape estimation for articulated objects, such as human beings and animals. The key of our pose estimation component is to embed the articulated deformation model with exponential-maps-based parametrization into a Gaussian Mixture Model. Benefiting from this probabilistic measurement model, our algorithm requires no explicit point correspondences as opposed to most existing methods. Consequently, our approach is less sensitive to local minimum and handles fast and complex motions well. Moreover, our novel shape adaptation algorithm based on the same probabilistic model automatically captures the shape of the subjects during the dynamic pose estimation process. The personalized shape model in turn improves the tracking accuracy. Furthermore, we propose novel approaches to use either a mesh model or a sphere-set model as the template for both pose and shape estimation under this unified framework. Extensive evaluations on publicly available data sets demonstrate that our method outperforms most state-of-the-art pose estimation algorithms with large margin, especially in the case of challenging motions. Furthermore, our shape estimation method achieves comparable accuracy with state of the arts, yet requires neither statistical shape model nor extra calibration procedure. Our algorithm is not only accurate but also fast, we have implemented the entire processing pipeline on GPU. It can achieve up to 60 frames per second on a middle-range graphics card.

Simultaneous pose estimation and patient-specific model reconstruction from single image using maximum penalized likelihood estimation (MPLE)

Pose estimation and shape reconstruction are two common problems in pattern recognition, which oftentimes are tackled separately. But in some medical applications, both pose and shape of a target anatomy are crucial and have to be estimated from intra-operative two-dimensional images. As pose estimation and shape reconstruction are two coupled problems, previous feature-based methods solved the problems in consecutive stages utilizing statistical shape models (SSMs). Only the mean shape of SSM is used to estimate the pose by finding paired correspondences in the first stage, based on which SSM-regularized surface deformations are performed in the following stages. Such a strategy heavily depends on the paired correspondences. In this paper, bypassing correspondence establishment, a novel method is proposed to simultaneously optimize pose and shape by formulating the coupled problems as a maximum penalized likelihood estimation (MPLE). It models oriented object contours as a mixture of von Mises–Fisher Gaussian distributions, and solves the MPLE effectively using a global optimizer. It utilize the entire knowledge of SSM in both solving pose and reconstructing shape, providing robustness to large offsets in initializations. Leave-one-out cross-validations on 19 dry cadaveric femurs were performed using simulated X-ray images with accurate ground-truth, under various initial conditions. Our method achieved sub-degree rotational and sub-millimeter in-plane translational pose estimation errors, and an approximately one millimeter average mean surface-to-surface distance in shape reconstruction. The reconstruction accuracy is comparable to those reported in the literature using two or more images. The experiment results are encouraging and indicate that an accurate simultaneous 3D–2D pose estimation and surface reconstruction is achievable from one single image.

Pose Tracking and Sensor Self-Calibration for an All-terrain Autonomous Vehicle

In this work we address the simultaneous pose tracking and sensor self-calibration problem by applying a pose-graph optimization approach. A factor-graph is employed to store robot pose estimates at different time instants and calibration parameters such as magnetometer hard and soft iron distortion and gyroscope bias. Specific factors are developed in this paper to handle Ackermann kinematic readings, inertial measurement units, magnetometers and global positioning systems. An experimental evaluation supports the viability of the approach considering an autonomous all-terrain vehicle, for which we perform calibration and real-time pose tracking during navigation.

Simultaneous Pose and Correspondence Estimation Based on Genetic Algorithm

Although several algorithms have been presented to solve the simultaneous pose and correspondence estimation problem, the correct solution may not be reached to with the traditional random-start initialization method. In this paper, we derive a novel method which estimates the initial value based on genetic algorithm, considering the influences of different initial guesses comprehensively. First, a set of random initial guesses is generated as candidate solutions. Second, the assignment matrix and the perspective projection error are computed for each candidate solution. And then each individual is modified (selection, crossover, and mutation) in current iterative process. Finally, the fittest individual is stochastically selected from the final population. With the presented initialization method, the proper initial guess could be first calculated and then the simultaneous pose and correspondence estimation problem could be solved easily. Simulation results with synthetic data and experiments on real images prove the effectiveness and robustness of our proposed method.

Extended gravitational pose estimation

The model-to-image registration problem is a problem of determining the position and orientation (the pose) of a three-dimensional object with respect to a camera coordinate system. When there is no additional information available to constrain the pose of the object and to constrain the correspondence of object features to image features, the problem is also known as simultaneous pose and correspondence problem, or correspondenceless pose estimation problem. In this paper, we present a new algorithm, called extended gravitational pose estimation (EGPE), for determining the pose and correspondence simultaneously. The algorithm is based on gravitational pose estimation (GPE) algorithm. In our algorithm, the original GPE has been revised to deal with the problem with false image points. For problems with both occluded object points and false image points, we firstly applied single-link agglomerative clustering algorithm to pick out occluded object points when a local minimum has been found, then the revised GPE is applied again on the clustering result to update rotation and translation of the object model. EGPE has been verified on both synthetic images and real images. Empirical results show that EGPE is faster, more stable and reliable than most current algorithms, and can be used in real applications.

Hybrid Force and Vision-Based Contour Following of Planar Robots

Manipulators interacted with uncalibrated environments have limited dexterity due to constraints imposed by unknown environments. However, to perform science or industrial operations, it is necessary to be able to position and orient these manipulators on targets in order to accomplish required control tasks. This article describes how one might enhance manipulator dexterity for planar contour following tasks using hybrid force and vision-based control. The proposed control approach can guarantee task precision employing only a single-axis force sensor and an imprecisely calibrated CCD camera whose optical axis is perpendicular to the planar workspace. The goal of the autonomous task is to drive an instrument mounted on the end-effector of a planar robotic manipulator to follow a visually determined planar contour and continue tracking the contour in desired pose, contact force, and speed, all demanding time-varying, with precision. The proposed control architecture is suitable for applications that require simultaneous force and pose control in unknown environments. Our approach is successfully validated in a real task environment by performing experiments with an industrial robotic manipulator.

Merging artificial objects with marker-less video sequences based on the interacting multiple model method

Inserting synthetic objects into video sequences has gained much interest in recent years. Fast and robust vision-based algorithms are necessary to make such an application possible. Traditional pose tracking schemes using recursive structure from motion techniques adopt one Kalman filter and thus only favor a certain type of camera motion. We propose a robust simultaneous pose tracking and structure recovery algorithm using the interacting multiple model (IMM) to improve performance. In particular, a set of three extended Kalman filters (EKFs), each describing a frequently occurring camera motion in real situations (general, pure translation, pure rotation), is applied within the IMM framework to track the pose of a scene. Another set of EKFs,one filter for each model point, is used to refine the positions of the model features in the 3-D space. The filters for pose tracking and structure refinement are executed in an interleaved manner. The results are used for inserting virtual objects into the original video footage. The performance of the algorithm is demonstrated with both synthetic and real data. Comparisons with different approaches have been performed and show that our method is more efficient and accurate.

Simultaneous pose recovery and camera registration from multiple views of a walking person

We present an algorithm to estimate the body pose of a walking person given synchronized video input from multiple uncalibrated cameras. We construct an appearance model of human walking motion by generating examples from the space of body poses and camera locations, and clustering them using expectation–maximization. Given a segmented input video sequence, we find the closest matching appearance cluster for each silhouette and use the sequence of matched clusters to extrapolate the position of the camera with respect to the person's direction of motion. For each frame, the matching cluster also provides an estimate of the walking phase. We combine these estimates from all views and find the most likely sequence of walking poses using a cyclical, feed-forward hidden Markov model. Our algorithm requires no manual initialization and no prior knowledge about the locations of the cameras.