Pose Estimation Process Research Articles

Enhancing precision in 3D printing for highly functional printing with high-speed vision

3D printing has revolutionized product design and manufacturing across various industries by enabling the creation of complex geometries with minimal waste. Despite its advancements, 3D printing still faces significant challenges, including spatial constraints and process control limitations. This paper introduces novel approaches to improve the functionality and precision of material extrusion 3D printers for the fused deposition modeling method, particularly for additional printing tasks such as printing on existing objects or continuing a print on a relocated object without prior knowledge of its position or even the environment is changed. We introduce a compensation system integrating a high-speed vision system for robot arms to address these challenges. Our system employs a three-step pose estimation process—fast point feature histograms (FPFH)-based, corner-based, and sub-pixel edge-based methods—to ensure high accuracy in restoring the position of printed pieces for additional printing tasks on a given object. Experimental results demonstrate substantial improvements in printing precision, with the system achieving sub-millimeter and sub-pixel accuracy. These advancements not only eliminate work area constraints but also enhance the adaptability and reliability of 3D printing processes.

PRSCS-Net: Progressive 3D/2D rigid Registration network with the guidance of Single-view Cycle Synthesis

The 3D/2D registration for 3D pre-operative images (computed tomography, CT) and 2D intra-operative images (X-ray) plays an important role in image-guided spine surgeries. Conventional iterative-based approaches suffer from time-consuming processes. Existing learning-based approaches require high computational costs and face poor performance on large misalignment because of projection-induced losses or ill-posed reconstruction. In this paper, we propose a Progressive 3D/2D rigid Registration network with the guidance of Single-view Cycle Synthesis, named PRSCS-Net. Specifically, we first introduce the differentiable backward/forward projection operator into the single-view cycle synthesis network, which reconstructs corresponding 3D geometry features from two 2D intra-operative view images (one from the input, and the other from the synthesis). In this way, the problem of limited views during reconstruction can be solved. Subsequently, we employ a self-reconstruction path to extract latent representation from pre-operative 3D CT images. The following pose estimation process will be performed in the 3D geometry feature space, which can solve the dimensional gap, greatly reduce the computational complexity, and ensure that the features extracted from pre-operative and intra-operative images are as relevant as possible to pose estimation. Furthermore, to enhance the ability of our model for handling large misalignment, we develop a progressive registration path, including two sub-registration networks, aiming to estimate the pose parameters via two-step warping volume features. Finally, our proposed method has been evaluated on a public dataset CTSpine1k and an in-house dataset C-ArmLSpine for 3D/2D registration. Results demonstrate that PRSCS-Net achieves state-of-the-art registration performance in terms of registration accuracy, robustness, and generalizability compared with existing methods. Thus, PRSCS-Net has potential for clinical spinal disease surgical planning and surgical navigation systems.

Personality expression in cartoon animal characters using Sasang typology

AbstractThe movement style is an adequate descriptor of different personalities. While many studies investigate the relationship between apparent personality and high‐level motion qualities in humans, similar research for animal characters still needs to be done. The variety in animals' skeletal configurations and texture complicates their pose estimation process. Our affect analysis framework includes a workflow for pose extraction in animal characters and a parameterization of the high‐level animal motion descriptors inspired by Laban movement analysis. Using a data set of quadruped walk cycles, we prove the display of typologies in cartoon animal characters, reporting the point‐biserial correlation between our motion parameters and the Sasang categories that reflect different personalities.

Subject-Specific Human Modeling for Human Pose Estimation

3-D human pose estimation or human tracking has always been the focus of research in the human–computer interaction community. As the calibration step of human pose estimation, subject-specific modeling is crucially important to the subsequent pose estimation process. It not only provides a priori knowledge but also clearly defines the tracking target. This article presents a fully automatic subject modeling framework to reconstruct human pose, shape, as well as the body texture in a challenging optimization scenario. By integrating powerful differentiable rendering into the subject-specific modeling pipeline, the proposed method transforms the texture reconstruction problem into analysis by synthesis minimization and solves it efficiently by a gradient-based method. Furthermore, a novel covariance matrix adaptation annealing algorithm is proposed to attack the high-dimensional multimodal optimization problem in an adaptive manner. The domain knowledge of hierarchical human anatomy is seamlessly injected to the annealing optimization process by using a soft covariance matrix mask. All together contributes to the novel algorithm robust to the temptation of local minima. Experiments on the Human3.6 M dataset and the People-Snapshot dataset demonstrate the competitive results to the state of the art both qualitatively and quantitatively.

Occlusion Handler Density Networks for 3D Multimodal Joint Location of Hand Pose Hypothesis

Predicting the pose parameters during the hand pose estimation (HPE) process is an ill-posed challenge. This is due to severe self-occluded joints of the hand. The existing approaches for predicting pose parameters of the hand, utilize a single-value mapping of an input image to generate final pose output. This way makes it difficult to handle occlusion especially when it comes from the multimodal pose hypothesis. This paper introduces an effective method of handling multimodal joint occlusion using the negative log-likelihood of a multimodal mixture-of-Gaussians through a hybrid hierarchical mixture density network (HHMDN). The proposed approach generates multiple feasible hypotheses of 3D poses with visibility, unimodal and multimodal distribution units to locate joint visibility. The visible features are extracted and fed into the Convolutional Neural Networks (CNN) layer of the HHMDN for feature learning. Finally, the effectiveness of the proposed method is proved on ICVL, NYU, and BigHand public hand pose datasets. The imperative results show that the proposed method in this paper is effective as it achieves a visibility error of 30.3mm, which is less error compared to many state-of-the-art approaches that use different distributions of visible and occluded joints.

AUTOMATIC ALGORITHM FOR GEOREFERENCING HISTORICAL-TO-NOWADAYS AERIAL IMAGES ACQUIRED IN NATURAL ENVIRONMENTS

Abstract. Automatic georeferencing for historical-to-nowadays aerial images represents the main ingredient for supplying territory evolution analysis and environmental monitoring. Existing georeferencing methods based on feature extraction and matching reported successful results for multi-epoch aerial images acquired in structured and man-made environments. While improving the state-of- the-art of the multi-epoch georeferencing problem, such frameworks present several limitations when applied to unstructured scenes, such as natural feature-less environments, characterized by homogenous or texture-less areas. This is mainly due to the lack of structured areas which often results in sparse and ambiguous feature matches, introducing inconsistencies during the pose estimation process. This paper addresses the automatic georeferencing problem for historical aerial images acquired in unstructured natural environments. The research work presented in this paper introduces a feature-less algorithm designed to perform historical-to- nowadays image matching for pose estimation in a fully automatic fashion. The proposed algorithm operates within two stages: (i) 2D patch extraction and matching and (ii) 3D patch-based local alignment. The final output is a set of 3D patch matches and the 3D rigid transformation relating each homologous patches. The obtained 3D point matches are designed to be injected into traditional multi-views pose optimisation engines. Experimental results on real datasets acquired over Fabas area situated in France demonstrate the effectiveness of the proposed method. Our findings illustrate that the proposed georeferencing technique provides accurate results in presence of large periods of time separating historical from nowadays aerial images (up to 48 years time span).

GAC3D: improving monocular 3D object detection with ground-guide model and adaptive convolution.

Monocular 3D object detection has recently become prevalent in autonomous driving and navigation applications due to its cost-efficiency and easy-to-embed to existent vehicles. The most challenging task in monocular vision is to estimate a reliable object’s location cause of the lack of depth information in RGB images. Many methods tackle this ill-posed problem by directly regressing the object’s depth or take the depth map as a supplement input to enhance the model’s results. However, the performance relies heavily on the estimated depth map quality, which is bias to the training data. In this work, we propose depth-adaptive convolution to replace the traditional 2D convolution to deal with the divergent context of the image’s features. This lead to significant improvement in both training convergence and testing accuracy. Second, we propose a ground plane model that utilizes geometric constraints in the pose estimation process. With the new method, named GAC3D, we achieve better detection results. We demonstrate our approach on the KITTI 3D Object Detection benchmark, which outperforms existing monocular methods.

UKF-Based Vehicle Pose Estimation under Randomly Occurring Deception Attacks

Considering various cyberattacks aiming at the Internet of Vehicles (IoV), secure pose estimation has become an essential problem for ground vehicles. This paper proposes a pose estimation approach for ground vehicles under randomly occurring deception attacks. By modeling attacks as signals added to measurements with a certain probability, the attack model has been presented and incorporated into the existing process and measurement equations of ground vehicle pose estimation based on multisensor fusion. An unscented Kalman filter-based secure pose estimator is then proposed to generate a stable estimate of the vehicle pose states; i.e., an upper bound for the estimation error covariance is guaranteed. Finally, the simulation and experiments are conducted on a simple but effective single-input-single-output dynamic system and the ground vehicle model to show the effectiveness of UKF-based secure pose estimation. Particularly, the proposed scheme outperforms the conventional Kalman filter, not only by resulting in more accurate estimation but also by providing a theoretically proved upper bound of error covariance matrices that could be used as an indication of the estimator’s status.

Fast 6D object pose estimation of shell parts for robotic assembly

Shell parts which have symmetric structure, even surface and flat color are common in industrial manufacturing applications. Moreover, the inner and outer surface of them are of similar shape and close to each other. These features make the pose estimation process of the shell parts challenging. Aiming at 6D pose error compensation of parts in high-precision robotic assembly tasks, this work proposes a fast 6D pose estimation method tailored for the shell parts. With a binocular structured light camera to acquire the point cloud data, the proposed method consists of two phases, namely the initial pose estimation phase and local pose estimation phase. In the former phase, principle component analysis algorithm is utilized to calculate a primary pose estimation. An initial pose correction and translation offset methods serve to solve the local optimal estimation problem of the iterative closest point (ICP) algorithm. In the latter phase, the voxel sampling and a novel weighted point-to-plane ICP algorithms are applied to boost the efficiency of the pose estimation method. With two typical shell parts, a simulation and an experiment of pose estimation are conducted to validate the effectiveness of the proposed method. Experiment results prove that both the accuracy and efficiency of the pose estimation method meet the requirement of the given assembly tasks.

Pose Estimation of Excavator Manipulator Based on Monocular Vision Marker System.

Excavation is one of the broadest activities in the construction industry, often affected by safety and productivity. To address these problems, it is necessary for construction sites to automatically monitor the poses of excavator manipulators in real time. Based on computer vision (CV) technology, an approach, through a monocular camera and marker, was proposed to estimate the pose parameters (including orientation and position) of the excavator manipulator. To simulate the pose estimation process, a measurement system was established with a common camera and marker. Through comprehensive experiments and error analysis, this approach showed that the maximum detectable depth of the system is greater than 11 m, the orientation error is less than 8.5°, and the position error is less than 22 mm. A prototype of the system that proved the feasibility of the proposed method was tested. Furthermore, this study provides an alternative CV technology for monitoring construction machines.

Pipe pose estimation based on machine vision

To realize the automatic assembly and connection of pipelines, one of the core tasks is target object identification and pose estimation. The problem is challenging due to low precision and efficiency caused by the pipes being textureless and self-occluding. In this work, we introduce a machine vision-based method for 6D pipe pose estimation. First, the pipe’s initial pose is estimated by a template matching algorithm. Second, a 3D-2D projection mapping relationship is established, and the distance between the edge pixel points and the edge of the projection model is optimized using the least squares method to obtain a more accurate pipe pose. Experiments demonstrate that the proposed method is able to robustly estimate pose in real environments, while achieving position and pitch accuracies of 0.0732 mm and 0.5°, respectively. Furthermore, the whole pose estimation process lasted 2–3 s, which meets the requirements of industrial applications.

OFM-SLAM: A Visual Semantic SLAM for Dynamic Indoor Environments

Most of the current visual Simultaneous Localization and Mapping (SLAM) algorithms are designed based on the assumption of a static environment, and their robustness and accuracy in the dynamic environment do not behave well. The reason is that moving objects in the scene will cause the mismatch of features in the pose estimation process, which further affects its positioning and mapping accuracy. In the meantime, the three-dimensional semantic map plays a key role in mobile robot navigation, path planning, and other tasks. In this paper, we present OFM-SLAM: Optical Flow combining MASK-RCNN SLAM, a novel visual SLAM for semantic mapping in dynamic indoor environments. Firstly, we use the Mask-RCNN network to detect potential moving objects which can generate masks of dynamic objects. Secondly, an optical flow method is adopted to detect dynamic feature points. Then, we combine the optical flow method and the MASK-RCNN for full dynamic points’ culling, and the SLAM system is able to track without these dynamic points. Finally, the semantic labels obtained from MASK-RCNN are mapped to the point cloud for generating a three-dimensional semantic map that only contains the static parts of the scenes and their semantic information. We evaluate our system in public TUM datasets. The results of our experiments demonstrate that our system is more effective in dynamic scenarios, and the OFM-SLAM can estimate the camera pose more accurately and acquire a more precise localization in the high dynamic environment.

Structure From Motion on XSlit Cameras.

We present a structure-from-motion (SfM) framework based on a special type of multi-perspective camera called the cross-slit or XSlit camera. Traditional perspective camera based SfM suffers from the scale ambiguity which is inherent to the pinhole camera geometry. In contrast, an XSlit camera captures rays passing through two oblique lines in 3D space and we show such ray geometry directly resolves the scale ambiguity when employed for SfM. To accommodate the XSlit cameras, we develop tailored feature matching, camera pose estimation, triangulation, and bundle adjustment techniques. Specifically, we devise a SIFT feature variant using non-uniform Gaussian kernels to handle the distortions in XSlit images for reliable feature matching. Moreover, we demonstrate that the XSlit camera exhibits ambiguities in pose estimation process which can not be handled by existing work. Consequently, we propose a 14 point algorithm to properly handle the XSlit degeneracy and estimate the relative pose between XSlit cameras from feature correspondences. We further exploit the unique depth-dependent aspect ratio (DDAR) property to improve the bundle adjustment for the XSlit camera. Synthetic and real experiments demonstrate that the proposed XSlit SfM can conduct reliable and high fidelity 3D reconstruction at an absolute scale.

Visual SLAM With Drift-Free Rotation Estimation in Manhattan World

This letter presents an efficient and accurate simultaneous localization and mapping (SLAM) system in man-made environments. The Manhattan world assumption is imposed, with which the global orientation is obtained. The drift-free rotational motion estimation is derived from the structural regularities using line features. In particular, a two-stage vanishing points (VPs) estimation method is developed, which consists of a short-term tracking module to track the clustered line features and a long-term searching module to generate abundant sets of VPs candidates and retrieve the optimal one. A least square problem is constructed and solved to provide refined VPs with the clusters of structural line features every frame. We make full use of the absolute orientation estimation to benefit the whole SLAM process. In particular, we utilize the absolute orientation estimation to increase the localization accuracy in the front end, and formulate a linear batch camera pose refinement problem with the known rotations to improve the real time performance in the back end. Experiments on both synthesized and real-world scenes reveal results with high-precision in the real time camera pose estimation process and high-speed in pose graph optimization process compared with the existing state-of-the-art methods.

Towards In Situ Backlash Estimation of Continuum Robots Using an Endoscopic Camera

Accurate control of continuum robots requires handling non-linear behaviors between actuators and distal effectors. In this letter, we develop a method for estimating the non-linearities of tendon-driven degrees of freedom of flexible endoscopic systems by using a distal endoscopic camera and encoders at the proximal side. The proposed approach separates the non-linearities in two parts, namely a pure non uniform backlash and a non-linear function. The backlash is estimated without relying on any model, while the non-linear function is obtained by a pose estimation process. Experiments realized on a robotic flexible endoscopy platform (STRAS) show the validity of the approach for estimating in situ the quasi-static behavior of the robot and for compensating the non-linearities of the motion transmission.

A Novel Texture-Less Object Oriented Visual SLAM System

Traditional mapping modules in Visual Simultaneous Localization and Mapping (i.e. Visual SLAM) systems can only estimate 3D information of isolated sparse or semi-dense feature points. But there are lots of object instances in the environments which geometric information can be utilized to enhance the quality of mapping and localization. Hence, it is required for the Visual SLAM system to utilize high-dimensional features like object instances or structural lines in mapping and localization. To meet the gap between the above requirements and the traditional implementation of Visual SLAM systems, we present in this paper a novel Visual SLAM method that can effectively utilize texture-less object instances for mapping and localization. The proposed Visual SLAM method includes newly designed feature extraction, matching, localization and mapping modules, which jointly use object features and point features to estimate camera 6-DOF poses and do richer map construction. A group of organized raster points is used to represent objects during feature matching and pose estimation process in the proposed Visual SLAM pipeline. Owing to the object feature fusion in the co-visibility graph it could conduct scale aware bundle adjustments to reduce accumulated error. The advantages of proposed Visual SLAM method are demonstrated through experiments conducted both on synthetic datasets and real-world datasets.

Model-free pose estimation using point cloud data

A model-free pose (relative attitude and position) estimation process using point cloud data is developed to support envisioned autonomous satellite servicing missions. Unlike model-based pose estimation that typically requires a three-dimensional geometric model of a satellite to be serviced, the model-free pose estimation presented in this paper employs object features detected in situ using point cloud data. Its overall process is similar to existing model-free pose estimation algorithms in that it requires edge detection, object boundary identification, edge vector determination to form an orthogonal triad for attitude estimation, and object area and centroid calculation for position estimation with respect to the sensor. However, significant improvements have been made toward fully autonomous, model-free pose estimation by employing the Hough transform that allows robust line fitting of jagged edges resulting from object rotations and the discrete nature of point cloud data. In addition, a plane fitting algorithm using the homogeneous transformation was developed to compute plane normal vectors to help improve attitude estimation accuracy. Using both measured and simulated point cloud data of test objects, the performance of the developed model-free pose estimation process was examined. It was able to produce accurate position and attitude of the objects. Future effort will be needed to include more general geometric features such as circles, ellipses, and arcs to provide capabilities needed toward fully autonomous model-free pose estimation process.

A multi-constraints based pose coordination model for large volume components assembly

In the assembly process of large volume product, engineering constraints limit the relative pose of components and serve as a standard for judging assembly quality. However, in the traditional process of target pose estimation, a general method is needed for establishing the correlation between engineering constraints and product pose, and it is difficult to evaluate pose by constraints comprehensively. Therefore, the process of target pose estimation and evaluation is separated. In this paper, a pose coordination model based on multi-constraints is proposed, which includes pre-processing, pose estimation, pose adjustment and evaluation. Firstly, engineering constraints are decoupled into 4 types of Minimum Geometrical Reference Constraints (MGRC), and the inequalities for solving target pose are formulated. Then the Constraint Coordination Index (CCI) is defined as the optimization objective to solve the target pose. Finally, with CCI as the numerical index, the target pose is evaluated to illustrate the quality of assembly. Taking the simulation experiment of wing-fuselage jointing as an example, the external and internal parameters of model are analyzed, and the pose estimation based on multi-constraints reduces the CCI by 12%, compared with the point-set-registration method.

Optical-aided, autonomous and optimal space rendezvous with a non-cooperative target

Performing rendezvous in close-proximity of a non-cooperative target is a challenging task, especially if certain performance requirements have to be met. When maneuvering in hazardous scenarios in order to accomplish either refuelling, inspection, repair or dismissal tasks, optimality and safety are relevant aspects to be sought. Additionally, autonomy is essential in unmanned missions. All these aspects can be achieved through purposely intended Guidance Navigation and Control (GNC) architectures. Optimality is obtained by means of the guidance system, that is by a proper optimal trajectory calculation. Safety and accuracy are guaranteed by evaluation of target's relative pose and shape, which is exploited by the navigation system. Among all the possible choices, optical hardware is becoming widely studied for its high accuracy and capability to reconstruct three-dimensional properties of the observed scene. The present work investigates the reciprocal influences between the guidance and navigation subsystems, putting higher focus on how the performance of an optimal rendezvous maneuver with a non-cooperative target can be influenced by the accuracy reached in the pose and shape estimation process. To this purpose, the chaser, or maneuvering spacecraft is considered to be equipped with a single camera and a distance sensor, as this architecture both provides high accuracy and meets mass, power and volume requirements to be implemented on-board a small platform. An algorithm including the optimization process and a purposely built filter, capable of receiving images as inputs and providing evaluation of target's relative pose and shape as output, is implemented and tested in several different scenarios to validate the feasibility of the maneuver.

Infrared LEDs-Based Pose Estimation With Underground Camera Model for Boom-Type Roadheader in Coal Mining

Accurate and reliable pose estimation of a boom-type roadheader is of great importance in order to maintain the efficiency of automatic coal mining. The stability and accuracy of conventional measurement methods are difficult to be guaranteed on account of vibration noise, magnetic disturbance, electrostatic interference, and other factors in an underground environment. In this paper, a vision-based non-contact measurement method for cutting-head pose estimation is presented, which deploy a 16-point infrared LED target on the boom-type roadheader to tackle the low-illumination, high-dust, and complicated background. By establishing a monocular vision measurement system, the cutting-head pose is estimated through processing the LED target images obtained from an explosion-proof industrial camera mounted on the roadheader. After analyzing the measurement mechanism, an underground camera model based on the equivalent focal length is built to eliminate refraction errors caused by the two-layer glasses for explosion-proof and dust removal. Then, the pose estimation processes, including infrared LEDs feature point extraction, spot center location, and an improved P4P method based on dual quaternions, are carried out. The influence factors of cutting-head pose estimation accuracy are further studied by modeling, and the error distribution of the main parameters is investigated and evaluated. The numerical simulation and experimental evaluation are designed to verify the performance of the proposed method. The results show that the pose estimation error is in line with the numerical prediction, achieving the requirements of cutting-head pose estimation in underground roadway construction in the coal mine.