Optimal Pose Research Articles

Precision optimized pose and trajectory planning for vertically articulated robot arms

Vertically articulated robot arms are widely used in various industries for the realization of conventional handling tasks due to their high flexibility, good workspace-to-footprint ratio and low price. However, due to their kinematic structure, their accuracy, precision and stiffness are highly dependent on the robots configuration. Standards such as EN-ISO-9283 define procedures for determining accuracy and precision by averaging measurements at different points in the robot’s main workspace. While these averages can be used to estimate the robots overall performance, they do not support process developers in identifying precision optimal poses. This paper presents a generic and comprehensive routine for the combined identification of precision-optimal robot poses and trajectories for pick and place processes. The routine consists of several cascading optimization loops with the objectives of minimizing the structural and dimensional sensitivity costs of the trajectories’ start and end configurations, minimizing the number of actuated joints and minimizing the required direction changes of the joints in motion. Using a sampling-based trajectory planner (OMPL in MoveIt! / ROS), a smooth and collision free trajectory for an exemplary pick and place process is generated and experimentally validated for a KUKA LWR iiwa 14. The optimized trajectory is benchmarked against a conventional, unoptimized reference trajectory for the same handling task. The experimental validation shows a significant increase in the repeatability of the positioning of 36µm (at 3σ) compared to both the nominal repeatability of 150µm and the repeatability of the reference trajectory of 94µm (at 3σ, 2.6-fold increase). Transferring the developed routine to inherently more precise robot models will lead to highly precise vertically articulated robot arms capable of solving more demanding assembly tasks.

Robot End-Effector Mounted Camera Pose Optimization in Object Detection-Based Tasks

Robot End-Effector Mounted Camera Pose Optimization in Object Detection-Based Tasks

WPO-Net: Windowed Pose Optimization Network for Monocular Visual Odometry Estimation

Visual odometry is the process of estimating incremental localization of the camera in 3-dimensional space for autonomous driving. There have been new learning-based methods which do not require camera calibration and are robust to external noise. In this work, a new method that do not require camera calibration called the “windowed pose optimization network” is proposed to estimate the 6 degrees of freedom pose of a monocular camera. The architecture of the proposed network is based on supervised learning-based methods with feature encoder and pose regressor that takes multiple consecutive two grayscale image stacks at each step for training and enforces the composite pose constraints. The KITTI dataset is used to evaluate the performance of the proposed method. The proposed method yielded rotational error of 3.12 deg/100 m, and the training time is 41.32 ms, while inference time is 7.87 ms. Experiments demonstrate the competitive performance of the proposed method to other state-of-the-art related works which shows the novelty of the proposed technique.

Long term and robust 6DoF motion tracking for highly dynamic stereo endoscopy videos.

Real-time augmented reality (AR) for minimally invasive surgery without extra tracking devices is a valuable yet challenging task, especially considering dynamic surgery environments. Multiple different motions between target organs are induced by respiration, cardiac motion or operative tools, and often must be characterized by a moving, manually positioned endoscope. Therefore, a 6DoF motion tracking method that takes advantage of the latest 2D target tracking methods and non-linear pose optimization and tracking loss retrieval in SLAM technologies is proposed and can be embedded into such an AR system. Specifically, the SiamMask deep learning-based target tracking method is incorporated to roughly exclude motion distractions and enable frame matching. This algorithm's light computation cost makes it possible for the proposed method to run in real-time. A global map and a set of keyframes as in ORB-SLAM are maintained for pose optimization and tracking loss retrieval. The stereo matching and frame matching methods are improved and a new strategy to select reference frames is introduced to make the first-time motion estimation of every arriving frame as accurate as possible. Experiments on both a clinical laparoscopic partial nephrectomy dataset and an ex-vivo porcine kidney dataset are conducted. The results show that the proposed method gives a more robust and accurate performance compared with ORB-SLAM2 in the presence of motion distractions or motion blur; however, heavy smoke still remains a big factor that reduces the tracking accuracy.

An Enhanced Pedestrian Visual-Inertial SLAM System Aided with Vanishing Point in Indoor Environments.

The visual-inertial simultaneous localization and mapping (SLAM) is a feasible indoor positioning system that combines the visual SLAM with inertial navigation. There are accumulated drift errors in inertial navigation due to the state propagation and the bias of the inertial measurement unit (IMU) sensor. The visual-inertial SLAM can correct the drift errors via loop detection and local pose optimization. However, if the trajectory is not a closed loop, the drift error might not be significantly reduced. This paper presents a novel pedestrian dead reckoning (PDR)-aided visual-inertial SLAM, taking advantage of the enhanced vanishing point (VP) observation. The VP is integrated into the visual-inertial SLAM as an external observation without drift error to correct the system drift error. Additionally, the estimated trajectory’s scale is affected by the IMU measurement errors in visual-inertial SLAM. Pedestrian dead reckoning (PDR) velocity is employed to constrain the double integration result of acceleration measurement from the IMU. Furthermore, to enhance the proposed system’s robustness and the positioning accuracy, the local optimization based on the sliding window and the global optimization based on the segmentation window are proposed. A series of experiments are conducted using the public ADVIO dataset and a self-collected dataset to compare the proposed system with the visual-inertial SLAM. Finally, the results demonstrate that the proposed optimization method can effectively correct the accumulated drift error in the proposed visual-inertial SLAM system.

A Point Cloud Fusion Method for Space Target 3D Laser Point Cloud and Visible Light Image Reconstruction Point Cloud

In this paper, a high-precision spatial target lidar for 3D reconstruction of point cloud and visible light image 3D reconstruction point cloud fusion method is proposed. This fusion method uses the solved 3D reconstruction to invert the camera pose and 3D point cloud model centroid position optimization initial value selection. The registration accuracy and registration efficiency of the ICP algorithm are improved. At the same time, according to the characteristics of the two sets of point clouds, the Euclidean distance threshold is used to delete the noise points of the 3D point cloud edge, and the high-precision 3D reconstruction point with the scale information fusion is obtained. cloud. The simulation experiment of the spatial target simulation model shows that the fusion method can effectively improve the point cloud density, fill the reconstruction vulnerability, and improve the point cloud accuracy of the spatial target 3D reconstruction.

Using a Two-Stage Method to Reject False Loop Closures and Improve the Accuracy of Collaborative SLAM Systems

Loop-closure detection is an essential means to reduce accumulated errors of simultaneous localization and mapping (SLAM) systems. However, even false positive loop closures could seriously interfere and even corrupt the back-end optimization process. For a collaborative SLAM system that generally uses both intra-robot and inter-robot loop closures to optimize the pose graph, it is a tough job to reject those false positive loop closures without a reliable a priori knowledge of the relative pose transformation between robots. Aiming at this solving problem, this paper proposes a two-stage false positive loop-closure rejection method based on three types of consistency checks. Firstly, a multi-robot pose-graph optimization model is given which transforms the multi-robot pose optimization problem into a maximum likelihood estimation model. Then, the principle of the false positive loop-closure rejection method based on χ2 test is proposed, in which clustering is used to reject those intra-robot false loop-closures in the first step, and a largest mutually consistent loop-based χ2 test is constructed to reject inter-robot false loop closures in the second step. Finally, an open dataset and synthetic data are used to evaluate the performance of the algorithms. The experimental results demonstrate that our method improves the accuracy and robustness of the back-end pose-graph optimization with a strong ability to reject false positive loop closures, and it is not sensitive to the initial pose at the same time. In the Computer Science and Artificial Intelligence Lab (CSAIL) dataset, the absolute position error is reduced by 55.37% compared to the dynamic scaling covariance method, and the absolute rotation error is reduced by 77.27%; in the city10,000 synthetic dataset, the absolute position error is reduced by 89.37% compared to the pairwise consistency maximization (PCM) and the absolute rotation error is reduced by 97.9%.

Optimal Vehicle Pose Estimation Network Based on Time Series and Spatial Tightness with 3D LiDARs

Vehicle pose estimation is essential in autonomous vehicle (AV) perception technology. However, due to the different density distributions of the point cloud, it is challenging to achieve sensitive direction extraction based on 3D LiDAR by using the existing pose estimation methods. In this paper, an optimal vehicle pose estimation network based on time series and spatial tightness (TS-OVPE) is proposed. This network uses five pose estimation algorithms proposed as candidate solutions to select each obstacle vehicle’s optimal pose estimation result. Among these pose estimation algorithms, we first propose the Basic Line algorithm, which uses the road direction as the prior knowledge. Secondly, we propose improving principal component analysis based on point cloud distribution to conduct rotating principal component analysis (RPCA) and diagonal principal component analysis (DPCA) algorithms. Finally, we propose two global algorithms independent of the prior direction. We provided four evaluation indexes to transform each algorithm into a unified dimension. These evaluation indexes’ results were input into the ensemble learning network to obtain the optimal pose estimation results from the five proposed algorithms. The spatial dimension evaluation indexes reflected the tightness of the bounding box and the time dimension evaluation index reflected the coherence of the direction estimation. Since the network was indirectly trained through the evaluation index, it could be directly used on untrained LiDAR and showed a good pose estimation performance. Our approach was verified on the SemanticKITTI dataset and our urban environment dataset. Compared with the two mainstream algorithms, the polygon intersection over union (P-IoU) average increased by about 5.25% and 9.67%, the average heading error decreased by about 29.49% and 44.11%, and the average speed direction error decreased by about 3.85% and 46.70%. The experiment results showed that the ensemble learning network could effectively select the optimal pose estimation from the five abovementioned algorithms, making pose estimation more accurate.

Stereo Vision-Based Relative Position and Attitude Estimation of Non-Cooperative Spacecraft

In on-orbit services, the relative position and attitude estimation of non-cooperative spacecraft has become the key issues to be solved in many space missions. Because of the lack of prior knowledge about manual marks and the inability to communicate between non-cooperative space targets, the relative position and attitude estimation system poses great challenges in terms of accuracy, intelligence, and power consumptions. To address these issues, this study uses a stereo camera to extract the feature points of a non-cooperative spacecraft. Then, the 3D position of the feature points is calculated according to the camera model to estimate the relationship. The optical flow method is also used to obtain the geometric constraint information between frames. In addition, an extended Kalman filter is used to update the measurement results and obtain more accurate pose optimization results. Moreover, we present a closed-loop simulation system, in which the Unity simulation engine is employed to simulate the relative motion of the spacecraft and binocular vision images, and a JetsonTX2 supercomputer is involved to conduct the proposed autonomous relative navigation algorithm. The simulation results show that our approach can estimate the non-cooperative target’s relative pose accurately.

A fast and global search method for grasping pose optimization in manufacturing

To use the electromagnetic chuck to precisely absorb industrial parts in manufacturing, this paper presents a hybrid algorithm for grasping pose optimization, especially for the part with a large surface area and irregular shape. The hybrid algorithm is based on the Gaussian distribution sampling and the hybrid particle swarm optimization (PSO). The Gaussian distribution sampling based on the geometric center point is used to initialize the population, and the dynamic Alpha-stable mutation enhances the global optimization capability of the hybrid algorithm. Compared with other algorithms, the experimental results show that ours achieves the best results on the dataset presented in this work. Moreover, the time cost of the hybrid algorithm is near a fifth of the conventional PSO in the discovery of optimal grasping pose. In summary, the proposed algorithm satisfies the real-time requirements in industrial production and still has the highest success rate, which has been deployed on the actual production line of SANY Group.

ROSEFusion

Online reconstruction based on RGB-D sequences has thus far been restrained to relatively slow camera motions (<1m/s). Under very fast camera motion (e.g., 3m/s), the reconstruction can easily crumble even for the state-of-the-art methods. Fast motion brings two challenges to depth fusion: 1) the high nonlinearity of camera pose optimization due to large inter-frame rotations and 2) the lack of reliably trackable features due to motion blur. We propose to tackle the difficulties of fast-motion camera tracking in the absence of inertial measurements using random optimization, in particular, the Particle Filter Optimization (PFO). To surmount the computation-intensive particle sampling and update in standard PFO, we propose to accelerate the randomized search via updating a particle swarm template (PST). PST is a set of particles pre-sampled uniformly within the unit sphere in the 6D space of camera pose. Through moving and rescaling the pre-sampled PST guided by swarm intelligence, our method is able to drive tens of thousands of particles to locate and cover a good local optimum extremely fast and robustly. The particles, representing candidate poses, are evaluated with a fitness function defined based on depth-model conformance. Therefore, our method, being depth-only and correspondence-free, mitigates the motion blur impediment as (ToF-based) depths are often resilient to motion blur. Thanks to the efficient template-based particle set evolution and the effective fitness function, our method attains good quality pose tracking under fast camera motion (up to 4m/s) in a realtime framerate without including loop closure or global pose optimization. Through extensive evaluations on public datasets of RGB-D sequences, especially on a newly proposed benchmark of fast camera motion, we demonstrate the significant advantage of our method over the state of the arts.

Robotic-arm-assisted flexible large field-of-view optical coherence tomography

Optical coherence tomography (OCT) is a three-dimensional non-invasive high-resolution imaging modality that has been widely used for applications ranging from medical diagnosis to industrial inspection. Common OCT systems are equipped with limited field-of-view (FOV) in both the axial depth direction (a few millimeters) and lateral direction (a few centimeters), prohibiting their applications for samples with large and irregular surface profiles. Image stitching techniques exist but are often limited to at most 3 degrees-of-freedom (DOF) scanning. In this work, we propose a robotic-arm-assisted OCT system with 7 DOF for flexible large FOV 3D imaging. The system consists of a depth camera, a robotic arm and a miniature OCT probe with an integrated RGB camera. The depth camera is used to get the spatial information of targeted sample at large scale while the RGB camera is used to obtain the exact position of target to align the image probe. Eventually, the real-time 3D OCT imaging is used to resolve the relative pose of the probe to the sample and as a feedback for imaging pose optimization when necessary. Flexible probe pose manipulation is enabled by the 7 DOF robotic arm. We demonstrate a prototype system and present experimental results with flexible tens of times enlarged FOV for plastic tube, phantom human finger, and letter stamps. It is expected that robotic-arm-assisted flexible large FOV OCT imaging will benefit a wide range of biomedical, industrial and other scientific applications.

Cooperative indoor 3D mapping and modeling using LiDAR data

Point clouds and models with semantic information facilitate various indoor automation, ranging from indoor robotics to emergency responses. Studies are currently being conducted on semantic labeling and modeling based on offline mapped point clouds, in which, the performance is strongly limited by the mapping process. To address this issue, we propose a framework to cooperatively perform the three tasks of semantic labeling, mapping, and 3D modeling of point clouds. First, our framework uses a deep-learning-assisted method to perform frame-level point cloud semantic labeling. Subsequently, point cloud frames with semantic labels are used to extract the structural planes of buildings, followed by the generation of line structures from the planes. Then, these frames are used to estimate the initial poses of a 3D sensor for data collection. In the subsequent pose optimization process, the initial poses are optimized under the constraints of the structural planes. Finally, the optimized poses are used to integrate semantic frames and line structures to generate a point cloud map and 3D line model of buildings. The experimental results show that the proposed method achieves better results than the state-of-the-art methods that separately perform one of the two tasks.

Characterizing dynamic deformation of marine propeller blades with stroboscopic stereo digital image correlation

Stereo digital image correlation (stereo-DIC) technique shows a powerful capacity on full-field three-dimensional (3D) deformation measurement of rotary machine structures. However, it remains a challenge for underwater applications due to difficulties in imaging, stereo calibration, light refraction, and so on. In this paper, its potential in characterizing 3D dynamic deformation of the underwater rotor blades is shown by solving these problems. A stroboscopic stereo-DIC system is established to capture clear speckle images of a blade that rotates in the windowed cavitation tunnel under different flow speeds. To calibrate the stereo camera for underwater object measurement, an improved planar pattern-based calibration method and a globally optimal relative pose estimation algorithm are proposed to calibrate the intrinsic and extrinsic parameters of the stereo-DIC system separately. In particular, a novel refractive 3D reconstruction method for underwater objects is presented to recover the true 3D shape according to flat refraction geometry, ensuring the correctness and reliability of the measured 3D displacement fields. Several experiments demonstrated that the proposed stereo-DIC system and methods are feasible and accurate. Based on this, the measured dynamic displacement fields of a propeller blade are presented and discussed. Results herald a possibility for monitoring the full-field 3D dynamic response and structural health of underwater rotating structures by the proposed technique.

A generic algorithm for computing optimal ergonomic postures during working in an industrial environment

The present study tries to decrease the risk of work-related musculoskeletal disorders for industry workers by proposing a generic algorithm that recommends an optimal ergonomic posture for accomplishing tasks in an industrial environment. In the case of a dangerous ergonomic pose, the optimization algorithm starts by heuristically changing it to a more ergonomic one. Each recommended posture's feasibility is tested with an inverse kinematic method that can predict the worker's behavior for accomplishing a task. This iterative optimization procedure continues until the optimal ergonomic pose for the worker is achieved. The algorithm's validity is tested in thirteen cases, people with different gender (50 percent male, 50 percent female) aged between 20 and 35, and different height and body morphologies. According to studies, there is a connection between musculoskeletal disorders and the wrong posture for accomplishing tasks in industries. We suggest an optimization algorithm that can indicate the worker the optimal ergonomic pose by considering task constraints in real-time.

DiSCO: Differentiable Scan Context With Orientation

Global localization is essential for robot navigation, of which the first step is to retrieve a query from the map database. This problem is called place recognition. In recent years, LiDAR scan based place recognition has drawn attention as it is robust against the appearance change. In this letter, we propose a LiDAR-based place recognition method, named Differentiable Scan Context with Orientation (DiSCO), which simultaneously finds the scan at a similar place and estimates their relative orientation. The orientation can further be used as the initial value for the down-stream local optimal metric pose estimation, improving the pose estimation especially when a large orientation between the current scan and retrieved scan exists. Our key idea is to transform the feature into the frequency domain. We utilize the magnitude of the spectrum as the place descriptor, which is theoretically rotation-invariant. In addition, based on the differentiable phase correlation, we can efficiently estimate the global optimal relative orientation using the spectrum. With such structural constraints, the network can be learned in an end-to-end manner, and the backbone is fully shared by the two tasks, achieving better interpretability and lightweight. Finally, DiSCO is validated on three datasets with long-term outdoor conditions, showing better performance than the compared methods. Codes are released at https://github.com/MaverickPeter/DiSCO-pytorch.

Musculoskeletal modelling of the Nile crocodile (Crocodylus niloticus) hindlimb: Effects of limb posture on leverage during terrestrial locomotion.

We developed a three‐dimensional, computational biomechanical model of a juvenile Nile crocodile (Crocodylus niloticus) pelvis and hindlimb, composed of 47 pelvic limb muscles, to investigate muscle function. We tested whether crocodiles, which are known to use a variety of limb postures during movement, use limb orientations (joint angles) that optimise the moment arms (leverages) or moment‐generating capacities of their muscles during different limb postures ranging from a high walk to a sprawling motion. We also describe the three‐dimensional (3D) kinematics of the crocodylian hindlimb during terrestrial locomotion across an instrumented walkway and a treadmill captured via X‐ray Reconstruction of Moving Morphology (biplanar fluoroscopy; ‘XROMM’). We reconstructed the 3D positions and orientations of each of the hindlimb bones and used dissection data for muscle lines of action to reconstruct a focal, subject‐specific 3D musculoskeletal model. Motion data for different styles of walking (a high, crouched, bended and two types of sprawling motion) were fed into the 3D model to identify whether any joints adopted near‐optimal poses for leverage across each of the behaviours. We found that (1) the hip adductors and knee extensors had their largest leverages during sprawling postures and (2) more erect postures typically involved greater peak moment arms about the hip (flexion‐extension), knee (flexion) and metatarsophalangeal (flexion) joints. The results did not fully support the hypothesis that optimal poses are present during different locomotory behaviours because the peak capacities were not always reached around mid‐stance phase. Furthermore, we obtained few clear trends for isometric moment‐generating capacities. Therefore, perhaps peak muscular leverage in Nile crocodiles is instead reached either in early/late stance or possibly during swing phase or other locomotory behaviours that were not studied here, such as non‐terrestrial movement. Alternatively, our findings could reflect a trade‐off between having to execute different postures, meaning that hindlimb muscle leverage is not optimised for any singular posture or behaviour. Our model, however, provides a comprehensive set of 3D estimates of muscle actions in extant crocodiles which can form a basis for investigating muscle function in extinct archosaurs.

A Closed-Form Solution for Estimating the Accuracy of Circular Feature’ Pose for Object 2D-3D Pose Estimation System

Many objects in the real world have circular feature. In general, circular feature’s pose is represented by 5-DoF (degree of freedom) vector ξ = X , Y , Z , α , β T . It is a difficult task to measure the accuracy of circular feature’s pose in each direction and the correlation between each direction. This paper proposes a closed-form solution for estimating the accuracy of pose transformation of circular feature. The covariance matrix of ξ is used to measure the accuracy of the pose. The relationship between the pose of the circular feature of 3D object and the 2D points is analyzed to yield an implicit function, and then Gauss–Newton theorem is employed to compute the partial derivatives of the function with respect to such point, and after that the covariance matrix is computed from both the 2D points and the extraction error. In addition, the method utilizes the covariance matrix of 5-DoF circular feature’s pose variables to optimize the pose estimator. Based on pose covariance, minimize the mean square error (Min-MSE) metric is introduced to guide good 2D imaging point selection, and the total amount of noise introduced into the pose estimator can be reduced. This work provides an accuracy method for object 2D-3D pose estimation using circular feature. At last, the effectiveness of the method for estimating the accuracy is validated based on both random data sets and synthetic images. Various synthetic image sequences are illustrated to show the performance and advantages of the proposed pose optimization method for estimating circular feature’s pose.

Multi-view 2D–3D alignment with hybrid bundle adjustment for visual metrology

High-precision measurement based on multi-view geometry benefits from aligning a template CAD model to the multi-view observations of a target object. It not only improves the multi-camera calibration but also assists the measurement by serving as a “scaffold.” A straightforward approach is to reconstruct the target object and perform a 3D registration with the CAD model. However, the accuracy of such 3D alignment cannot meet the high-precision requirement. We formulate the problem as a bundle adjustment where we jointly optimize the 6DoF poses of both the template model and the multiple cameras. To accommodate the manufacturing error of products, we propose a simple and robust solution based on a discrete–continuous optimization which interleaves between correspondence selection and pose optimization. In the discrete step, a robust RANSAC-based selection process selects well-matched 2D–3D feature points according to the current model/camera poses. In the continuous step, it jointly optimizes the 6D poses of the CAD model and the multiple cameras. This interleaving optimization constitutes a novel hybrid bundle adjustment (HBA). In HBA, the reprojection error of a feature point is measured either with the 2D–3D correspondence between the observation image and the CAD model or with the cross-view correspondence between multiple images, whichever is more reliable according to the discrete selection step. Through extensive evaluation on real-world data, we demonstrate that HBA achieves high-precision multi-camera calibration, outperforming alternative approaches significantly.

GOLEM: Automated and Robust Cryo-EM-Guided Ligand Docking with Explicit Water Molecules

A detailed understanding of ligand-protein interactions underpins rational drug-design strategies. In recent years, technological advances in cryo-electron microscopy (cryo-EM) brought a new era for the structural determination of biological macromolecules and assemblies at a high resolution, marking cryo-EM a promising tool for studying ligand-protein interactions. However, even in high-resolution cryo-EM structures, the bound ligand's densities are often of lower quality due to its dynamic nature, frustrating an accurate coordinate assignment. Computational methods have been offering complementary efforts to aid structure modeling, such as MDFF (molecular dynamics flexible fitting) to fit atomic structures of macromolecules into corresponding cryo-EM densities. To address the challenge of ligand modeling in cryo-EM maps, here we report the development and validation of GOLEM (Genetic Optimization of Ligand in Experimental Maps), an automated and robust ligand docking method that models the ligand's optimal pose and conformation in cryo-EM maps. GOLEM employs a Lamarckian genetic algorithm, which performs a hybrid global/local search to explore the ligand's conformational, orientational, and positional space, with explicit consideration of water displacement and bridging water molecules' position and orientation. GOLEM takes into account both energetics and the correlation with the electron density maps in its scoring function, which is a combination of the system's energy and the energy of the ligand in a cryo-EM-derived external potential. GOLEM is validated in multiple cryo-EM structures with a wide range of map resolutions and ligand types, producing ligand poses in excellent agreement with the densities. As a VMD plugin, GOLEM is free-of-charge and accessible to the community. With these features, we believe GOLEM will be a valuable tool for ligand modeling in cryo-EM efforts towards drug discoveries.