Real-time Pose Estimation Research Articles

$\mathcal {PLC}$-LiSLAM: LiDAR SLAM With Planes, Lines, and Cylinders

Planes, lines, and cylinders widely exist in man-made environments. This letter introduces a LiDAR simultaneous localization and mapping (SLAM) system using those three types of landmarks. Our algorithm has three components including local mapping, global mapping, and localization. The local and global mapping jointly adjust planes, lines, and cylinders with LiDAR poses to minimize the point-to-model cost, which is referred to as plane-line-cylinder adjustment (PLCA). We prove that, with some preprocessing, PLCA is independent of the number of points captured from the three types of landmarks, which makes efficiently solving a large-scale PLCA problem feasible. The localization component conducts real-time pose estimation through registering local planes, lines, and cylinders to the global ones, which is referred to as plane-line-cylinder registration (PLCR). We present an efficient solution for PLCR. The detection and data association may introduce errors. We correct these errors through checking the cost in the back-end. It is difficult for the registration-based algorithm, such as LOAM and ICP, to correct these errors, as they do not maintain the data association. Experimental results show that out algorithm outperforms the state-of-the-art LiDAR SLAM algorithms and achieves real-time performance.

DOPE++: 6D pose estimation algorithm for weakly textured objects based on deep neural networks.

This paper focuses on 6D pose estimation for weakly textured targets from RGB-D images. A 6D pose estimation algorithm (DOPE++) based on a deep neural network for weakly textured objects is proposed to solve the poor real-time pose estimation and low recognition efficiency in the robot grasping process of parts with weak texture. More specifically, we first introduce the depthwise separable convolution operation to lighten the original deep object pose estimation (DOPE) network structure to improve the network operation speed. Second, an attention mechanism is introduced to improve network accuracy. In response to the low recognition efficiency of the original DOPE network for parts with occlusion relationships and the false recognition problem in recognizing parts with scales that are too large or too small, a random mask local processing method and a multiscale fusion pose estimation module are proposed. The results show that our proposed DOPE++ network improves the real-time performance of 6D pose estimation and enhances the recognition of parts at different scales without loss of accuracy. To address the problem of a single background representation of the part pose estimation dataset, a virtual dataset is constructed for data expansion to form a hybrid dataset.

Visual Positioning System Based on 6D Object Pose Estimation Using Mobile Web

Recently, the demand for location-based services using mobile devices in indoor spaces without a global positioning system (GPS) has increased. However, to the best of our knowledge, solutions that are fully applicable to indoor positioning and navigation and ensure real-time mobility on mobile devices, such as global navigation satellite system (GNSS) solutions, cannot achieve remarkable researches in indoor circumstances. Indoor single-shot image positioning using smartphone cameras does not require a dedicated infrastructure and offers the advantages of low price and large potential markets owing to the popularization of smartphones. However, existing methods or systems based on smartphone cameras and image algorithms encounter various limitations when implemented in indoor environments. To address this, we designed an indoor visual positioning system for mobile devices that can locate users in indoor scenes. The proposed method uses a smartphone camera to detect objects through a single image in a web environment and calculates the location of the smartphone to find users in an indoor space. The system is inexpensive because it integrates deep learning and computer vision algorithms and does not require additional infrastructure. We present a novel method of detecting 3D model objects from single-shot RGB data, estimating the 6D pose and position of the camera and correcting errors based on voxels. To this end, the popular convolutional neural network (CNN) is improved by real-time pose estimation to handle the entire 6D pose estimate the location and direction of the camera. The estimated position of the camera is addressed to a voxel to determine a stable user position. Our VPS system provides the user with indoor information in 3D AR model. The voxel address optimization approach with camera 6D position estimation using RGB images in a mobile web environment outperforms real-time performance and accuracy compared to current state-of-the-art methods using RGB depth or point cloud.

RPEOD: A Real-Time Pose Estimation and Object Detection System for Aerial Robot Target Tracking

Pose estimation and environmental perception are the fundamental capabilities of autonomous robots. In this paper, a novel real-time pose estimation and object detection (RPEOD) strategy for aerial robot target tracking is presented. The aerial robot is equipped with a binocular fisheye camera for pose estimation and a depth camera to capture the spatial position of the tracked target. The RPEOD system uses a sparse optical flow algorithm to track image corner features, and the local bundle adjustment is restricted in a sliding window. Ulteriorly, we proposed YZNet, a lightweight neural inference structure, and took it as the backbone in YOLOV5 (the state-of-the-art real-time object detector). The RPEOD system can dramatically reduce the computational complexity in reprojection error minimization and the neural network inference process; Thus, it can calculate real-time on the onboard computer carried by the aerial robot. The RPEOD system is evaluated using both simulated and real-world experiments, demonstrating clear advantages over state-of-the-art approaches, and is significantly more fast.

A factor graph optimization mapping based on normaldistributions transform

This paper aims to achieve highly accurate mapping results and real time pose estimation of autonomous vehicle by using the normal distribution transform (NDT) algoritm. A factor graph optimization algorithm (FGO-NDT) is proposed to address the poor real-time performance and pose drift errors of the NDT algorithm. Smooth point cloud data are obtained by multisensor calibration and data preprocessing. NDT registration is then used for lidar odometry and feature matching. The global navigation satellite system (GNSS) data and loop detection results are added to the factor graph framework as the pose constraint factors to optimize the pose trajectory and eliminate the pose drift error generated during mapping. In addition, a sliding window method is used for map registration to extract a local map to shorten the map loading time. Thus, the real-time performance of creating point cloud maps of large scenes is significantly improved. Several experiments are conducted in different environments to verify the accuracy and performance of the FGO-NDT. The experimental results demonstrate that the proposed method can eliminate the pose estimation error caused by drift, improve the local structure, and reduce and root mean square error.

Faster than Real-Time Surface Pose Estimation with Application to Autonomous Robotic Grasping

Motivated by grasp planning applications within cluttered environments, this paper presents a novel approach to performing real-time surface segmentations of never-before-seen objects scattered across a given scene. This approach utilizes an input 2D depth map, where a first principles-based algorithm is utilized to exploit the fact that continuous surfaces are bounded by contours of high gradient. From these regions, the associated object surfaces can be isolated and further adapted for grasp planning. This paper also provides details for extracting the six-DOF pose for an isolated surface and presents the case of leveraging such a pose to execute planar grasping to achieve both force and torque closure. As a consequence of the highly parallel software implementation, the algorithm is shown to outperform prior approaches across all notable metrics and is also shown to be invariant to object rotation, scale, orientation relative to other objects, clutter, and varying degree of noise. This allows for a robust set of operations that could be applied to many areas of robotics research. The algorithm is faster than real time in the sense that it is nearly two times faster than the sensor rate of 30 fps.

LiDAR-odometry based UAV pose estimation in young forest environment

In this study, we propose a real-time pose estimation solution for Unmanned Aerial Vehicle in a seedling pine forest environment. Our method uses graph-based approach to fuse data from an onboard IMU sensors, a GNSS receiver and a 3D LiDAR. Features are detected from every LiDAR scan. A local map is built from the detected features and used to compute the LiDAR odometry in real time for the incoming scans. In order to obtain a robust estimate of the state of the vehicle, the noise covariance of the LiDAR odometry is updated at each iteration using the fitness score of the LiDAR. The proposed solution provides promising trajectory and velocity estimates even in GNSS denied scenario. Both the local and global consistencies of the estimated trajectory are encouraging.

Robust Pose Estimation via Hybrid Point and Twin Line Reprojection for RGB-D Vision Navigation

Pose estimation is a crucial technique to achieve robot localization using natural features for an RGB-D vision navigation system. However, it still suffers from the impoverishment of robustness and accuracy in complex workspaces. A robust real-time pose estimation approach using hybrid point and twin line reprojection is proposed to allow accurate localization for a vision-based robot under various challenging scenarios including low-texture, low-structure, occlusion, and viewpoint and illumination changes. Firstly, an improved adaptive threshold (IAT)-based oriented fast and rotated Brief (ORB) feature detection method is developed to extract sufficient IAT-ORB point features, which are matched by a multi-stage KNN-RTC-CS-RANSAC (KRCR) refinement feature matching strategy to generate reliable point matches. Then a robust EDLines line feature matching method is developed by integrating dynamic matching (DM) and median absolute deviation (MAD) to obtain fine line matches. Furthermore, the hybrid point reprojection errors are built from the fusion of 3D-2D and 3D-3D point pairs, and the twin line reprojection errors are constructed by integrating virtual right-eye lines generated by depth measurement, which both are incorporated into a novel unified errors optimization model to achieve optimal motion estimation by minimizing the reprojection errors of points and lines concurrently. Finally, the extensive experiments verify the robustness, accuracy, and real-time performance of the proposed approach on the public datasets and our real indoor scenarios. The quantitative results demonstrate that our proposed approach gains more accurate and stable localization than other state-of-the-art methods under challenging environments.

Yoga Pose Perfection using Deep Learning

Abstract
 Even with lots of attention and work in the computer vision and artificial intelligence field, human body pose detection is still a daunting task. The application of human pose detection is wide-ranging from health monitoring to public security. This paper focuses on the application in yoga, an art that has been performed for over a millennium. In modern society yoga has become a common method of exercise and there-in arises a demand for instructions on how to do yoga properly. Doing certain yoga postures improperly may lead to injuries and fatigue and hence the presence of a trainer becomes important. As many people don’t have the resources to have a yoga instructor or guide, artificial intelligence can act as a substitute and advise people on their poses. Currently, the research surrounding pose estimation for yoga mainly discusses the classification of yogic poses. In this work, we propose a method, using the Tensorflow MoveNet Thunder model, that allows real-time pose estimation to detect the error in a person's pose, thereby allowing them to correct it.

Towards Real-time pose estimation of the Mitral Valve Robot under C-arm X-ray Fluoroscopy.

Mitral regurgitation (MR) is a condition caused by a deformity in the mitral valve leading to the backflow of blood into the left atrium. MR can be treated through a minimally invasive procedure and our lab is currently developing a robot that could potentially be used to treat MR. The robot would carry a clip that latches onto the valve's leaflets and closes them to minimize leakage. The robot's accurate localization is needed to navigate the clip to the leaflets successfully. This paper discusses algorithms used to track the clip's position and orientation under real-time using C-arm fluoroscopy. The positions are found through a deep learning semantic segmentation framework and the pose is found by calculating its bending and rotational angles. The robot's bending angle and the clip's rotational angle is found through an equivalent ellipse algorithm and an SVM classifier, respectively, and were validated by comparing orientations obtained from an electromagnetic tracker. The bending angle calculation has an average error of 7.7° and the rotational angle calculation is 76% for classifying them into five classes. Execution times are within 100ms and hence this could be a promising approach in real-time pose estimation.

An RGB-D Camera Based Visual Positioning System for Assistive Navigation by a Robotic Navigation Aid

There are about 253 million people with visual impairment worldwide. Many of them use a white cane and/or a guide dog as the mobility tool for daily travel. Despite decades of efforts, electronic navigation aid that can replace white cane is still research in progress. In this paper, we propose an RGB-D camera based visual positioning system (VPS) for real-time localization of a robotic navigation aid (RNA) in an architectural floor plan for assistive navigation. The core of the system is the combination of a new 6-DOF depth-enhanced visual-inertial odometry (DVIO) method and a particle filter localization (PFL) method. DVIO estimates RNA's pose by using the data from an RGB-D camera and an inertial measurement unit (IMU). It extracts the floor plane from the camera's depth data and tightly couples the floor plane, the visual features (with and without depth data), and the IMU's inertial data in a graph optimization framework to estimate the device's 6-DOF pose. Due to the use of the floor plane and depth data from the RGB-D camera, DVIO has a better pose estimation accuracy than the conventional VIO method. To reduce the accumulated pose error of DVIO for navigation in a large indoor space, we developed the PFL method to locate RNA in the floor plan. PFL leverages geometric information of the architectural CAD drawing of an indoor space to further reduce the error of the DVIO-estimated pose. Based on VPS, an assistive navigation system is developed for the RNA prototype to assist a visually impaired person in navigating a large indoor space. Experimental results demonstrate that: 1) DVIO method achieves better pose estimation accuracy than the state-of-the-art VIO method and performs real-time pose estimation (18 Hz pose update rate) on a UP Board computer; 2) PFL reduces the DVIO-accrued pose error by 82.5% on average and allows for accurate wayfinding (endpoint position error ≤ 45 cm) in large indoor spaces.

Smart healthcare system-a brain-like computing approach for analyzing the performance of detectron2 and PoseNet models for anomalous action detection in aged people with movement impairments

In this era of artificial intelligence, a wide variety of techniques are available in healthcare industry especially to study about various changes happening in the human body. Intelligent assistance using brain-like framework helps to understand and analyze various types of complex data by utilizing most recent innovations such as deep learning and computer vision. Activities are complex practices, including continuous actions as well as interleaved actions that could be processed with fully interconnected neuron-like processing machine in a way the human brain works. Human postures have the ability to express different body movements in different environments. An optimal method is required to identify and analyze different kinds of postures so that the recognition rate has to be increased. The system should handle ambiguous circumstances that include diverse body movements, multiple views and changes in the environments. The objective of this research is to apply real-time pose estimation models for object detection and abnormal activity recognition with vision-based complex key point analysis. Object detection based on bounding box with a mask is successfully implemented with detectron2 deep learning model. Using PoseNet model, normal and abnormal activities are successfully distinguished, and the performance is evaluated. The proposed system implemented a state of the art computing model for the development of public healthcare industry. The experimental results show that the models have high levels of accuracy for detecting sudden changes in movements under varying environments.

DeepLabStream enables closed-loop behavioral experiments using deep learning-based markerless, real-time posture detection

In general, animal behavior can be described as the neuronal-driven sequence of reoccurring postures through time. Most of the available current technologies focus on offline pose estimation with high spatiotemporal resolution. However, to correlate behavior with neuronal activity it is often necessary to detect and react online to behavioral expressions. Here we present DeepLabStream, a versatile closed-loop tool providing real-time pose estimation to deliver posture dependent stimulations. DeepLabStream has a temporal resolution in the millisecond range, can utilize different input, as well as output devices and can be tailored to multiple experimental designs. We employ DeepLabStream to semi-autonomously run a second-order olfactory conditioning task with freely moving mice and optogenetically label neuronal ensembles active during specific head directions.

PVNet: Pixel-Wise Voting Network for 6DoF Object Pose Estimation.

This paper addresses the problem of instance-level 6DoF object pose estimation from a single RGB image. Many recent works have shown that a two-stage approach, which first detects keypoints and then solves a Perspective-n-Point (PnP) problem for pose estimation, achieves remarkable performance. However, most of these methods only localize a set of sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion and truncation. Instead, we introduce a Pixel-wise Voting Network (PVNet) to regress pixel-wise vectors pointing to the keypoints and use these vectors to vote for keypoint locations. This creates a flexible representation for localizing occluded or truncated keypoints. Another important feature of this representation is that it provides uncertainties of keypoint locations that can be further leveraged by the PnP solver. Experiments show that the proposed approach outperforms the state of the art on the LINEMOD, Occluded LINEMOD, YCB-Video, and Tless datasets, while being efficient for real-time pose estimation. We further create a Truncated LINEMOD dataset to validate the robustness of our approach against truncation. The code is available at https://github.com/zju3dv/pvnet.

Real-time, low-latency closed-loop feedback using markerless posture tracking.

The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here, we provide a new <monospace>DeepLabCut-Live!</monospace> package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called <monospace>DLC-Live! GUI</monospace>), and integration into (2) <monospace>Bonsai,</monospace> and (3) <monospace>AutoPilot</monospace>. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.

TRAJECTORY SIMULATION OF BADMINTON ROBOT BASED ON FRACTAL BROWN MOTION

This paper focuses on the design of badminton robots, and designs high-precision binocular stereo vision synchronous acquisition system hardware and multithreaded acquisition programs to ensure the left and right camera exposure synchronization and timely reading of data. Aiming at specific weak moving targets, a shape-based Brown motion model based on dynamic threshold adjustment based on singular value decomposition is proposed, and a discriminative threshold is set according to the similarity between the background and the foreground to improve detection accuracy. The three-dimensional trajectory points are extended by Kalman filter and the kinematics equation of badminton is established. The parameters of the kinematics equation of badminton are solved by the method of least squares. Based on the fractal Brownian motion algorithm, a real-time robot pose estimation algorithm is proposed to realize the real-time accurate pose estimation of the robot. A PID control model for the badminton robot executive mechanism is established between the omnidirectional wheel speed and the robot’s translation and rotation movements to achieve the precise movement of the badminton robot. All the algorithms can meet the system’s requirements for real-time performance, realize the badminton robot’s simple hit to the ball, and prospect the future research direction.

Smart Body Posture Guidance System

Body posture plays a crucial role in the measurement of body vitals. Accurate measurement of blood pressure, body weight, etc. requires correct posture which greatly affects the result. To achieve the correct posture the person has to depend on the lab technician, who helps the person to sit in the correct pose, to place the cuff in the right place to measure blood pressure, to stand in the correct pose to measure body weight, etc. To eliminate the need for lab technician, we develop a system, to capture image and recognize the pose using a camera, image processing using pose estimation algorithm, guide the person to correct if the recognized pose is incorrect using a User Interface developed using pyqt. The pose estimation algorithm works by estimating and connecting the body keypoints. Hence we are creating a prototype model for real time pose estimation and guidance.

3-D Human Pose Estimation Using Iterative Conditional Squeeze and Excitation Networks.

We propose a new method for single-camera real-world 3-D human pose estimation. Our method uses multitask training together with iterative pose refinement using a novel conditional attention mechanism. For iterative pose refinement, the output of each convolutional layer is conditioned on the latest pose estimate, using a conditioned squeeze-and-excitation network architecture that incorporates novel feedback connections. Multitask training on both an in-the-wild 2-D pose dataset and a controlled 3-D pose dataset allows for real-world 3-D pose estimation without the need for a large-scale in-the-wild 3-D pose dataset, which is unavailable. Experiments are performed on several real-world datasets, as well as the Human 3.6 Million and HumanEva-I datasets, to show that the combined attention mechanism, iterative refinement scheme, and multitask training allow us to achieve robust and competitive performance with only a simple network architecture. In addition, we show that our method is efficient enough to run on commodity hardware, producing pose estimates in real time.

Real-time 3D Pose Estimation by RGB-D Camera and Its Evaluation

Real-time 3D Pose Estimation by RGB-D Camera and Its Evaluation

POP: A Generic Framework for Real-Time Pose Estimation of Planar Objects

Accurate pose estimation of planar objects is a key computation in visual localization tasks, with recent studies showing remarkable progress on a handful of baseline datasets. Nonetheless, achieving similar performance on sequences in unconstrained environments is still an ongoing quest to be accomplished, largely due to the existence of several sources of errors, which are correlated but often only partly tackled in the literature. In this article, we propose POP, a generic real-time planar-object pose-estimation framework which is designed to handle the aforementioned types of errors while not losing generality to a specific choice of keypoint detection or tracking algorithm. The essence of POP lies in activating keypoint detection module in the background as well as adding several refinement steps in order to reduce correlated sources of errors within the pipeline. We provide extensive experimental evaluations against state-of-the-art planar object tracking algorithms on baseline and more challenging datasets, empirically demonstrating the effectiveness of the POP framework for scenes with large environmental variations.