Object Keypoints Research Articles

Multi-Modal Pose Representations for 6-DOF Object Tracking

Pose estimation methods for robotics should return a distribution of poses rather than just a single pose estimate. Motivated by this, in this work we investigate multi-modal pose representations for reliable 6-DoF object tracking. A neural network architecture for simultaneous object segmentation and estimation of fiducial points of the object on RGB images is proposed. Given a priori probability distribution of object poses a particle filter is employed to estimate the posterior probability distribution of object poses. An advanced observation model relying on matching the projected 3D model with the segmented object and a distance transform-based object representation is used to weight samples representing the probability distribution. Afterwards, the object pose determined by the PnP algorithm is included in the probability distribution via replacing a particle with the smallest weight. Next, a k-means++ algorithm is executed to determine modes in a multi-modal probability distribution. A multi-swarm particle swarm optimization is then executed to determine the finest modes in the probability distribution. A subset of particles for final pose optimization is found in a multi-criteria analysis using the TOPSIS algorithm. They are verified using conflicting criteria that are determined on the basis of object keypoints, segmented object, and the distance transform. On the challenging YCB-Video dataset it outperforms recent algorithms for both object pose estimation and object pose tracking.

Trajectory Recognition and Working Condition Analysis of Rod Pumping Systems Based on Pose Estimation Method with Heatmap-Free Joint Detection

Summary Rod pump systems are the primary production tools in oilfield development. Analyzing their working conditions provides a theoretical foundation for formulating production optimization plans and adjusting equipment parameters. Existing machine learning–based condition analysis methods rely on dynamometer cards and cannot capture the actual operational trajectory of the pumping unit. To address this issue, this paper proposes a keypoint detection method for pumping units based on pose estimation of heatmap-free joint detection from video data. A data annotation scheme suitable for the task of detecting pumping unit keypoints was developed, and the learning criteria for this task were optimized. An end-to-end heatmap-free pose estimation algorithm was used to process images of the pumping unit, yielding predicted keypoint positions, thereby enabling the identification of the keypoint motion trajectories of the pumping unit. Experiments validated the proposed method and compared it with general learning criteria. Results show that this method accurately captures the keypoint positions of the pumping unit, with the optimized learning criteria model improving by more than 5% compared with general methods and increasing the keypoint object keypoint similarity (OKS) by more than 30%. The model’s results can be used for the actual operational trajectory recognition of the pumping unit, automatically calculating the motion parameters of the polished rod, and intelligently assessing the balance and working condition analysis of the pumping unit. This realizes the intelligent application of video surveillance data, significantly contributing to the dynamic study of rod pump systems.

Automatic Perception of Typical Abnormal Situations in Cage-Reared Ducks Using Computer Vision.

Overturning and death are common abnormalities in cage-reared ducks. To achieve timely and accurate detection, this study focused on 10-day-old cage-reared ducks, which are prone to these conditions, and established prior data on such situations. Using the original YOLOv8 as the base network, multiple GAM attention mechanisms were embedded into the feature fusion part (neck) to enhance the network's focus on the abnormal regions in images of cage-reared ducks. Additionally, the Wise-IoU loss function replaced the CIoU loss function by employing a dynamic non-monotonic focusing mechanism to balance the data samples and mitigate excessive penalties from geometric parameters in the model. The image brightness was adjusted by factors of 0.85 and 1.25, and mainstream object-detection algorithms were adopted to test and compare the generalization and performance of the proposed method. Based on six key points around the head, beak, chest, tail, left foot, and right foot of cage-reared ducks, the body structure of the abnormal ducks was refined. Accurate estimation of the overturning and dead postures was achieved using the HRNet-48. The results demonstrated that the proposed method accurately recognized these states, achieving a mean Average Precision (mAP) value of 0.924, which was 1.65% higher than that of the original YOLOv8. The method effectively addressed the recognition interference caused by lighting differences, and exhibited an excellent generalization ability and comprehensive detection performance. Furthermore, the proposed abnormal cage-reared duck pose-estimation model achieved an Object Key point Similarity (OKS) value of 0.921, with a single-frame processing time of 0.528 s, accurately detecting multiple key points of the abnormal cage-reared duck bodies and generating correct posture expressions.

Deep learning-based automatic measurement system for patellar height: a multicenter retrospective study

BackgroundThe patellar height index is important; however, the measurement procedures are time-consuming and prone to significant variability among and within observers. We developed a deep learning-based automatic measurement system for the patellar height and evaluated its performance and generalization ability to accurately measure the patellar height index.MethodsWe developed a dataset containing 3,923 lateral knee X-ray images. Notably, all X-ray images were from three tertiary level A hospitals, and 2,341 cases were included in the analysis after screening. By manually labeling key points, the model was trained using the residual network (ResNet) and high-resolution network (HRNet) for human pose estimation architectures to measure the patellar height index. Various data enhancement techniques were used to enhance the robustness of the model. The root mean square error (RMSE), object keypoint similarity (OKS), and percentage of correct keypoint (PCK) metrics were used to evaluate the training results. In addition, we used the intraclass correlation coefficient (ICC) to assess the consistency between manual and automatic measurements.ResultsThe HRNet model performed excellently in keypoint detection tasks by comparing different deep learning models. Furthermore, the pose_hrnet_w48 model was particularly outstanding in the RMSE, OKS, and PCK metrics, and the Insall–Salvati index (ISI) automatically calculated by this model was also highly consistent with the manual measurements (intraclass correlation coefficient [ICC], 0.809–0.885). This evidence demonstrates the accuracy and generalizability of this deep learning system in practical applications.ConclusionWe successfully developed a deep learning-based automatic measurement system for the patellar height. The system demonstrated accuracy comparable to that of experienced radiologists and a strong generalizability across different datasets. It provides an essential tool for assessing and treating knee diseases early and monitoring and rehabilitation after knee surgery. Due to the potential bias in the selection of datasets in this study, different datasets should be examined in the future to optimize the model so that it can be reliably applied in clinical practice.Trial registrationThe study was registered at the Medical Research Registration and Filing Information System (medicalresearch.org.cn) MR-61-23-013065. Date of registration: May 04, 2023 (retrospectively registered).

Machine learning approaches to evaluate infants’ general movements in the writhing stage—a pilot study

The goals of this study are to describe machine learning techniques employing computer-vision movement algorithms to automatically evaluate infants’ general movements (GMs) in the writhing stage. This is a retrospective study of infants admitted 07/2019 to 11/2021 to a level IV neonatal intensive care unit (NICU). Infant GMs, classified by certified expert, were analyzed in two-steps (1) determination of anatomic key point location using a NICU-trained pose estimation model [accuracy determined using object key point similarity (OKS)]; (2) development of a preliminary movement model to distinguish normal versus cramped-synchronized (CS) GMs using cosine similarity and autocorrelation of major joints. GMs were analyzed using 85 videos from 74 infants; gestational age at birth 28.9 ± 4.1 weeks and postmenstrual age (PMA) at time of video 35.9 ± 4.6 weeks The NICU-trained pose estimation model was more accurate (0.91 ± 0.008 OKS) than a generic model (0.83 ± 0.032 OKS, p < 0.001). Autocorrelation values in the lower limbs were significantly different between normal (5 videos) and CS GMs (5 videos, p < 0.05). These data indicate that automated pose estimation of anatomical key points is feasible in NICU patients and that a NICU-trained model can distinguish between normal and CS GMs. These preliminary data indicate that machine learning techniques may represent a promising tool for earlier CP risk assessment in the writhing stage and prior to hospital discharge.

Deep learning-based augmented reality work instruction assistance system for complex manual assembly

The manual assembly process of complex products is lengthy, the assembly requirements are difficult to recall, and the assembly quality requirements are high. The separation of the operation guidance information from the real physical object in the traditional paper manual easily distracts the operator, increasing their cognitive burden. To address this issue, we integrate augmented reality (AR) and artificial intelligence (AI) technologies for manual assembly assistance. A novel encoding and decoding convolutional neural network (CNN) is built to realize accurate AR registration under mark-less assembly environment. An assembly quality inspection method integrating a neural network and virtual model matching was proposed for AR systems. The specific process of the proposed method includes two stages: offline and online. In the first stage, a monocular RGB image dataset was built for assembling object keypoints and assembly object detection. CNN models were designed and trained for deployment in an AR-assisted assembly system. Second, the deployed CNN models were used to perform AR registration and assembly quality inspection. Experimental results demonstrate that the proposed method can accurately present AR work instruction guidance and assembly quality inspection for manual assembly. The proposed AR registration method can achieve a frame rate of 25FPS, which satisfies the timeliness requirements of the AR system. The accuracy rate of the proposed assembly quality inspection method for the MONA component assembly reached 92.5%.

CatTrack: Single-Stage Category-Level 6D Object Pose Tracking via Convolution and Vision Transformer

In the current research, many researchers have focused on instance-level pose tracking, which requires a 3D model of the object in advance, making it challenging to apply in practice. To address this limitation, some researchers have proposed the category-level object pose tracking method. Achieving accurate and speedy monocular category-level pose tracking is an essential research goal. In this paper, we propose CatTrack, a new single-stage keypoints-based monocular category-level multi-object pose tracking network. A significant issue in object pose tracking tasks is utilizing the information from the previous frame to guide pose estimation for the next frame. However, as the object poses and camera information in each frame are different, we need to remove irrelevant information and emphasize useful features. To this end, we propose a transformer-based temporal information capture module to leverage the position information of keypoints from the previous frame. Furthermore, we propose a new keypoint matching module to enable the grouping and matching of object keypoints in complex scenes. We have successfully applied CatTrack to the Objectron dataset and achieved superior results in comparison to existing methods. Furthermore, we have also evaluated the generalization of CatTrack and successfully applied it to track the 6D pose of unseen real-world objects. A video is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/Yminjdtsgwk</uri> .

Multi-person dance tiered posture recognition with cross progressive multi-resolution representation integration.

Recognizing postures in multi-person dance scenarios presents challenges due to mutual body part obstruction and varying distortions across different dance actions. These challenges include differences in proximity and size, demanding precision in capturing fine details to convey action expressiveness. Robustness in recognition becomes crucial in complex real-world environments. To tackle these issues, our study introduces a novel approach, i.e., Multi-Person Dance Tiered Posture Recognition with Cross Progressive Multi-Resolution Representation Integration (CPMRI) and Tiered Posture Recognition (TPR) modules. The CPMRI module seamlessly merges high-level features, rich in semantic information, with low-level features that provide precise spatial details. Leveraging a cross progressive approach, it retains semantic understanding while enhancing spatial precision, bolstering the network's feature representation capabilities. Through innovative feature concatenation techniques, it efficiently blends high-resolution and low-resolution features, forming a comprehensive multi-resolution representation. This approach significantly improves posture recognition robustness, especially in intricate dance postures involving scale variations. The TPR module classifies body key points into core torso joints and extremity joints based on distinct distortion characteristics. Employing a three-tier tiered network, it progressively refines posture recognition. By addressing the optimal matching problem between torso and extremity joints, the module ensures accurate connections, refining the precision of body key point locations. Experimental evaluations against state-of-the-art methods using MSCOCO2017 and a custom Chinese dance dataset validate our approach's effectiveness. Evaluation metrics including Object Keypoint Similarity (OKS)-based Average Precision (AP), mean Average Precision (mAP), and Average Recall (AR) underscore the efficacy of the proposed method.

AttentionPose: Attention-driven end-to-end model for precise 6D pose estimation

Abstract Addressing the complex problem of 6D pose estimation from single RGB images is essential for robotics, augmented reality, and autonomous driving applications. The aim of this study is to overcome limitations in handling scenes with high object occlusion and clutter. We introduce an attention-driven end-to-end model that builds upon existing methods employing pixel-wise unit vectors and voting for object keypoints. Integrating attention mechanisms allows the model to focus computational resources on salient features, enhancing accuracy. Experimental results using the LINEMOD benchmark dataset demonstrate an accuracy rate of 99.73%, outperforming state-of-the-art approaches. The model also exhibits strong generalization capabilities, achieving an average accuracy of 97.36% on objects not included in the dataset. This work concludes that the attention mechanism significantly elevates the performance and robustness of 6D pose estimation, particularly in challenging environments, and opens new avenues for real-world applications.

LiteDEKR: End‐to‐end lite 2D human pose estimation network

AbstractThe 2D human pose estimation plays an important role in human‐computer interaction and action recognition. Although the method based on high‐resolution network has superior performance, there is still room for improvement in terms of speed and lightweight. Here, a LiteDEKR, a 2D pose estimation method that combines lightweight and accuracy, is proposed by designing a lightweight network based on DEKR and constructing two scientifically valid loss functions. The method, constructs a multi‐instance bias regression loss that matches the true distribution of keypoint bias, improves the accuracy of bias regression, and constructs a keypoint similarity loss with the object keypoint similarity index of keypoints as the optimization objective to achieve end‐to‐end training of the network. In addition, this paper has designed a lightweight DEKR, using LitePose as the backbone network. With the optimization of the above two loss functions, LiteDEKR not only achieves lightweight but also has high accuracy. Comparative experiments on the COCO and CrowdPose datasets show that compared to the current state‐of‐the‐art Contextual Instance Decoupling, LiteDEKR achieves a similar accuracy with only 10% of its network complexity. It also shows better robustness to low‐resolution input images.

Automated Assessment of Radiographic Bone Loss in the Posterior Maxilla Utilizing a Multi-Object Detection Artificial Intelligence Algorithm

Periodontitis is one of the most prevalent diseases worldwide. The degree of radiographic bone loss can be used to assess the course of therapy or the severity of the disease. Since automated bone loss detection has many benefits, our goal was to develop a multi-object detection algorithm based on artificial intelligence that would be able to detect and quantify radiographic bone loss using standard two-dimensional radiographic images in the maxillary posterior region. This study was conducted by combining three recent online databases and validating the results using an external validation dataset from our organization. There were 1414 images for training and testing and 341 for external validation in the final dataset. We applied a Keypoint RCNN with a ResNet-50-FPN backbone network for both boundary box and keypoint detection. The intersection over union (IoU) and the object keypoint similarity (OKS) were used for model evaluation. The evaluation of the boundary box metrics showed a moderate overlapping with the ground truth, revealing an average precision of up to 0.758. The average precision and recall over all five folds were 0.694 and 0.611, respectively. Mean average precision and recall for the keypoint detection were 0.632 and 0.579, respectively. Despite only using a small and heterogeneous set of images for training, our results indicate that the algorithm is able to learn the objects of interest, although without sufficient accuracy due to the limited number of images and a large amount of information available in panoramic radiographs. Considering the widespread availability of panoramic radiographs as well as the increasing use of online databases, the presented model can be further improved in the future to facilitate its implementation in clinics.

Performance analysis of CNN speed and power consumption among CPUs, GPUs and an FPGA for EU GDPR 2016/679 compliant automotive applications

This work aims at using Field Programmable Gate Arrays (FPGAs) to detect license plates and pedestrians in real-time with superior energy and speed performance than Graphics Processing Units (GPUs) and Central Processing Units (CPUs) based on Convolutional Neural Networks (CNNs). Yolo V4 and PoseNet models are applied to personal data anonymization to comply with the EU General Data Protection Regulation (GDPR). The models are analyzed through the mean Average Precision (mAP) and object keypoint similarity (OKS). Optimization is done by model pruning and improves the frame rate of the YOLO V4 algorithm from 35 FPS (GPU) to 195 FPS (FPGA), reducing the energy consumption by 50.8% (1.192 mJ/Frame) compared to the GPU with a slight accuracy loss. PoseNet has obtained a mean performance of 312 FPS against 145 FPS on the GPU and a 51.9% (1.104 mJ/Frame) reduction.

ER-Pose: Learning edge representation for 6D pose estimation of texture-less objects

6D pose estimation from a single RGB image is a fundamental task in computer vision. We introduce a two-stage 6D pose estimation method for texture-less objects. Instead of utilizing the object mask in almost current monocular methods, we propose an edge representation for texture-less objects. An object is represented by the combination of visible edges corresponding to the object’s 6D pose, allowing the neural network to focus more on the object’s invariant global shape and structure, rather than indistinguishable local patches with image noise and similar texture. Given an RGB image, the proposed method predicts the direction and distance to a certain object keypoint from all object pixels within the range of object edge representation, establishes voting-based sparse 2D-3D correspondences, and solves the object pose with PnP algorithm. In the experiments, we found that directly replacing the object mask with the edge representation can bring a performance improvement in two current two-stage pipelines without any modification. Further evaluations on three different benchmark datasets containing symmetric and occluded objects show our method outperforms the state-of-the-art methods using RGB images only.

DRUNet: A Method for Infrared Point Target Detection

Deep learning is widely used in vision tasks, but feature extraction of IR small targets is difficult due to the inconspicuous contours and lack of color information. This paper proposes a new convolutional neural network–based (CNN-based) method for IR small target detection called DRUNet. The algorithm is divided into two parts: the feature extraction network and the prediction head. For the small IR targets, which are difficult to accurately label, Gaussian soft labels are added to supervise the training process and make the network converge faster. We use a simplified object keypoint similarity to evaluate the network accuracy by the ratio of the distance to the radius of the inner tangent circle of the target box and a fair method for evaluating the model inference speed after GPU preheating. The experimental results show that our proposed algorithm performs better when compared with commonly used algorithms in the field of small target detection. The model size is 10.5 M, and the test speed reaches 133 FPS under the RTX3090 experimental platform.

A Robust Convolutional Neural Network for 6D Object Pose Estimation from RGB Image with Distance Regularization Voting Loss

Six-degree (6D) pose estimation of objects is important for robot manipulation but at the same time challenging when dealing with occluded and textureless objects. To overcome this challenge, the proposed method presents an end-to-end robust network for real-time 6D pose estimation of rigid objects using the RGB image. In this proposed method, a fully convolutional network with a features pyramid is developed that effectively boosts the accuracy of pixelwise labeling and direction unit vector field that take part in the voting process for object keypoints estimation. The network further takes into account measuring the distance between pixel and keypoint, which aims to help select accurate hypotheses in the RANSAC process. This avoids hypothesis deviations caused by the errors due to direction unit vectors in cases of distant pixels from keypoints. A vectorial distance regularization loss function is used to help Perspective-n-Point find 2D-3D correspondences between 3D object keypoints and their estimated corresponding 2D counterparts. Experiments are performed on widely used LINEMOD and occlusion LINEMOD datasets with ADD (-S) and 2D projection evaluation metrics. The results show that our method improves pose estimation performance compared to the state-of-the-art while still achieving real-time efficiency.

E2EK: End-to-End Regression Network Based on Keypoint for 6D Pose Estimation

The methods based on deep learning are the mainstream of 6D object pose estimation, which mainly include direct regression and two-stage pipelines. The former are keen by many scholars at first due to their simplicity and differentiability to poses, but they usually lack in accuracy when compared with the latter that estimate the intermediate variables relating to geometries such as object keypoints or 2D-3D correspondence before PnP/RANSAC algorithm. However, the loss function of the two-stage method is non-differentiable to the 6D pose, which is hard to apply in the tasks requiring the differentiable poses. To overcome the disadvantages of the above methods, we propose an end-to-end regression network based on keypoints for 6D pose estimation. Specifically, we supervise the point-wise keypoint offsets that help the network to learn the geometric information and directly regress the 6D pose through aggregating keypoints to achieve differentiability to the pose. Furthermore, we improve the sampling method by sampling points around objects that benefits the small object and design a unit loss function that helps the learning of the keypoints. Experimental results show that our approach outperforms most methods on LM, LM-O and YCB-V datasets.

Model-Agnostic Temporal Regularizer for Object Localization Using Motion Fields

Video analysis often requires locating and tracking target objects. In some applications, the localization system has access to the full video, which allows fine-grain motion information to be estimated. This paper proposes capturing this information through motion fields and using it to improve the localization results. The learned motion fields act as a model-agnostic temporal regularizer that can be used with any localization system based on keypoints. Unlike optical flow-based strategies, our motion fields are estimated from the model domain, based on the trajectories described by the object keypoints. Therefore, they are not affected by poor imaging conditions. The benefits of the proposed strategy are shown on three applications: 1) segmentation of cardiac magnetic resonance; 2) facial model alignment; and 3) vehicle tracking. In each case, combining popular localization methods with the proposed regularizer leads to improvement in overall accuracies and reduces gross errors.

Coarse-to-Fine Hand-Object Pose Estimation with Interaction-Aware Graph Convolutional Network.

The analysis of hand–object poses from RGB images is important for understanding and imitating human behavior and acts as a key factor in various applications. In this paper, we propose a novel coarse-to-fine two-stage framework for hand–object pose estimation, which explicitly models hand–object relations in 3D pose refinement rather than in the process of converting 2D poses to 3D poses. Specifically, in the coarse stage, 2D heatmaps of hand and object keypoints are obtained from RGB image and subsequently fed into pose regressor to derive coarse 3D poses. As for the fine stage, an interaction-aware graph convolutional network called InterGCN is introduced to perform pose refinement by fully leveraging the hand–object relations in 3D context. One major challenge in 3D pose refinement lies in the fact that relations between hand and object change dynamically according to different HOI scenarios. In response to this issue, we leverage both general and interaction-specific relation graphs to significantly enhance the capacity of the network to cover variations of HOI scenarios for successful 3D pose refinement. Extensive experiments demonstrate state-of-the-art performance of our approach on benchmark hand–object datasets.

Monocular 3D Detection With Geometric Constraint Embedding and Semi-Supervised Training

In this work, we propose a novel one-stage and keypoint-based framework for monocular 3D object detection using only RGB images, called KM3D-Net. 2D detection only requires a deep neural network to predict 2D properties of objects, as it is a semanticity-aware task. For image-based 3D detection, we argue that the combination of a deep neural network and geometric constraints are needed to synergistically estimate appearance-related and spatial-related information. Here, we design a fully convolutional model to predict object keypoints, dimension, and orientation, and combine these with perspective geometry constraints to compute position attributes. Further, we reformulate the geometric constraints as a differentiable version and embed this in the network to reduce running time while maintaining the consistency of model outputs in an end-to-end fashion. Benefiting from this simple structure, we propose an effective semi-supervised training strategy for settings where labeled training data are scarce. In this strategy, we enforce a consensus prediction of two shared-weights KM3D-Net for the same unlabeled image under different input augmentation conditions and network regularization. In particular, we unify the coordinate-dependent augmentations as the affine transformation for the differential recovering position of objects, and propose a keypoint-dropout module for network regularization. Our model only requires RGB images, without synthetic data, instance segmentation, CAD model, or depth generator. Extensive experiments on the popular KITTI 3D detection dataset indicate that the KM3D-Net surpasses state-of-the-art methods by a large margin in both efficiency and accuracy. And also, to the best of our knowledge, this is the first application of semi-supervised learning in monocular 3D object detection. We surpass most of the previous fully supervised methods with only 13% labeled data on KITTI.

Self-Supervised Attention-Aware Reinforcement Learning

Visual saliency has emerged as a major visualization tool for interpreting deep reinforcement learning (RL) agents. However, much of the existing research uses it as an analyzing tool rather than an inductive bias for policy learning. In this work, we use visual attention as an inductive bias for RL agents. We propose a novel self-supervised attention learning approach which can 1. learn to select regions of interest without explicit annotations, and 2. act as a plug for existing deep RL methods to improve the learning performance. We empirically show that the self-supervised attention-aware deep RL methods outperform the baselines in the context of both the rate of convergence and performance. Furthermore, the proposed self-supervised attention is not tied with specific policies, nor restricted to a specific scene. We posit that the proposed approach is a general self-supervised attention module for multi-task learning and transfer learning, and empirically validate the generalization ability of the proposed method. Finally, we show that our method learns meaningful object keypoints highlighting improvements both qualitatively and quantitatively.