Keypoint Localization Research Articles

Intelligent Wearable Wrist Pulse Detection System Based on Piezoelectric Sensor Array

The human radial artery pulse carries a rich array of biomedical information. Accurate detection of pulse signal waveform and the identification of the corresponding pulse condition are helpful in understanding the health status of the human body. In the process of pulse detection, there are some problems, such as inaccurate location of radial artery key points, poor signal noise reduction effect and low accuracy of pulse recognition. In this system, the pulse signal waveform is collected by the main control circuit and the new piezoelectric sensor array combined with the wearable wristband, creating the hardware circuit. The key points of radial artery are located by an adaptive pulse finding algorithm. The pulse signal is denoised by wavelet transform, iterative sliding window and prediction reconstruction algorithm. The slippery pulse and the normal pulse are recognized by feature extraction and classification algorithm, so as to analyze the health status of the human body. The system has accurate pulse positioning, good noise reduction effect, and the accuracy of intelligent analysis is up to 98.4%, which can meet the needs of family health care.

KVNet: An iterative 3D keypoints voting network for real-time 6-DoF object pose estimation

Accurate and efficient object pose estimation holds the indispensable part of virtual/augmented reality (VR/AR) and many other applications. While previous works focus on directly regressing 6D pose from RGB and depth image and thus suffer from the non-linearity of rotation space, we propose an iterative 3D keypoints voting network, named as KVNet. Specifically, our method decouples the pose into separate translation and rotation branch, both estimated by Hough voting scheme. By treating the uncertainty of keypoints’ vote as the Lipschitz continuous function of seed points’ fused embedding feature, our method is able to adaptively select the optimal keypoints vote. In this way, we argue that KVNet bridges the gap between the non-linear rotation space and linear Euclidean space, which introduces inductive bias for our network to learn the intrinsic pattern and infer 6D pose from RGB and depth images. Furthermore, our model will refine the initial keypoints localization with iterative fashion. Experiments show that across three challenging benchmark datasets (LineMOD, YCB-Video and Occlusion LineMOD), our method exhibits excellent performance.

PolarPose: Single-stage Multi-person Pose Estimation in Polar Coordinates.

Regression based multi-person pose estimation receives increasing attention because of its promising potential in achieving realtime inference. However, the challenges in long-range 2D offset regression have restricted the regression accuracy, leading to a considerable performance gap compared with heatmap based methods. This paper tackles the challenge of long-range regression through simplifying the 2D offset regression to a classification task. We present a simple yet effective method, named PolarPose, to perform 2D regression in Polar coordinate. Through transforming the 2D offset regression in Cartesian coordinate to quantized orientation classification and 1D length estimation in the Polar coordinate, PolarPose effectively simplifies the regression task, making the framework easier to optimize. Moreover, to further boost the keypoint localization accuracy in PolarPose, we propose a multi-center regression to relieve the quantization error during orientation quantization. The resulting PolarPose framework is able to regress the keypoint offsets in a more reliable way, and achieves more accurate keypoint localization. Tested with the single-model and single-scale setting, PolarPose achieves the AP of 70.2% on COCO test-dev dataset, outperforming the state-of-the-art regression based methods. PolarPose also achieves promising efficiency, e.g., 71.5% AP at 21.5FPS and 68.5%AP at 24.2FPS and 65.5%AP at 27.2FPS on COCO val2017 dataset, faster than current state-of-the-art.

An Automatic Image Processing Method Based on Artificial Intelligence for Locating the Key Boundary Points in the Central Serous Chorioretinopathy Lesion Area.

Accurately and rapidly measuring the diameter of central serous chorioretinopathy (CSCR) lesion area is the key to judge the severity of CSCR and evaluate the efficacy of the corresponding treatments. Currently, the manual measurement scheme based on a single or a small number of optical coherence tomography (OCT) B-scan images encounters the dilemma of incredibility. Although manually measuring the diameters of all OCT B-scan images of a single patient can alleviate the previous issue, the situation of inefficiency will thus arise. Additionally, manual operation is subject to subjective factors of ophthalmologists, resulting in unrepeatable measurement results. Therefore, an automatic image processing method (i.e., a joint framework) based on artificial intelligence (AI) is innovatively proposed for locating the key boundary points of CSCR lesion area to assist the diameter measurement. Firstly, the initial location module (ILM) benefiting from multitask learning is properly adjusted and tentatively achieves the preliminary location of key boundary points. Secondly, the location task is formulated as a Markov decision process, aiming at further improving the location accuracy by utilizing the single agent reinforcement learning module (SARLM). Finally, the joint framework based on the ILM and SARLM is skillfully established, in which ILM provides an initial starting point for SARLM to narrow the active region of agent, and SARLM makes up for the defect of low generalization of ILM by virtue of the independent exploration ability of agent. Experiments reveal the AI-based method which joins the multitask learning, and single agent reinforcement learning paradigms enable agents to work in local region, alleviating the time-consuming problem of SARLM, performing location task in a global scope, and improving the location accuracy of ILM, thus reflecting its effectiveness and clinical application value in the task of rapidly and accurately measuring the diameter of CSCR lesions.

Solid Waste Detection in Cities Using Remote Sensing Imagery Based on a Location-Guided Key Point Network With Multiple Enhancements

Solid waste is a widespread problem that is having a negative effect on the global environment. Owing to the ability of macroscopic observation, it is reasonable to believe that remote sensing could be an effective way to realize the detection and monitoring of solid waste. Solid waste is usually a mixture of various materials, with a randomly scattered distribution, which brings great difficulty to precise detection. In this article, we propose a deep learning network for solid waste detection in urban areas, aiming to realize the fast and automatic extraction of solid waste from the complicated and large-scale urban background. A novel dataset for solid waste detection was constructed by collecting 3192 images from Google Earth (with a resolution from 0.13 to 0.52 m), and then a location-guided key point network with multiple enhancements (LKN-ME) is proposed to perform the urban solid waste detection task. The LKN-ME method uses corner pooling and central convolution to capture the key points of an object. The location guidance is realized through constraining the key point locations situated of the annotated bounding box of an object. Multiple enhancements, including data mosaicing, an attention enhancement, and path aggregation, are integrated to improve the detection accuracy. The results show that the LKN-ME method can achieve a state-of-the-art AR <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">100</sub> (the average recall computed over 100 detections per image) of 71.8% and an average precision of 44.0% for the DSWD dataset, outperforming the classic object detection methods in solving the solid waste detection problem.

Pedestrian attribute recognition based on dual self-attention mechanism

Recognizing pedestrian attributes has recently obtained increasing attention due to its great potential in person re-identification, recommendation system, and other applications. Existing methods have achieved good results, but these methods do not fully utilize region information and the correlation between attributes. This paper aims at proposing a robust pedestrian attribute recognition framework. Specifically, we first propose an end-to-end framework for attribute recognition. Secondly, spatial and semantic self-attention mechanism is used for key points localization and bounding boxes generation. Finally, a hierarchical recognition strategy is proposed, the whole region is used for the global attribute recognition, and the relevant regions are used for the local attribute recognition. Experimental results on two pedestrian attribute datasets PETA and RAP show that the mean recognition accuracy reaches 84.63% and 82.70%. The heatmap analysis shows that our method can effectively improve the spatial and the semantic correlation between attributes. Compared with existing methods, it can achieve better recognition effect.

Filtering 2D-3D Outliers by Camera Adjustment for Visual Odometry

We study the problem of the discrepancy between model predictions and image measurements in the form of keypoint locations for perspective cameras. In this process, the prediction is made by projecting given 3D points using the known pose of a calibrated camera. We test whether some small camera pose adjustment exists for each measurement such that the mentioned discrepancy vanishes. Such adjustment would allow us to quantify the effect of each measurement on the camera pose. In this paper, we show for the first time that the pose influence assessment of individual measurements can be used to select a subset of the correspondences for accurate 3D triangulation from two views. We further demonstrate via several experiments that the obtained 3D points are well suited to the task of absolute localization. When the 3D points are provided from an anonymized source, the proposed method also selects a suitable subset of 3D points for accurate localization around an initial guess. The long-term effectiveness of our filtration method is demonstrated by integrating the method within a typical framework of visual odometry. The proposed method is evaluated on ETH3D and EuRoC benchmarks with real-world data. The results indicate that the proposed method outperforms the state-of-the-art methods in terms of the point uncertainty measure and camera pose estimation accuracy.

From Keypoints to Object Landmarks via Self-Training Correspondence: A novel approach to Unsupervised Landmark Discovery.

This paper proposes a novel paradigm for the unsupervised learning of object landmark detectors. Contrary to existing methods that build on auxiliary tasks such as image generation or equivariance, we propose a self-training approach where, departing from generic keypoints, a landmark detector and descriptor is trained to improve itself, tuning the keypoints into distinctive landmarks. To this end, we propose an iterative algorithm that alternates between producing new pseudo-labels through feature clustering and learning distinctive features for each pseudo-class through contrastive learning. With a shared backbone for the landmark detector and descriptor, the keypoint locations progressively converge to stable landmarks, filtering those less stable. Compared to previous works, our approach can learn points that are more flexible in terms of capturing large viewpoint changes. We validate our method on a variety of difficult datasets, including LS3D, BBCPose, Human3.6M and PennAction, achieving new state of the art results. Code and models can be found at https://github.com/malldimi1/KeypointsToLandmarks.

End-to-end Monocular Pose Estimation for Uncooperative Spacecraft based on Direct Regression Network

Deep learning shows good performance in monocular pose estimation and has been used by some space researchers to solve the monocular pose estimation problem of uncooperative spacecraft. However, existing deep learning-based methods are mostly trained with keypoint regression errors unnecessarily reflecting actual pose errors, limiting their learning performance. In this paper, an end-to-end pose estimation network based on the convolutional neural network (CNN) is proposed for the uncooperative spacecraft. First, we design a keypoint regression sub-network based on the multi-branch structure to regress the 2D keypoint locations. Then, we propose a pose estimation sub-network to estimate the pose of the target spacecraft from the predicted 2D keypoints and the corresponding 3D keypoints of the target model, which allows the end-to-end training of the overall pose estimation network with actual pose error. The experimental results on two public datasets demonstrate that the proposed method can accurately estimate the target spacecraft pose in the presence of scale variance and dynamic Earth background and has better pose estimation accuracy than the current state-of-the-art methods. In addition, the proposed method shows good generalization performance and near real-time efficiency.

REG-Net: Improving 6DoF Object Pose Estimation With 2D Keypoint Long-Short-Range-Aware Registration

TheSixdegrees of freedom 6DoF pose estimation of texture-less objects provides a spatial understanding of industrial scenes and is the basis for accurate object manipulation. Recent studies have shown that the introduction of known model information and initial poses helps CNNs-based methods achieve better performance in complex scenes. However, the mapping from image space to pose space learned by neural networks is dimensionally lifting. Due to the lack of depth information, it is difficult for neural networks to capture local clues on texture-less surfaces and directly regress the relative 3-D translation and 3-D rotation. Instead, we propose a novel framework named REG-Net, which transforms the 6DoF pose estimation task into a 2-D keypoint registration problem. The proposed network first encodes regional prior information using multi-representation, utilizes the globally-consistent offset attention module to align 2-D keypoint features in a long range, and then estimates offsets and potential regions of keypoints. The proposed regional PnP simultaneously adjusts the keypoint locations in a short range and outputs the pose. This framework compresses the learning space of the network from 3-D to 2-D. Extensive experiments on two benchmark datasets demonstrate the robustness and accuracy of REG-Net. We further demonstrate the effectiveness of REG-Net in the reflective industrial part grasping applications.

Cephalometric landmark keypoints localization based on convolution-enhanced Transformer

Cephalometric landmark keypoints localization based on convolution-enhanced Transformer

ECLNet: Center localization of eye structures based on Adaptive Gaussian Ellipse Heatmap

Accurately localizing the center of specific biological structures in medical images is of great significance for clinical treatment. The center localization task can be viewed as an estimation problem of keypoints, and the heatmap is often used to describe the probability of the location of keypoints during estimation. Existing methods construct the heatmap from a Gaussian kernel function with a fixed standard deviation, therefore cannot adapt to morphologic changes of the target region. In this paper, we build a deep network, ECLNet, to localize the center of eye-related structures in medical images. Meanwhile, we propose a method called Adaptive Gaussian Ellipse Heatmap (AGEH), which can efficiently utilize the gradient feature of the target region to adjust the morphology of the heatmap. The ECLNet localizes the optic disc and fovea center with mean Euclidean Distance of 17.995 and 39.446 pixels, respectively, for IDRiD dataset. The ECLNet also successfully localizes the eye center with the mean absolute Position Error of 0.186±0.027 mm for CATARACT dataset. The results show that our proposed method has a better performance compared with some state-of-the-art methods.

Estimating Ground Reaction Forces from Two-Dimensional Pose Data: A Biomechanics-Based Comparison of AlphaPose, BlazePose, and OpenPose.

The adoption of computer vision pose estimation approaches, used to identify keypoint locations which are intended to reflect the necessary anatomical landmarks relied upon by biomechanists for musculoskeletal modelling, has gained increasing traction in recent years. This uptake has been further accelerated by keypoint use as inputs into machine learning models used to estimate biomechanical parameters such as ground reaction forces (GRFs) in the absence of instrumentation required for direct measurement. This study first aimed to investigate the keypoint detection rate of three open-source pose estimation models (AlphaPose, BlazePose, and OpenPose) across varying movements, camera views, and trial lengths. Second, this study aimed to assess the suitability and interchangeability of keypoints detected by each pose estimation model when used as inputs into machine learning models for the estimation of GRFs. The keypoint detection rate of BlazePose was distinctly lower than that of AlphaPose and OpenPose. All pose estimation models achieved a high keypoint detection rate at the centre of an image frame and a lower detection rate in the true sagittal plane camera field of view, compared with slightly anteriorly or posteriorly located quasi-sagittal plane camera views. The three-dimensional ground reaction force, instantaneous loading rate, and peak force for running could be estimated using the keypoints of all three pose estimation models. However, only AlphaPose and OpenPose keypoints could be used interchangeably with a machine learning model trained to estimate GRFs based on AlphaPose keypoints resulting in a high estimation accuracy when OpenPose keypoints were used as inputs and vice versa. The findings of this study highlight the need for further evaluation of computer vision-based pose estimation models for application in biomechanical human modelling, and the limitations of machine learning-based GRF estimation models that rely on 2D keypoints. This is of particular relevance given that machine learning models informing athlete monitoring guidelines are being developed for application related to athlete well-being.

ASSESSMENT SYSTEM FOR SPORTS EXERCISES BY NEURAL NETWORK VIDEO ANALYSIS

This article describes the initial stage of developing an information system for evaluating a sports exercise based on the use of neural networks. The current approach to the evaluation of sports exercises and its shortcomings, as well as the advantages of introducing computer technology, were considered. The analysis of initial data is carried out. Next, the approaches that are used to analyze the position of the athlete's body during the exercise are considered, and an approach is chosen to obtain the position of the athlete's body in space and time. The definition of the concept of key points is given and their location, number on the human body are described. A number of pre-trained neural networks are considered that determine the location of key points on the human body, and the best option for solving the problem posed in this article is selected. The data sets that are used in the training of neural networks, in tasks of determining the position of the human body, are studied. The problem of image analysis is posed and an algorithm for solving the problem is introduced. As a result of the work, a prototype of an information system was developed that is capable of receiving frames from the original video sequence, processing frames using a neural network, recording the position of key points in the image to a text file, and processing data from the file for analysis.

Motion pose estimation of inshore ships based on point cloud

This paper proposes a new method to obtain the pose information estimation of the ship at multiple points based on point cloud data. Compared with the traditional pose estimation method, which regards the ship as a point to estimate the pose information of the ship, the proposed algorithm can effectively obtain the pose information of the ship from multiple points. This paper uses the spatial location of numerous key points from point cloud data by studying the spatial structure information of the ship. By tracking the spatial trajectory changes of the key points of the ship, the real-time angle, distance, velocity, and other information of the ship’s multiple points are finally estimated. The algorithm is verified by simulating the point cloud data when the inshore is berthing. The results show that the error of the proposed method is much smaller than that of other existing methods, and the proposed method can provide more accurate and abundant ship pose information. The method presented in this paper is competitive with other existing nearshore attitude estimation methods.data is at: https://drive.google.com/file/d/1LYN_tMrqtncnENzC2E5Fer3DTwL8tXH9/view?usp=sharingcode is at: https://github.com/serendipitylxd/Estimation-of-ship-pose

VP-KLNet: efficient 6D object pose estimation with an enhanced vector-field prediction network and a keypoint localization network

6D object pose estimation is essential for many applications with high demands in accuracy and speed. Compared with end-to-end approaches, pixel-wise voting network (PVNet), a vector-field based two-stage approach, has shown the superiority in accuracy but the inferiority in speed because of the time-consuming RANSAC-based voting strategy. To resolve this problem, we propose an efficient deep architecture that consists of an enhanced vector-field prediction network (VPNet) and a keypoint localization network (KLNet) and call it VP-KLNet. Specifically, the KLNet replaces PVNet’s time-consuming voting scheme by directly regressing 2D keypoints from the vector fields, which significantly improves the inference speed. Furthermore, to capture multiscale contextual information, we embed the pyramid pooling module between the encoder and decoder in VPNet to obtain more accurate object segmentation and vector-field prediction, with negligible speed loss. Experiments demonstrate that our method has more than 50% improved in the running speed to the baseline method PVNet and achieves comparable accuracy with the state-of-the-art methods on the LINEMOD and occlusion LINEMOD datasets.

Understanding the topology of the road network and identifying key bayonet nodes to avoid traffic congestion

Network topology is the basic for the development of traffic management and control. In a road network, bayonets with installation of surveillance facilities are key components to recognize traffic congestion from time to time. Therefore, identifying the essential bayonets in a road network becomes one of the most efficient ways to alleviate traffic congestion for traffic engineers and transport department. To do so, this paper aims to propose a novel sorting algorithm based on similarity measurements and traffic flow information to efficiently identify key bayonets in road networks. Our research results show that by analyzing the bayonet data in a fixed period of time in a medium-sized city of China, we have successfully identified the location of key bayonet points. Most of these key bayonet points are closed to residential areas and important traffic stations. The rank of these bayonet points can help the city managers better understand the topological characteristics of the road network as well as the propagation of congestion so as to make the traffic policies or control strategies for traffic congestion alleviation.

Automatic Assessment of the Type and Intensity of Agitated Hand Movements

With increasing numbers of people living with dementia, there is growing interest in the automatic monitoring of agitation. Current assessments rely on carer observations within a framework of behavioural scales. Automatic monitoring of agitation can supplement existing assessments, providing carers and clinicians with a greater understanding of the causes and extent of agitation. Despite agitation frequently manifesting in repetitive hand movements, the automatic assessment of repetitive hand movements remains a sparsely researched field. Monitoring hand movements is problematic due to the subtle differences between different types of hand movements and variations in how they can be carried out; the lack of training data creates additional challenges. This paper proposes a novel approach to assess the type and intensity of repetitive hand movements using skeletal model data derived from video. We introduce a video-based dataset of five repetitive hand movements symptomatic of agitation. Using skeletal keypoint locations extracted from video, we demonstrate a system to recognise repetitive hand movements using discriminative poses. By first learning characteristics of the movement, our system can accurately identify changes in the intensity of repetitive movements. Wide inter-subject variation in agitated behaviours suggests the benefit of personalising the recognition model with some end-user information. Our results suggest that data captured using a single conventional RGB video camera can be used to automatically monitor agitated hand movements of sedentary patients.

Multi-Person Pose Estimation With Accurate Heatmap Regression and Greedy Association

Multi-person pose estimation aims at localizing the 2D keypoints (or body joints) for all the people in the image. There are mainly two paradigms to perform this task: top-down and bottom-up. In this paper, we present an advanced bottom-up approach based on accurate keypoint heatmap regression and greedy keypoint association. Firstly, we develop an encoding-decoding method with Gaussian heatmaps and guiding offset fields to represent multi-person pose information, encompassing keypoint positions and adjacent keypoint associations of all individuals in the scene. In particular, we analyze the deficiency of the Gaussian heatmap representation as regards keypoint localization precision if conventional element-wise <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{2}$ </tex-math></inline-formula> -type loss is employed merely for heatmap supervision. Therefore, we introduce a peak regularization loss to jointly supervise the heatmap regression. In addition, we present an improved Hourglass Network with multi-scale heatmap aggregation to simultaneously infer the said encoding. Finally, we propose a novel focal <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{2}$ </tex-math></inline-formula> loss to help the network cope with the imbalanced problem of keypoint detection in heatmaps. Our results show that the proposed approach surpasses other bottom-up approaches on COCO dataset, and even outperforms the top-down approaches on CrowdPose dataset containing more crowded scenes.

Temporal-Variation Skeleton Point Correction Algorithm for Improved Accuracy of Human Action Recognition

The technique of human action recognition can be applied in a number of fields, such as medical rehabilitation, posture in the domain of sports, and emotion perception in counselling. Deep learning action recognition models that consider the continuous changes in the human skeleton can efficiently identify action states, such as rehabilitation processes, posture, and changes in emotion. However, the keypoints of the human skeleton are susceptible to occlusion due to body movement, resulting in a lower confidence level in their positions, which in turn affects the training and execution of human action recognition models. This paper proposes a keypoint correction algorithm based on continuous-time images of temporal variation that can predict and correct the skeleton keypoints with low confidence, based on the previous and subsequent multiple images. The proposed temporal-variation skeleton keypoint correction algorithm can provide accurate feature association training for the locations of the skeleton keypoints. The proposed method is shown to improve the accuracy of human action recognition by at least 30% over three alternative models in the literature: STV-GCN, GCN-NAS, and 2S-AGCN. The proposed algorithm is also improved by at least 30% compared with the above-mentioned methods in the literature, based on an action recognition model with a Gaussian filter.