Key Feature Points Research Articles

Application of Sparse-Channel Lidar Sensors on Viewpoint-Invariant Loop Closing Task

Loop closing is an essential task for SLAM system, which can reduce the pose drifts and map distortion. However, challenges from viewpoint changes and sparse-channel lidar are still open problems. In this paper, a novel efficient and viewpoint-invariant loop closing method is presented, which well fits sparse-channel lidars. To keep the metric information of 3D point clouds, elevation maps are explored to represent sparse point clouds, based on which a novel and viewpoint-invariant key point feature detection method is proposed. To improve efficiency, Bag of Words technique is applied to accelerate the process of scene description and matching. In addition, a strict geometric consistency check of matched key point pairs is proposed to reject false positive detections, which makes our method robust to scenes with similar layouts in urban environments. Extensive experiments on challenging public datasets verify the validity and superiority of the proposed method.

Color Image Feature Matching Method Based on the Improved Firework Algorithm

An improved ORB feature point purification method has been proposed to address the problem of large feature point matching error and low image registration rate in the oriented FAST and rotated BRIEF (ORB) feature point color image matching algorithm. Pure virtual quaternions were used to represent the color image pixels in this method, and an improved FAST algorithm was used to detect the color feature points at first. The firework algorithm has been used to divide the detected feature points into key areas, auxiliary areas, and small influence areas, depending on the degree of attachment. The feature points of the key area and the subsidiary area are the required feature points. In order to further improve the efficiency of matching the color image feature points, the firework explosion radius formula and the explosion number formula in the firework algorithm have been improved, and an improved firework algorithm is proposed. This improved algorithm purifies the quaternion-represented color image feature points. For feature matching, the hamming distance was used. Experiments show that, when compared to the traditional ORB algorithm, the improved algorithm retains the key feature points of the color image with high accuracy, removes a large number of irrelevant feature points and noise points, and provides significantly higher accuracy and efficiency of color image matching.

A Visual Human-Computer Interaction System Based on Hybrid Visual Model

The traditional human-computer interaction is mainly through the mouse, keyboard, remote control, and other peripheral equipment electromagnetic signal transmission. This paper aims to build a visual human-computer interaction system through a series of deep learning and machine vision models, so that people can achieve complete human-computer interaction only through the camera and screen. The established visual human-computer interaction system mainly includes the function modes of three basic peripherals in human-computer interaction: keyboard, mouse (X-Y position indicator), and remote control. The convex hull method was used to switch between these three modes. After issuing the mode command, Gaussian mixture was used to quickly identify the moving human body to narrow the scope of our image processing. Subsequently, finger detection in human body was realized based on the Faster-RCNN-ResNET50-FPN model structure, and realized the function of moving mouse and keyboard through the relationship between different fingers. At the same time, the recognition of human body posture was done by using MediaPipe BlazePose, and the action classification models were established through the Angle between body movements so as to realize the control function of remote control. In order to ensure the real-time performance of the interactive system, according to the characteristics of different data processing processes, CPU and GPU computing power resources are used to cross-process images to ensure the real-time performance. The recognition accuracy of the human-computer interaction system is above 0.9 for the key feature points of human body, and above 0.87 for the recognition accuracy of four kinds of command actions. It is hoped that vision-based human-computer interaction will become a widely used interaction mode in the future.

Murine Motion Behavior Recognition Based on DeepLabCut and Convolutional Long Short-Term Memory Network

Murine behavior recognition is widely used in biology, neuroscience, pharmacology, and other aspects of research, and provides a basis for judging the psychological and physiological state of mice. To solve the problem whereby traditional behavior recognition methods only model behavioral changes in mice over time or space, we propose a symmetrical algorithm that can capture spatiotemporal information based on behavioral changes. The algorithm first uses the improved DeepLabCut keypoint detection algorithm to locate the nose, left ear, right ear, and tail root of the mouse, and then uses the ConvLSTM network to extract spatiotemporal information from the keypoint feature map sequence to classify five behaviors of mice: walking straight, resting, grooming, standing upright, and turning. We developed a murine keypoint detection and behavior recognition dataset, and experiments showed that the method achieved a percentage of correct keypoints (PCK) of 87±1% at three scales and against four backgrounds, while the classification accuracy for the five kinds of behaviors reached 93±1%. The proposed method is thus accurate for keypoint detection and behavior recognition, and is a useful tool for murine motion behavior recognition.

Keypoint Message Passing for Video-Based Person Re-identification

Video-based person re-identification~(re-ID) is an important technique in visual surveillance systems which aims to match video snippets of people captured by different cameras. Existing methods are mostly based on convolutional neural networks~(CNNs), whose building blocks either process local neighbor pixels at a time, or, when 3D convolutions are used to model temporal information, suffer from the misalignment problem caused by person movement. In this paper, we propose to overcome the limitations of normal convolutions with a human-oriented graph method. Specifically, features located at person joint keypoints are extracted and connected as a spatial-temporal graph. These keypoint features are then updated by message passing from their connected nodes with a graph convolutional network~(GCN). During training, the GCN can be attached to any CNN-based person re-ID model to assist representation learning on feature maps, whilst it can be dropped after training for better inference speed. Our method brings significant improvements over the CNN-based baseline model on the MARS dataset with generated person keypoints and a newly annotated dataset: PoseTrackReID. It also defines a new state-of-the-art method in terms of top-1 accuracy and mean average precision in comparison to prior works.

Statistical terrain model with geometric feature detection based on GPU using LiDAR on vehicles

Digital terrain models based on light detection and ranging (LiDAR) are widely used in ground robots to achieve autonomous navigation. Very few studies have been conducted on terrain modelling for preview information, which acts as an input for a vehicle’s active suspension system. In this study, we proposed an algorithm to construct a real-time terrain model from multi-frame LiDAR points according to the applications for preview control. The terrain model consisted of an elevation map with confidence intervals and feature detection. First, the statistical confidence interval of the elevation map represented the reliability of the mean height. Further, we used spatial signal processing techniques to detect the uneven geometric features of the terrain. The key point feature on the trajectory was detected by the Laplacian of Gaussian method. Finally, the raw point cloud data processing procedure was based on parallel multi-thread techniques combined with GPU programming, which significantly promotes computation efficiency to enable real-time implementation. Thus, the proposed terrain model integrates reliability issues and geometric features, which act as inputs for preview controllers or motion planning. Experimental results proved that our method has a competitive performance (in terms of precision and efficiency) compared with the state-of-the-art methods, and runs in real-time at 50 Hz.

A Triangulation-Based Visual Localization for Field Robots

Dear Editor, Visual localization relies on local features and searches a pre-stored GPS-tagged image database to retrieve the reference image with the highest similarity in feature spaces to predict the current location [1]–[3]. For the conventional methods [4]–[6], local features are generally explored by multiple-stage feature extraction which first detects and then describes key-point features [4], [7]. The multiple-stage feature extraction is redundantly implemented, which is not memory and run-time efficient. Its performance degrades with challenging conditions such as poor lighting and weather variations (as shown in Fig. 1(a)) because the multiple-stage design may lose information in the quantization step which produces inadequately key-point features for matching. Another critical issue for existing visual localization is that most of the conventional methods are one-directional-based approaches, which only use one-directional images (front images) to search and match GPS-tag references [4], [8]. With the increase of database size, one-directional inputs can be homogeneous which makes it difficult for the localization algorithms to work robustly (as shown in Fig. 1(b)).

3D Dynamic Target Detection Algorithm Based on Voxel Point Cloud Fusion

Current methods used in the field of 3D target detection generally have the problem that target context features are not rich enough to achieve accurate dynamic multi-target detection in feature extraction. A 3D dynamic object detection algorithm based on voxel point cloud fusion is proposed. The algorithm uses a two-stage detection architecture of multiple and fusion of multi-scale feature. In the first stage, point cloud is directly processed to extract key point features and the voxel space is divided to extract multi-scale voxel features, and two features are firstly fused to generate preselection frame. In the second stage, a reference point is set in each voxel and surrounding key points are absorbed for the second feature fusion, and the final feature is input to a detection module to achieve optimization of the preselection box. Additionally, for the problem of inconsistency between classification and location reliability, a mandatory consistency loss function is proposed to further improve accuracy of detection. Compared with other algorithms in Kitti, Waymo and nuScene datasets, the accuracy of 3D dynamic target detection is 92.10%, the results on the real vehicle platform show that the algorithm has strong robustness, portability and generalization ability.

Collaborative Feature Learning for Gait Recognition Under Cloth Changes

Since gait can be utilized to identify individuals from a far distance without their interaction and coordination, recently many gait recognition methods have been proposed. However, due to a real-world scenario of clothing changes, a degradation occurs for most of these methods. Thus in this paper, a more efficient gait recognition method is proposed to address the problem of clothing variances. First, part-based gait features are formulated from two different perspectives, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the separated body parts that are more robust to clothing changes and the estimated human skeleton key-point regions. It is reasonable to formulate such features for cloth-changing gait recognition, because these two perspectives are both less vulnerable to clothing changes. Given that each feature has its own advantages and disadvantages, a more efficient gait feature is generated in this paper by assembling these two features together. Moreover, since local features are more discriminative than global features, in this paper more attention is focused on the local short-range features. Also, unlike most methods, in our method we treat the estimated key-point features as a set of word embeddings, and a transformer encoder is specifically used to learn the dependence of each correlative key-points. The robustness and effectiveness of our proposed method are certified by experiments on CASIA Gait Dataset B, and it has achieved the state-of-the-art performance on this dataset.

A SELF-SUPERVISED KEYPOINT DETECTION NETWORK FOR MULTIMODAL REMOTE SENSING IMAGES

Abstract. Currently, multimodal remote sensing images have complex geometric and radiometric distortions, which are beyond the reach of classical hand-crafted feature-based matching. Although keypoint matching methods have been developed in recent decades, most manual and deep learning-based techniques cannot effectively extract highly repeatable keypoints. To address that, we design a Siamese network with self-supervised training to generate similar keypoint feature maps between multimodal images, and detect highly repeatable keypoints by computing local spatial- and channel-domain peaks of the feature maps. We exploit the confidence level of keypoints to enable the detection network to evaluate potential keypoints with end-to-end trainability. Unlike most trainable detectors, it does not require the generation of pseudo-ground truth points. In the experiments, the proposed method is evaluated using various SAR and optical images covering different scenes. The results prove its superior keypoint detection performance compared with current state-of-art matching methods based on keypoints.

A pose-aware dynamic weighting model using feature integration for driver action recognition

Traffic accidents caused by distracted driving are on the rise, posing a serious threat to the safety of people’s lives and property. Recognition and early warning of the driver’s actions is particularly important. Considering the differences in local details of driver actions, we use the keypoint information of drivers that reflects the category differences. Specifically, we explicitly model keypoints features and propose a pose-aware driver action recognition model. We design a pose-based feature fusion module incorporating the attention mechanism, to fuse the global features and keypoint features of different scales in driving images. In addition, we propose an input-dependent weighting module to enhance the discrimination of the fused features. We use dynamic convolution to apply channel attention to convolution weights. According to the input driving image, the corresponding weights are adaptively generated for multiple convolution kernels, and the final weighted summation is carried out. This is a soft gate scheme, which opens a new perspective for driver action recognition. The proposed model achieves 90.7% and 95.3% accuracy on StateFarm dataset and SEU-Driving dataset, which are improved by 1.4% and 3.1% respectively compared to SOTA (MSA-CNN).

QRS complexes and T waves localization in multi-lead ECG signals based on deep learning and electrophysiology knowledge

Accurate locations of key feature points for QRS complexes and T waves in electrocardiograms (ECGs) play a vital role in the analysis of cardiovascular diseases, such as arrhythmia and myocardial infarction. Because accuracy is affected by specific morphological changes of QRS waves and ST-T segments, traditional methods of detecting these waves need to be improved, which depend on empirical parameters and lack of robustness. This paper described a novel QRS complex and T wave localization method based on a U-Net framework combining the convolution neural network and long short-term memory with ensemble learning for multi-lead ECG signals. Using multi-lead ECG as an input, the model extracted strong temporal correlation features and fine morphological features in the training process, and its output was the probability value for every sample point. Dynamic threshold adaptive adjustment strategies during decision-making were then used to locate QRS complexes and T waves. Based on electrophysiology knowledge (EK), this method considered each lead information and was used to reduce missed and false detections. The proposed method based on deep learning and EK was also validated by three public databases. The experimental results indicated that the proposed method achieved a comparable or even better performance, when compared with present state-of-art methods. The currently proposed method produced results with high sensitivity and positive predictive values ranging from 99.45% to 100% for the locations of QRS complex waves, and from 99.99% to 97.88% for the locations of T waves from three datasets. Moreover, definitive analyses for missed and false detections of ECG signals were also used. These signals contained specific QRS complexes and T waves, such as higher amplitude T waves and QRS complexes, which contained fine structures, inverted U waves behind T waves, and waves along with atrial fibrillation and ventricular premature events.

Visible–infrared person re-identification based on key-point feature extraction and optimization

Feature extraction for visible–infrared person re-identification (VI-ReID) is challenging because of the cross-modality discrepancy in the images taken by different spectral cameras. Most of the existing VI-ReID methods often ignore the potential relationship between features. In this paper, we intend to transform low-order person features into high-order graph features, and make full use of the hidden information between person features. Therefore, we propose a multi-hop attention graph convolution network (MAGC) to extract robust person joint feature information using residual attention mechanism while reducing the impact of environmental noise. The transfer of higher order graph features within MAGC enables the network to learn the hidden relationship between features. We also introduce the self-attention semantic perception layer (SSPL) which can adaptively select more discriminant features to further promote the transmission of useful information. The experiments on VI-ReID datasets demonstrate its effectiveness.

Keypoints-Based Deep Feature Fusion for Cooperative Vehicle Detection of Autonomous Driving

Sharing collective perception messages (CPM) between vehicles is investigated to decrease occlusions so as to improve the perception accuracy and safety of autonomous driving. However, highly accurate data sharing and low communication overhead is a big challenge for collective perception, especially when real-time communication is required among connected and automated vehicles. In this letter, we propose an efficient and effective keypoints-based deep feature fusion framework built on the 3D object detector PV-RCNN, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Fusion</i> PV-RCNN ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FPV-RCNN</b> for short), for collective perception. We introduce a high-performance bounding box proposal matching module and a keypoints selection strategy to compress the CPM size and solve the multi-vehicle data fusion problem. Besides, we also propose an effective localization error correction module based on the maximum consensus principle to increase the robustness of the data fusion. Compared to a bird’s-eye view (BEV) keypoints feature fusion, <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FPV-RCNN</b> achieves improved detection accuracy by about 9% at a high evaluation criterion (IoU 0.7) on the synthetic dataset COMAP dedicated to collective perception. In addition, its performance is comparable to two raw data fusion baselines that have no data loss in sharing. Moreover, our method also significantly decreases the CPM size to less than 0.3 KB, and is thus about 50 times smaller than the BEV feature map sharing used in previous works. Even with further decreased CPM feature channels, i. e., from 128 to 32, the detection performance does not show apparent drops. The code of our method is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/YuanYunshuang/FPV_RCNN</uri> .

Research on Badminton Teaching Technology Based on Human Pose Estimation Algorithm

Human pose estimation is an important task in physical education, which can provide a valuable reference for teachers and students. We propose a human pose estimation method based on part affinity field. Firstly, the correlation of position information and orientation information between limb regions is maintained by part affinity field. Then the key points of limb pose are localized by part confidence map, and finally, the part affinity field is integrated to correlate all the acquired feature key points to obtain the human pose estimation. With the aid of computer vision technology, the students’ training movements can be compared with the standard movements. It enables the students to feel the standard movements and badminton hitting points more intuitively. In the experiment, we set up a comparison experiment to compare the teaching mode of the method in this paper with the traditional teaching mode. The experimental results prove that through the teaching mode of our method, students have more standard strokes, more smooth skill switching between badminton serves and strokes, and higher badminton stroke scores. At the same time, such a teaching system adds a lot of fun to the course and makes the students’ participation higher.

Analysis of Competition and Training Videos of Speed Climbing Athletes Using Feature and Human Body Keypoint Detection Algorithms

Compared to 25 years ago, the climbing sport itself has changed dramatically. From a rock climbing modification to a separation in three independent disciplines, the requirements to athletes and trainers increased rapidly. To ensure continuous improvement of the sport itself, the usage of measurement and sensor technology is unavoidable. Especially in the field of the discipline speed climbing, which will be performed as a single discipline at the Olympic Games 2024 in Paris, the current state of the art of movement analysis only consists of video analysis and the benefit of the experience of trainers. Therefore, this paper presents a novel method, which supports trainers and athletes and enables analysis of motion sequences and techniques. Prerecorded video footage is combined with existing feature and human body keypoint detection algorithms and standardized boundary conditions. Therefore, several image processing steps are necessary to convert the recorded movement of different speed climbing athletes to significant parameters for detailed analysis. By studying climbing trials of professional athletes and the used techniques in different sections of the speed climbing wall, the aim among others is to get comparable results and detect mistakes. As a conclusion, the presented method enables powerful analysis of speed climbing training and competition and serves with the aid of a user-friendly designed interface as a support for trainers and athletes for the evaluation of motion sequences.

AFE-RCNN: Adaptive Feature Enhancement RCNN for 3D Object Detection

The point clouds scanned by lidar are generally sparse, which can result in fewer sampling points of objects. To perform precise and effective 3D object detection, it is necessary to improve the feature representation ability to extract more feature information of the object points. Therefore, we propose an adaptive feature enhanced 3D object detection network based on point clouds (AFE-RCNN). AFE-RCNN is a point-voxel integrated network. We first voxelize the raw point clouds and obtain the voxel features through the 3D voxel convolutional neural network. Then, the 3D feature vectors are projected to the 2D bird’s eye view (BEV), and the relationship between the features in both spatial dimension and channel dimension is learned by the proposed residual of dual attention proposal generation module. The high-quality 3D box proposals are generated based on the BEV features and anchor-based approach. Next, we sample key points from raw point clouds to summarize the information of the voxel features, and obtain the key point features by the multi-scale feature extraction module based on adaptive feature adjustment. The neighboring contextual information is integrated into each key point through this module, and the robustness of feature processing is also guaranteed. Lastly, we aggregate the features of the BEV, voxels, and point clouds as the key point features that are used for proposal refinement. In addition, to ensure the correlation among the vertices of the bounding box, we propose a refinement loss function module with vertex associativity. Our AFE-RCNN exhibits comparable performance on the KITTI dataset and Waymo open dataset to state-of-the-art methods. On the KITTI 3D detection benchmark, for the moderate difficulty level of the car and the cyclist classes, the 3D detection mean average precisions of AFE-RCNN can reach 81.53% and 67.50%, respectively.

DAMR: A deep gaussian mixture point cloud registration method with the dual attention mechanism

Point cloud registration is important for the processing of 3D model reconstruction, but with the challenge of low registration accuracy. In order to overcome this obstacle, this paper proposed a dual attention mechanism registration (DAMR) method for the point cloud registration. Firstly, the dual attention mechanism is utilized to extract key point features with different dimensions via assigning weights to the input point clouds. Secondly, the matching parameters are obtained by estimating the position correspondence between feature points and gaussian mixture model. Finally, the parameter unit blocks of two models are applied to recover the optimal transformation by matching parameters. In order to test the performance of our method, four groups of datasets include public data models and cultural relic models are adopted, respectively. Compared with traditional methods, our method has shown better performance to effectively ensured the registration accuracy of 3D point cloud models.

Bird posture recognition based on target keypoints estimation in dual-task convolutional neural networks

• The first study focuses on bird’s posture recognition using bird’s keypoints. • End-to-end dual-task network executes keypoints detection and posture recognition. • A component generation module utilizes the keypoints to generate the components. • Fusing global and local features improve the accuracy of bird posture recognition. • New bird posture dataset containing eight types of postures named IMLab-P8-2020. Bird behavior reflects its healthy and the habitat condition in which the bird’s posture recognition is the basis of bird behavior research. This paper proposes an end-to-end bird posture recognition based on target keypoints estimation in the dual-task network. The network composes two sub-networks, keypoint feature extraction network and local feature extraction network. In the keypoint feature extraction network, we take the high-resolution network (HRNet) as the backbone to detect the bird’s keypoints in the high-resolution branch and extract the global features from the low-resolution branch for posture recognition. We design the components generation module in the local feature extraction network, which takes detected keypoints into the clustering algorithm to generate the bird’s components. We then utilize the components into convolutional neural networks (CNNs) to extract the local features for posture recognition. Finally, we fuse the global and local features to execute the bird’s posture recognition. Our main contributions are fourfold: (1) We have been modified the first point of contribution. We take the bird’s keypoints, which have never been used in the bird’s posture recognition, to estimate the bird’s posture and achieved good results. (2) We propose an end-to-end dual-task network for bird’s keypoints detection and posture recognition. (3) The components generation module utilizes the bird’s keypoints to crop the bird components and generates the local feature for bird posture recognition with a shallow network. (4) Creating a bird posture image dataset named IMLab-P8-2020 containing eight bird postures. We experimentally evaluated our proposed net on self-collected data set and verified that the proposed net yields better performance and greater accuracy relative to well-known counterparts concerning various metrics.

FastNet: Fast high-resolution network for human pose estimation

Aiming at the problem of developing efficient models for human pose estimation algorithms under computation-limited resources. In this paper, we proposed an efficient high-resolution network for human pose estimation, which named FastNet. First of all, The Lite-HRNet was constructed by using four parallel subnetworks. Each subnetwork contained multiple bottleneck block for feature extraction. The Lite-HRNet can effectively learn the high-resolution representation features of key points because of maintaining high-resolution. Secondly, instead of standard convolution, the asymmetric convolution was introduced to build an asymmetric bottleneck module. The asymmetric bottleneck module has different aspect ratios, and can be used to exact image features of keypoints with multi-scale characteristics and reduce the number of parameters. Thirdly, the waterfall module composed of multiple parallel convolutions were proposed to aggregates features with the same spatial size which can efficiently obtain delicate local representations. It retains rich spatial information and results in precise keypoint localization. Finally, the bottleneck blocks were replaced with asymmetric bottleneck modules in the third subnetwork of LiteHRNet. By which, The waterfall module is embedded into the structure of FastNet. Comprehensive experiments demonstrate that the proposed method achieves superior results on two benchmark datasets, MSCOCO and MPII. Moreover, FastNet demonstrates superior results on human pose estimation over popular lightweight networks.